366 Commits

Author	SHA1	Message	Date
Marcin Rataj	1f7bbcbf6a	fix(deps): bump boxo and kad-dht for resource leak fixes (#11192) ... https://github.com/ipfs/boxo/pull/1103 https://github.com/libp2p/go-libp2p-kad-dht/pull/1226	2026-02-11 01:32:37 +01:00
Marcin Rataj	36c29c55f0	feat: update to Go 1.26 (#11189) ... * feat: update to Go 1.26 replace deprecated httputil.NewSingleHostReverseProxy (Director) with ReverseProxy.Rewrite, switch math/rand to math/rand/v2 in production code, update Dockerfile base image. * fix test to accept response with HTTP status of 307 and 308 where 302 and 301 are expected --------- Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com>	2026-02-11 00:08:28 +01:00
Marcin Rataj	c6702eaf88	fix: allow dag import of 1MiB chunks wrapped in dag-pb (#11185) ... IPIP-499's unixfs-v1-2025 profile uses 1MiB chunks. with --raw-leaves=false, protobuf wrapping pushes blocks slightly over 1MiB. the previous 1MiB SoftBlockLimit rejected these blocks on dag import. raise SoftBlockLimit to 2MiB to match the bitswap spec, which requires implementations to support blocks up to 2MiB. - raise SoftBlockLimit to 2MiB per the bitswap spec - update error messages and help text - bump boxo to main with ipfs/boxo#1101 (raised ChunkSizeLimit/BlockSizeLimit, 256-byte overhead budget) - update sharness tests for 2MiB boundary - add test/cli boundary tests for block put, dag put, dag import, ipfs add (raw and wrapped leaves), and bitswap exchange including regression tests for the libp2p message size hard limit	2026-02-06 23:55:40 +01:00
Marcin Rataj	3a6b1ee122	feat(gateway): IPIP-0524 Gateway.AllowCodecConversion config option (#11090) ... * feat(gateway): IPIP-0524 Gateway.AllowCodecConversion config option Wire up boxo's AllowCodecConversion config to control codec conversion behavior per IPIP-0524. When false (default), the gateway returns 406 Not Acceptable if the requested format doesn't match the block's codec. Clients should fetch raw blocks (`?format=raw`) and convert client-side. Ref: https://github.com/ipfs/specs/pull/524 Ref: https://github.com/ipfs/boxo/pull/1077 Ref: https://github.com/ipfs/gateway-conformance/pull/254 * chore: update boxo for improved 406 codec conversion error boxo now returns an actionable hint when codec conversion is rejected: suggests fetching raw block with ?format=raw and converting client-side. * chore: bump boxo and gateway-conformance to v0.10 * docs: add IPLD Logical Format note to AllowCodecConversion	2026-02-06 02:21:35 +01:00
Marcin Rataj	67c89bbd7e	feat(config): add Import.* for CID Profiles from IPIP-499 (#11148) ... * feat(config): Import.* and unixfs-v1-2025 profile implements IPIP-499: add config options for controlling UnixFS DAG determinism and introduces `unixfs-v1-2025` and `unixfs-v0-2015` profiles for cross-implementation CID reproducibility. changes: - add Import.* fields: HAMTDirectorySizeEstimation, SymlinkMode, DAGLayout, IncludeEmptyDirectories, IncludeHidden - add validation for all Import.* config values - add unixfs-v1-2025 profile (recommended for new data) - add unixfs-v0-2015 profile (alias: legacy-cid-v0) - remove deprecated test-cid-v1 and test-cid-v1-wide profiles - wire Import.HAMTSizeEstimationMode() to boxo globals - update go.mod to use boxo with SizeEstimationMode support ref: https://specs.ipfs.tech/ipips/ipip-0499/ * feat(add): add --dereference-symlinks, --empty-dirs, --hidden CLI flags add CLI flags for controlling file collection behavior during ipfs add: - `--dereference-symlinks`: recursively resolve symlinks to their target content (replaces deprecated --dereference-args which only worked on CLI arguments). wired through go-ipfs-cmds to boxo's SerialFileOptions. - `--empty-dirs` / `-E`: include empty directories (default: true) - `--hidden` / `-H`: include hidden files (default: false) these flags are CLI-only and not wired to Import.* config options because go-ipfs-cmds library handles input file filtering before the directory tree is passed to kubo. removed unused Import.UnixFSSymlinkMode config option that was defined but never actually read by the CLI. also: - wire --trickle to Import.UnixFSDAGLayout config default - update go-ipfs-cmds to v0.15.1-0.20260117043932-17687e216294 - add SYMLINK HANDLING section to ipfs add help text - add CLI tests for all three flags ref: https://github.com/ipfs/specs/pull/499 * test(add): add CID profile tests and wire SizeEstimationMode add comprehensive test suite for UnixFS CID determinism per IPIP-499: - verify exact HAMT threshold boundary for both estimation modes: - v0-2015 (links): sum(name_len + cid_len) == 262144 - v1-2025 (block): serialized block size == 262144 - verify HAMT triggers at threshold + 1 byte for both profiles - add all deterministic CIDs for cross-implementation testing also wires SizeEstimationMode through CLI/API, allowing Import.UnixFSHAMTSizeEstimation config to take effect. bumps boxo to ipfs/boxo@6707376 which aligns HAMT threshold with JS implementation (uses > instead of >=), fixing CID determinism at the exact 256 KiB boundary. * feat(add): --dereference-symlinks now resolves all symlinks Previously, resolving symlinks required two flags: - --dereference-args: resolved symlinks passed as CLI arguments - --dereference-symlinks: resolved symlinks inside directories Now --dereference-symlinks handles both cases. Users only need one flag to fully dereference symlinks when adding files to IPFS. The deprecated --dereference-args still works for backwards compatibility but is no longer necessary. * chore: update boxo and improve changelog - update boxo to ebdaf07c (nil filter fix, thread-safety docs) - simplify changelog for IPIP-499 section - shorten test names, move context to comments * chore: update boxo to 5cf22196 * chore: apply suggestions from code review Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com> * test(add): verify balanced DAG layout produces uniform leaf depth add test that confirms kubo uses balanced layout (all leaves at same depth) rather than balanced-packed (varying depths). creates 45MiB file to trigger multi-level DAG and walks it to verify leaf depth uniformity. includes trickle subtest to validate test logic can detect varying depths. supports CAR export via DAG_LAYOUT_CAR_OUTPUT env var for test vectors. * chore(deps): update boxo to 6141039ad8ef switches to 6141039ad8 changes since 5cf22196ad0b: - refactor(unixfs): use arithmetic for exact block size calculation - refactor(unixfs): unify size tracking and make SizeEstimationMode immutable - feat(unixfs): optimize SizeEstimationBlock and add mode/mtime tests also clarifies that directory sharding globals affect both `ipfs add` and MFS. * test(cli): improve HAMT threshold tests with exact +1 byte verification - add UnixFSDataType() helper to directly check UnixFS type via protobuf - refactor threshold tests to use exact +1 byte calculations instead of +1 file - verify directory type directly (ft.TDirectory vs ft.THAMTShard) instead of inferring from link count - clean up helper function signatures by removing unused cidLength parameter * test(cli): consolidate profile tests into cid_profiles_test.go remove duplicate profile threshold tests from add_test.go since they are fully covered by the data-driven tests in cid_profiles_test.go. changes: - improve test names to describe what threshold is being tested - add inline documentation explaining each test's purpose - add byte-precise helper IPFSAddDeterministicBytes for threshold tests - remove ~200 lines of duplicated test code from add_test.go - keep non-profile tests (pinning, symlinks, hidden files) in add_test.go * chore: update to rebased boxo and go-ipfs-cmds PRs * docs: add HAMT threshold fix details to changelog * feat(mfs): use Import config for CID version and hash function make MFS commands (files cp, files write, files mkdir, files chcid) respect Import.CidVersion and Import.HashFunction config settings when CLI options are not explicitly provided. also add tests for: - files write respects Import.UnixFSRawLeaves=true - single-block file: files write produces same CID as ipfs add - updated comments clarifying CID parity with ipfs add * feat(files): wire Import.UnixFSChunker and UnixFSDirectoryMaxLinks to MFS `ipfs files` commands now respect these Import.* config options: - UnixFSChunker: configures chunk size for `files write` - UnixFSDirectoryMaxLinks: triggers HAMT sharding in `files mkdir` - UnixFSHAMTDirectorySizeEstimation: controls size estimation mode previously, MFS used hardcoded defaults ignoring user config. changes: - config/import.go: add UnixFSSplitterFunc() returning chunk.SplitterGen - core/node/core.go: pass chunker, maxLinks, sizeEstimationMode to mfs.NewRoot() via new boxo RootOption API - core/commands/files.go: pass maxLinks and sizeEstimationMode to mfs.Mkdir() and ensureContainingDirectoryExists(); document that UnixFSFileMaxLinks doesn't apply to files write (trickle DAG limitation) - test/cli/files_test.go: add tests for UnixFSDirectoryMaxLinks and UnixFSChunker, including CID parity test with `ipfs add --trickle` related: boxo@54e044f1b265 * feat(files): wire Import.UnixFSHAMTDirectoryMaxFanout and UnixFSHAMTDirectorySizeThreshold wire remaining HAMT config options to MFS root: - Import.UnixFSHAMTDirectoryMaxFanout via mfs.WithMaxHAMTFanout - Import.UnixFSHAMTDirectorySizeThreshold via mfs.WithHAMTShardingSize add CLI tests: - files mkdir respects Import.UnixFSHAMTDirectoryMaxFanout - files mkdir respects Import.UnixFSHAMTDirectorySizeThreshold - config change takes effect after daemon restart add UnixFSHAMTFanout() helper to test harness update boxo to ac97424d99ab90e097fc7c36f285988b596b6f05 * fix(mfs): single-block files in CIDv1 dirs now produce raw CIDs problem: `ipfs files write` in CIDv1 directories wrapped single-block files in dag-pb even when raw-leaves was enabled, producing different CIDs than `ipfs add --raw-leaves` for the same content. fix: boxo now collapses single-block ProtoNode wrappers (with no metadata) to RawNode in DagModifier.GetNode(). files with mtime/mode stay as dag-pb since raw blocks cannot store UnixFS metadata. also fixes sparse file writes where writing past EOF would lose data because expandSparse didn't update the internal node pointer. updates boxo to v0.36.1-0.20260203003133-7884ae23aaff updates t0250-files-api.sh test hashes to match new behavior * chore(test): use Go 1.22+ range-over-int syntax * chore: update boxo to c6829fe26860 - fix typo in files write help text - update boxo with CI fixes (gofumpt, race condition in test) * chore: update go-ipfs-cmds to 192ec9d15c1f includes binary content types fix: gzip, zip, vnd.ipld.car, vnd.ipld.raw, vnd.ipfs.ipns-record * chore: update boxo to 0a22cde9225c includes refactor of maxLinks check in addLinkChild (review feedback). * ci: fix helia-interop and improve caching skip '@helia/mfs - should have the same CID after creating a file' test until helia implements IPIP-499 (tracking: https://github.com/ipfs/helia/issues/941) the test fails because kubo now collapses single-block files to raw CIDs while helia explicitly uses reduceSingleLeafToSelf: false changes: - run aegir directly instead of helia-interop binary (binary ignores --grep flags) - cache node_modules keyed by @helia/interop version from npm registry - skip npm install on cache hit (matches ipfs-webui caching pattern) * chore: update boxo to 1e30b954 includes latest upstream changes from boxo main * chore: update go-ipfs-cmds to 1b2a641ed6f6 * chore: update boxo to f188f79fd412 switches to boxo@main after merging https://github.com/ipfs/boxo/pull/1088 * chore: update go-ipfs-cmds to af9bcbaf5709 switches to go-ipfs-cmds@master after merging https://github.com/ipfs/go-ipfs-cmds/pull/315 --------- Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com>	2026-02-04 22:03:15 +01:00
Marcin Rataj	df9574e090	test: IPIP-523 format query precedence over Accept header (#11086) ... * test: IPIP-523 format query precedence over Accept header update boxo to ipfs/boxo#1074 and gateway-conformance tests to ipfs/gateway-conformance#252 for testing IPIP-523 changes where ?format= URL query parameter takes precedence over Accept HTTP header * chore: update boxo and gateway-conformance for IPIP-523 testing - boxo@9aeb0c835899fedb4f886040a4ccf3ba09fd97d4 - gateway-conformance@c82a2a9bc79303e0f07216a80cf454ef2a1e042c * chore(ci): switch to gateway-conformance@v0.9 * chore: update boxo with IPIP-523 changes * chore: update boxo to main after ipfs/boxo#1074 merge * docs: add gateway-conformance v0.9 to changelog	2026-02-04 05:48:54 +01:00
Andrew Gillis	dd882c0bdb	fix: recognize content-type application/x-tar (#11180) ... * fix: recognize content-type application/x-tar For context, see: https://github.com/ipfs/go-ipfs-cmds/pull/320 Closes: #11179	2026-02-03 07:32:21 -08:00
dependabot[bot]	3522136c4f	chore(deps): bump github.com/multiformats/go-multiaddr-dns from 0.4.1 to 0.5.0 in the ipfs-ecosystem group (#11177) ... * chore(deps): bump github.com/multiformats/go-multiaddr-dns Bumps the ipfs-ecosystem group with 1 update: [github.com/multiformats/go-multiaddr-dns](https://github.com/multiformats/go-multiaddr-dns). Updates `github.com/multiformats/go-multiaddr-dns` from 0.4.1 to 0.5.0 - [Release notes](https://github.com/multiformats/go-multiaddr-dns/releases) - [Commits](https://github.com/multiformats/go-multiaddr-dns/compare/v0.4.1...v0.5.0) --- updated-dependencies: - dependency-name: github.com/multiformats/go-multiaddr-dns dependency-version: 0.5.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: ipfs-ecosystem ... Signed-off-by: dependabot[bot] <support@github.com> * chore: run make mod_tidy --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com>	2026-02-02 08:41:16 -08:00
Andrew Gillis	77ed3dd0ef	feat(rpc): Content-Type headers and IPNS record get/put (#11067) ... * fix http header when compress enabled for get command Closes #2376 * fix(rpc): set Content-Type for ipfs get based on output format - set application/x-tar when outputting tar (default and --archive) - set application/gzip when compression is enabled (--compress) - update go-ipfs-cmds with Tar encoding type and RFC 6713 compliant MIME types (application/gzip instead of application/x-gzip) * test(rpc): add Content-Type header tests for ipfs get * feat(rpc): add Content-Type headers for binary responses set proper Content-Type headers for RPC endpoints that return binary data: - `dag export`: application/vnd.ipld.car - `block get`: application/vnd.ipld.raw - `diag profile`: application/zip - `get`: application/x-tar or application/gzip (already worked, migrated to new API) uses the new OctetStream encoding type and SetContentType() method from go-ipfs-cmds to specify custom MIME types for binary responses. refs: https://github.com/ipfs/kubo/issues/2376 * feat(rpc): add `ipfs name get` command for IPNS record retrieval add dedicated command to retrieve raw signed IPNS records from the routing system. returns protobuf-encoded IPNS record with Content-Type `application/vnd.ipfs.ipns-record`. this provides a more convenient alternative to `ipfs routing get /ipns/<name>` which returns JSON with base64-encoded data. the raw output can be piped directly to `ipfs name inspect`: ipfs name get <name> | ipfs name inspect spec: https://specs.ipfs.tech/ipns/ipns-record/ * feat(rpc): add `ipfs name put` command for IPNS record storage adds `ipfs name put` to complement `ipfs name get`, allowing users to store IPNS records obtained from external sources without needing the private key. useful for backup, restore, and debugging workflows. the command validates records by default (signature, sequence number). use `--force` to bypass validation for testing how routing handles malformed or outdated records. also reorganizes test/cli files: - rename http_rpc_* -> rpc_* to match existing convention - merge name_get_put_test.go into name_test.go - add file header comments documenting test purposes * chore(deps): update go-ipfs-cmds to latest master includes SetContentType() for dynamic Content-Type headers --------- Co-authored-by: Marcin Rataj <lidel@lidel.org>	2026-01-30 23:41:55 +01:00
Andrew Gillis	a137401272	Upgrade to Boxo v0.36.0 (#11175) ... * Upgrade to Boxo v0.36.0 * sharness: add missing metrics	2026-01-27 15:51:01 -08:00
Andrew Gillis	f6a5c5f5c5	chore: upgrade go-ds-pebble to v0.5.9 (#11170) ... - upgrade go-ds-pebble to [v0.5.9](https://github.com/ipfs/go-ds-pebble/releases/tag/v0.5.9) - git ignore ipfswatch binary	2026-01-22 23:38:59 -08:00
Marcin Rataj	bc9b388610	fix(kad-dht): peerstore and exploration loop (#11139) ... * fix(routing): update kad-dht with peerstore address clone fix closes #11116 See https://github.com/ipfs/kubo/issues/11116 for context of this fix * fix(routing): update kad-dht with CPL exploration fix fixes an infinite loop when all peers share the same CPL during provider exploration See https://github.com/libp2p/go-libp2p-kad-dht/pull/1216 * fix(routing): update kad-dht with shutdown loop check https://github.com/libp2p/go-libp2p-kad-dht/pull/1217 * depend on latest kad-dht fix * bump kad-dht to v0.37.0 --------- Co-authored-by: guillaumemichel <guillaume@michel.id> Co-authored-by: Guillaume Michel <guillaumemichel@users.noreply.github.com>	2026-01-23 06:41:47 +01:00
Marcin Rataj	edb7056747	feat(config): add Gateway.MaxRequestDuration option (#11138) ... * feat(config): add Gateway.MaxRequestDuration option exposes the previously hardcoded 1 hour gateway request deadline as a configurable option, allowing operators to adjust it to fit deployment needs. protects gateway from edge cases and slow client attacks. boxo: https://github.com/ipfs/boxo/pull/1079 * test(gateway): add MaxRequestDuration integration test verifies config is wired correctly and 504 is returned when exceeded * docs: add MaxRequestDuration to gateway production guide --------- Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com>	2026-01-15 19:31:16 +01:00
Andrew Gillis	6a595c27a9	upgrade go-libp2p-pubsub to v0.15.0 (#11144)	2026-01-13 20:27:58 -08:00
Andrew Gillis	ec973aeb38	feat: improved go-ds-flatfs (#11092) ... * datastore: upgrade go-ds-flatfs to v0.6.0 See: https://github.com/ipfs/go-ds-flatfs/pull/142 * docs(changelog): add go-ds-flatfs atomic batch writes *documents the new flatfs batch implementation that uses atomic operations via temp directory, preventing orphan blocks on interrupted imports and reducing memory usage. * includes improved tests, batch cleanup fixes, and docs * docs(changelog): reframe go-ds-flatfs entry for users focus on user benefits instead of implementation details	2026-01-13 02:42:23 -08:00
Marcin Rataj	1301710a91	fix(ci): parallelize gotest, cleanup output, flakiness (#11113) ... * ci: parallelize gotest by separating test/cli into own job split the Go Test workflow into two parallel jobs: - `unit-tests`: runs unit tests (excluding test/cli) - `cli-tests`: runs test/cli end-to-end tests test/cli takes ~3 minutes (~50% of total gotest time), so running it in parallel should reduce wall-clock CI time by ~1.5-2.5 minutes. both jobs produce JUnit XML and HTML reports for consistent debugging. * ci(gotest): reduce noise on test timeout panics add GOTRACEBACK=single to show only one goroutine stack instead of all when a test timeout panic occurs. this makes CI output much cleaner when tests hang. * fix(ci): prevent stderr from corrupting test JSON output - remove 2>&1 which mixed "go: downloading" stderr messages into JSON - add JSON validation before parsing - print failed test names for easier debugging * ci(gotest): use gotestsum for human-readable test output - replace per-package coverage loop with single gotestsum invocation - both unit-tests and cli-tests now show human-readable output - simplified coverage collection (single -coverprofile, no gocovmerge) - clarified step names to indicate they run tests * ci: fix codecov uploads by adding token - add CODECOV_TOKEN to gotest.yml and sharness.yml - update codecov-action to v5.5.2 - add fail_ci_if_error: false for robustness codecov stopped receiving coverage data ~1 year ago when they started requiring tokens for public repos * refactor(make): add test_unit and test_cli targets - add `make test_unit` for unit tests with coverage (used by CI) - add `make test_cli` for CLI integration tests (used by CI) - only disable colors when CI env var is set (local dev gets colors) - remove legacy targets: test_go_test, test_go_short, test_go_race, test_go_expensive - update gotest.yml to use make targets instead of inline commands - add test artifacts to .gitignore * fix(ci): move client/rpc tests to cli-tests job client/rpc tests use test/cli/harness which requires the ipfs binary. Move them from test_unit to test_cli where the binary is built. also: - update gotestsum to v1.13.0 - simplify workflow step names * fix(ci): use build tags when listing test packages go list needs build tags to properly exclude packages like fuse/mfs when running with TEST_FUSE=0 (nofuse tag). * fix(ci): move test/integration to cli-tests job test/integration tests need the ipfs binary, move them from test_unit to test_cli. * fix(test): fix flaky kubo-as-a-library and GetClosestPeers tests kubo-as-a-library: use `Bootstrap()` instead of raw `Swarm().Connect()` to fix race condition between swarm connection and bitswap peer discovery. `Bootstrap()` properly integrates peers into the routing system, ensuring bitswap learns about connected peers synchronously. GetClosestPeers: simplify retry logic using `EventuallyWithT` with 10-minute timeout. tests all 4 routing types (`auto`, `autoclient`, `dht`, `dhtclient`) against real bootstrap peers with patient polling. * fix(example): use bidirectional Swarm().Connect() for reliable bitswap - connect nodes bidirectionally (A→B and B→A) to simulate mutual peering - mutual peering protects connection from resource manager culling - use port 0 for random available ports (avoids CI conflicts) - enable LoopbackAddressesOnLanDHT for local testing - move retry logic to test file using require.Eventually * fix(ci): add test_examples target and parallel example-tests job - add `make test_examples` target to mk/golang.mk for consistency with test_unit/test_cli - move example tests to separate parallel CI job (example-tests) - example: use Bootstrap() with autoconf.FallbackBootstrapPeers for reliable bitswap - example: increase context timeout to 10 minutes - test: add 60s per-request timeout to GetClosestPeers (server has 30s routing timeout) - test: reduce EventuallyWithT to 3 minutes (locally passes in under 1 minute) * fix(ci): improve test targets, exclusion patterns, and artifact naming - define COVERPKG_EXCLUDE and UNIT_EXCLUDE as documented variables - use grep -vE with single regex instead of multiple grep -v calls - add mkdir -p before rm to ensure directories exist - add DEPS_GO dependency to test_cli target - make CLI test timeout configurable via TEST_CLI_TIMEOUT (default 10m) - fix test_examples cleanup on failure using subshell - reduce GetClosestPeers test wait time from 3m to 2m - rename artifacts to match job names: unit-tests-{junit,html}, cli-tests-{junit,html} - update cli-tests upload-artifact from v5 to v6 * fix(ci): fix unit test exclusion and speed up example test - fix UNIT_EXCLUDE regex to match client/rpc at end of path - remove public bootstrap peers from example (only connect to nodeA) - example test now runs in ~3s instead of timing out * fix(test): fix flaky TestAddMultipleGCLive race condition added time.Sleep after spawning GC goroutines to ensure they reach GCLock() before the test proceeds. without this, the adder's maybePauseForGC() might check GCRequested() before GC has even requested the lock, causing the lock to not be released and GC to block indefinitely. this matches the existing pattern in TestAddGCLive which already had this sleep. also replaced context.Background() with t.Context() in both TestAddMultipleGCLive and TestAddGCLive for proper test lifecycle management. * fix(example): use test harness settings for reliable CI the kubo-as-a-library example was flaky on CI. applied test-harness-like settings that match what transports_test.go uses: - TCP-only on 127.0.0.1 with random port (no QUIC/UDP) - explicitly disable non-TCP transports (QUIC, Relay, WebTransport, etc) - use NilRouterOption (no routing) since we connect peers directly - bitswap works with directly connected peers without DHT lookups - 2-minute context timeout - streaming output in test for debugging	2026-01-08 05:07:08 +01:00
Andrew Gillis	828526e515	datastore: upgrade go-ds-pebble to v0.5.8 (#11129) ... * datastore: upgrade go-ds-pebble to v0.5.8 - update 'go-ds-pebble` to [v0.5.8](https://github.com/ipfs/go-ds-pebble/releases/tag/v0.5.8) - updates github.com/cockroachdb/pebble to [v2.1.3](https://github.com/cockroachdb/pebble/releases/tag/v2.1.3) - enables Go 1.26 support	2026-01-06 01:38:03 -08:00
Andrew Gillis	edc2cadc85	output stdout and stderr on example test failure (#11119)	2025-12-19 17:08:22 -08:00
Marcin Rataj	91d392d9ae	chore: update go-libp2p 0.46 (#11105) ... * fix: update go-libp2p to v0.46.0 - reduced WebRTC log noise (go-libp2p#3426) - fixed mDNS discovery on Windows/macOS (go-libp2p#3434) - includes quic-go v0.57.1 (v0.56.0 + v0.57.0) * fix(example): kubo-as-a-library test timeout - use custom ports (4010/4011) to avoid conflicts with default 4001 - add 2-minute context timeout to fail fast - get peer addresses dynamically instead of hardcoding wrong port - wait for peer connection synchronously instead of fire-and-forget - update comments to reference autoconf.FallbackBootstrapPeers * chore: update p2p-forge to v0.7.0 * fix(test): wait for DHT readiness in GetClosestPeers test the test was failing for `routing_type=auto` because it only waited for swarm connections but not for the DHT routing table to be populated. added a separate probe loop that waits for GetClosestPeers to succeed before running the actual test assertions.	2025-12-16 19:03:08 +01:00
vlerdman	d29c0b9c01	Add bytes progress tracker for ipfs pin add (#11074) ... * Add bytes progress tracker for ipfs pin add * upgrade to boxo that has ipfs/boxo#1071	2025-12-02 17:11:22 -08:00
Andrew Gillis	21f50ac931	upgrade go-libp2p-kad-dht to v0.36.0 (#11079)	2025-11-26 11:31:22 -08:00
Andrew Gillis	1141220674	upgrade pebble to 2.1.2 (#11075)	2025-11-25 15:25:55 +00:00
Hector Sanjuan	73ab037d1d	feat: support GetClosesPeers (IPIP-476) and ExposeRoutingAPI by default (#10954) ... This allows Kubo to respond to the GetClosestPeers() http routing v1 endpoint as spec'ed here: https://github.com/ipfs/specs/pull/476 It is based on work from https://github.com/ipfs/boxo/pull/1021 We let IpfsNode implmement the contentRouter.Client interface with the new method. We use our WAN-DHT to get the closest peers. Additionally, Routing V1 HTTP API is exposed by default which enables light clients in browsers to use Kubo Gateway as delegated routing backend Co-authored-by: Marcin Rataj <lidel@lidel.org>	2025-11-19 10:51:56 +00:00
Guillaume Michel	702c63b6db	feat: enable DHT Provide Sweep by default (#10955) ... Co-authored-by: Marcin Rataj <lidel@lidel.org> Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com>	2025-11-12 23:55:17 +01:00
Andrew Gillis	149ca2fd3b	Upgrade to Boxo v0.35.2 (#11050) ... * Upgrade to Boxo v0.35.2	2025-11-06 10:30:03 -08:00
Marcin Rataj	f067a86170	fix(go-log@2.9/go-libp2p@0.45): dynamic log level control and tail (#11039) ... This fix restores dynamic log level control and tail for go-libp2p loggers Updated to: https://github.com/libp2p/go-libp2p/releases/tag/v0.45.0 https://github.com/ipfs/go-log/releases/tag/v2.9.0 these changes restore dynamic log level control and tail for go-libp2p subsystems after the migration to slog, fixing the regression introduced in https://github.com/libp2p/go-libp2p/pull/3364 Fixes https://github.com/ipfs/kubo/issues/11035 For details why and how, see explainer in https://github.com/ipfs/go-log/releases/tag/v2.9.0	2025-11-06 16:00:56 +01:00
Andrew Gillis	a4323abc10	Upgrade to Boxo v0.35.1 (#11043) ... * Upgrade to Boxo v0.35.1 * use tagged boxo release * fix lint error	2025-10-29 17:18:34 -07:00
Guillaume Michel	c2bf0f9515	feat(provider): resume cycle (#11031) ... * bump kad-dht: resume reprovide cycle * daemon: --provide-fresh-start flag * changelog * docs * go-fmt * chore: latest go-libp2p-kad-dht#1170 after conflict resolution, to confirm CI is still green * kad-dht: depend on latest master * move daemon flag to Provider.DHT.ResumeEnabled config * refactor: sweep provider datastore * bump kad-dht * bump kad-dht * bump kad-dht * make datastore keys constant * use kad-dht master * add emoji to changelog entry * go-fmt * bump kad-dht * test(provider): add tests for resume cycle feature validates Provide.DHT.ResumeEnabled behavior: - preserves cycle state when enabled (default) - resets cycle when disabled tests verify current_time_offset across restarts using JSON output --------- Co-authored-by: Marcin Rataj <lidel@lidel.org>	2025-10-29 11:07:46 +01:00
Andrew Gillis	2e9c4ec500	chore: upgrade pebble to v2.1.1 (#11040)	2025-10-28 23:15:32 +01:00
Guillaume Michel	8c5f302d25	fix(cli): provide stat cosmetics (#11034)	2025-10-28 22:38:37 +01:00
Marcin Rataj	ae78c7821c	fix: go-libp2p v0.44 with self-healing UPnP port mappings (#11032) ... * chore(deps): update go-libp2p to v0.44.0 - includes self-healing UPnP port mappings after router restarts - update go-netroute to v0.3.0 - update quic-go to v0.55.0 - add changelog entry for UPnP fix * docs: improve provide and UPnP clarity in changelog and docs - add alert polling rationale to changelog - add UPnP config note with default clarification - clarify sweep timing and prefix length explanations - add concrete examples for time offset and record holders - improve workers stats formatting - add See Also section to provide-stats.md * docs: add RISC-V prebuilt binaries to changelog and README - highlight linux-riscv64 availability with open hardware context - update README with arm64 builds, remove 32-bit examples	2025-10-25 01:13:34 +02:00
Guillaume Michel	16479ec692	feat(provide): detailed ipfs provide stat (#11019) ... * feat: provide stats * added N/A * format * workers stats alignment * ipfs provide stat --all --compact * consolidating compact stat * update column alignment * flags combinations errors * command description * change schedule AvgPrefixLen to float * changelog * alignments * provide stat description draft * rephrased provide-stats.md * linking provide-stats.md from command description * documentation test * fix: refactor provide stat command type handling - add extractSweepingProvider() helper to reduce nested type switching - extract lowWorkerThreshold constant for worker availability check - fix --lan error handling to work with buffered providers * docs: add clarifying comments * fix(commands): improve provide stat compact mode - prevent panic when both columns are empty - fix column alignment with UTF-8 characters - only track col0MaxWidth for first column (as intended) * test: add tests for ipfs provide stat command - test basic functionality, flags, JSON output - test legacy provider behavior - test integration with content scheduling - test disabled provider configurations - add parseSweepStats helper with t.Helper() * docs: improve provide command help text - update tagline to "Control and monitor content providing" - simplify help descriptions - make error messages more consistent - update tests to match new error messages * metrics rename ``` Next reprovide at: Next prefix: ``` updated to: ``` Next region prefix: Next region reprovide: ``` * docs: improve Provide system documentation clarity Enhance documentation for the Provide system to better explain how provider records work and the differences between sweep and legacy modes. Changes to docs/config.md: - Provide section: add clear explanation of provider records and their role - Provide.DHT: add provider record lifecycle and two provider systems overview - Provide.DHT.Interval: explain relationship to expiration, contrast sweep vs legacy behavior - Provide.DHT.SweepEnabled: rewrite to explain legacy problem, sweep solution, and efficiency gains - Monitoring section: prioritize command-line tools (ipfs provide stat) before Prometheus Changes to core/commands/provide.go: - ipfs provide stat help: add explanation of provider records, TTL expiration, and how sweep batching works Changes to docs/changelogs/v0.39.md: - Add context about why stats matter for monitoring provider health - Emphasize real-time monitoring workflow with watch command - Explain what users can observe (rates, queues, worker availability) * depend on latest kad-dht master * docs: nits --------- Co-authored-by: Marcin Rataj <lidel@lidel.org>	2025-10-23 20:29:36 +02:00
Andrew Gillis	f9dc739933	Upgrade to boxo master (#11026) ... Upgrade to latest go-dsqueue and go-ds-pebble	2025-10-16 17:06:01 -07:00
Andrew Gillis	2b5adeedcc	Upgrade go-ds-pebble to v0.5.3 (#11011) ... This fixes a panic that happens when reusing a batch go-datastore Batch.	2025-10-03 23:33:40 -07:00
Guillaume Michel	42a4935abf	chore: bump go-libp2p-kad-dht to v0.35.0 (#11002) ... * chore: bump go-libp2p-kad-dht * docs: update dependency versions in changelogs - update go-libp2p-kad-dht to v0.35.0 release link in v0.38 - move go-ds-pebble v0.5.2 entry to v0.39 changelog --------- Co-authored-by: Marcin Rataj <lidel@lidel.org>	2025-10-01 15:53:36 +02:00
Andrew Gillis	9370004b5f	upgrade go-ds-pebble to v0.5.2 (#11000) ... * upgrade go-ds-pebble to v0.5.2 * Change pebble option `MaxConcurrentCompactions` to `CompactionConcurrencyRange`	2025-10-01 01:10:49 -07:00
Andrew Gillis	a7ce33c722	Upgrade to Boxo v0.35.0 (#10999) ... * Upgrade to Boxo v0.35.0	2025-09-30 18:32:36 -07:00
gammazero	a86df5feef	Upgrade to Boxo v0.35.0	2025-09-30 14:12:57 -10:00
Marcin Rataj	13fbb76de4	chore: update boxo and kad-dht dependencies (#10995) ... - update boxo to v0.34.1-0.20250926171300-4c0aa3a121fb - update go-libp2p-kad-dht to v0.34.1-0.20250926161957-861573b39723 - update changelog to reference webui v4.9.1	2025-09-27 02:42:00 +02:00
Marcin Rataj	a688b7eeac	fix(mfs): add soft limit for --flush=false (#10985) ... * fix: add MFS operation limit for --flush=false adds a global counter that tracks consecutive MFS operations performed with --flush=false and fails with clear error after limit is reached. this prevents unbounded memory growth while avoiding the data corruption risks of auto-flushing. - adds Internal.MFSNoFlushLimit config - operations fail with actionable error at limit - counter resets on successful flush or any --flush=true operation - operations with --flush=true reset and don't count this commit removes automatic flush from https://github.com/ipfs/kubo/pull/10971 and instead errors to encourage users of --flush=false to develop a habit of calling 'ipfs files flush' periodically. boxo will no longer auto-flush (https://github.com/ipfs/boxo/pull/1041) to avoid corruption issues, and kubo applies the limit to 'ipfs files' commands instead. closes #10842 * test: add tests for MFSNoFlushLimit tests verify the new Internal.MFSNoFlushLimit config option: - default limit of 256 operations - custom limit configuration - counter reset on flush=true - counter reset on explicit flush command - limit=0 disables the feature - multiple MFS command types count towards limit * docs: explain why MFS operations fail instead of auto-flushing addresses feedback from https://github.com/ipfs/kubo/pull/10985#pullrequestreview-3256250970 - clarify that automatic flushing at limit was considered but rejected - explain the data corruption risks of auto-flushing - guide users who want auto-flush to use --flush=true (default) - document benefits of explicit failure for batch operations	2025-09-26 01:25:23 +02:00
Hector Sanjuan	f63887ae96	fix: provide Filestore nodes (#10990) ... * Filestore: provide Filestore nodes When strategy is set to "all" (the blockstore does all the providing when a block is written), no providing was happening to Filestore blocks that were not written to the underlying blockstore (so, the DAG leaves, as they live in the filesystem directly). This fixes that. * docs: clarify filestore and urlstore fix in changelog both filestore (local file references) and urlstore (HTTP/HTTPS URL references) blocks are now properly provided shortly after initial add	2025-09-26 00:49:02 +02:00
Marcin Rataj	f6a9b347cb	fix(cmds): cleanup unicode identify strings (#9465) ... preserve private use characters as specified in https://github.com/libp2p/specs/pull/491 enforce 128 rune limit on untrusted peer data	2025-09-19 04:46:38 +02:00
Marcin Rataj	d37b92bfcd	fix: ipfs pin ls <cid> --names (#10970) ... * fix: use CheckIfPinnedWithType for pin ls with names updates to use CheckIfPinnedWithType method from https://github.com/ipfs/boxo/pull/1035, enabling efficient pin name retrieval for 'ipfs pin ls <cid> --names' - uses new CheckIfPinnedWithType from boxo for type-specific pin checks - pin names are now returned when listing specific CIDs with --names flag * test: add CLI tests for pin ls with names tests cover: - pin ls with specific CIDs returning names - pin ls without CID listing all pins with names - pin ls with --type and --names combinations - JSON output with and without names - pin update preserving names - error cases (invalid CID, unpinned CID) * docs: add pin name improvements to v0.38 changelog covers fix for ipfs pin ls --names with specific CIDs and RPC pin name leak fix * fix(rpc): support pin names in Add() passes the Name field from PinAddSettings to the API request adds test to verify pin names work via RPC * test: add coverage for pin names functionality - test special characters, unicode, long names - test concurrent operations - test persistence across daemon restarts - test garbage collection preservation - fix indirect pin test logic * chore: boxo@main with boxo#1039 * fix(pin): improve pin ls robustness and validation - add nil check for n.Pinning with early fail-fast validation - use pin.StringToMode() for consistent type validation - add edge case tests for invalid types and unpinned CIDs	2025-09-19 03:17:45 +02:00
Guillaume Michel	006f9dc704	feat: opt-in new Sweep provide system (#10834) ... * reprovide sweep draft * update reprovider dep * go mod tidy * fix provider type * change router type * dual reprovider * revert to provider.System * back to start * SweepingReprovider test * fix nil pointer deref * noop provider for nil dht * disabled initial network estimation * another iteration * suppress missing self addrs err * silence empty rt err on lan dht * comments * new attempt at integrating * reverting changes in core/node/libp2p/routing.go * removing SweepingProvider * make reprovider optional * add noop reprovider * update KeyChanFunc type alias * restore boxo KeyChanFunc * fix missing KeyChanFunc * test(sharness): PARALLEL=1 and timeout 30m running sequentially to see where timeout occurs * initialize MHStore * revert workflow debug * config * config docs * merged IpfsNode provider and reprovider * move Provider interface to from kad-dht to node * moved Provider interface from kad-dht to kubo/core/node * mod_tidy * Add Clear to Provider interface * use latest kad-dht commit * make linter happy * updated boxo provide interface * boxo PR fix * using latest kad-dht commit * use latest boxo release * fix fx * fx cyclic deps * fix merge issues * extended tests * don't provide LAN DHT * docs * restore dual dht provider * don't start provider before it is online * address linter * dual/provider fix * add delay in provider tests for dht bootstrap * add OfflineDelay parameter to config * remove increase number of workers in test * improved keystore gc process * fix: replace incorrect logger import in coreapi replaced github.com/labstack/gommon/log with the standard github.com/ipfs/go-log/v2 logger used throughout kubo. removed unused labstack dependency from go.mod files. * fix: remove duplicate WithDefault call in provider config * fix: use correct option method for burst workers * fix: improve error messages for experimental sweeping provider updated error messages to clearly indicate when commands are unavailable due to experimental sweeping provider being enabled via Reprovider.Sweep.Enabled=true * docs: remove obsolete KeyStoreGCInterval config removed from config.md as option no longer exists (removed in b540fba1a) updated keystore description to reflect gc happens at reprovide interval * docs: add TODO placeholder changelog for experimental sweeping DHT provider using v0.38-TODO.md name to avoid merge conflicts with master branch and allow CI tests to run. will be renamed to v0.38.md once config migration is added to the PR * fix: provideKeysRec go routine * clear keystore on close * fix: datastore prefix * fix: improve error handling in provideKeysRec - close errCh channel to distinguish between nil and pending errors - check for pending errors when provided.New closes - handle context cancellation during error send - prevent race condition where errors could be silently lost this ensures DAG walk errors are always propagated correctly * address gammazero's review * rename BurstProvider to LegacyProvider * use latest provider/keystore * boxo: make mfs StartProviding async * bump boxo * chore: update boxo to f2b4e12fb9a8ac138ccb82aae3b51ec51d9f631c - updated boxo dependency to specified commit - updated go.mod and go.sum files across all modules * use latest kad-dht/boxo * Buffered SweepingProvider wrapper * use latest kad-dht commit * allow no DHT router * use latest kad-dht & boxo --------- Co-authored-by: Marcin Rataj <lidel@lidel.org> Co-authored-by: gammazero <11790789+gammazero@users.noreply.github.com>	2025-09-18 17:22:04 +02:00
Marcin Rataj	20f8151628	fix: enforce identity CID size limits (#10949) ... * fix: enforce identity CID size limits - validate --inline-limit against verifcid.MaxDigestSize - add error when --hash=identity exceeds size limit - add tests for identity CID overflow scenarios - update help text to show maximum inline limit This prevents creation of unbounded identity CIDs by enforcing the 128-byte limit defined in https://github.com/ipfs/boxo/pull/1018 Fixes #6011 IPIP: https://github.com/ipfs/specs/pull/512	2025-09-09 20:22:21 +02:00
Marcin Rataj	906ce802bf	feat(gateway): improved error page with retrieval state details (#10950) ... * feat(gateway): add DiagnosticServiceURL config - add Gateway.DiagnosticServiceURL to kubo config - pass diagnostic service URL to boxo gateway - document new config option in docs/config.md - default to https://check.ipfs.network * docs(changelog): add gateway error UX improvements to v0.38 - document improved 504 error pages with retrieval diagnostics - highlight new Gateway.DiagnosticServiceURL config option - include screenshot showing enhanced error page UX	2025-09-08 19:49:00 +02:00
Andrew Gillis	c468f44fc1	upgrade to go-test v0.2.3 (#10923)	2025-08-19 11:13:03 -07:00
Andrew Gillis	7109534464	Upgrade to Boxo v0.34.0 (#10917) ... * Use ValidatingBlockstore when HashOnRead is configured * fix ValidatingBlockstore creation * upgrade to boxo v0.34.0 * Use tagged boxo release	2025-08-15 14:43:26 -07:00
Marcin Rataj	64c47c374a	feat(config): Gateway.RetrievalTimeout|MaxConcurrentRequests (#10905) ... * feat(gateway): concurrency and timeout limits Depends on https://github.com/ipfs/boxo/pull/994 * chore: boxo master with final boxo#994 this includes race-condition fixes from ipfs/boxo#994 and increased `DefaultMaxConcurrentRequests = 4096` * docs: concise config.md and changelog	2025-08-15 02:36:33 +02:00
Andrew Gillis	501b1f8bc3	feat: require go1.25 for building kubo (#10913) ... * Require go1.25 for building kubo * update Dockerfile to fix warnings	2025-08-13 18:01:50 -07:00