300 Commits

Author	SHA1	Message	Date
Marcin Rataj	d405dfd1ae	docs: loud deprecation of badger v1 datastore (#11187) ... * docs: loud deprecation of badger v1 datastore badger v1 (go-ds-badger) has not been maintained by its upstream maintainers for years and has known bugs (startup timeouts, shutdown hangs, fd exhaustion). make the deprecation loud and unavoidable: - print ERROR log line and bordered stderr message on every daemon start when a badger-based datastore is detected - mark badgerds and badgerds-measure init profiles as DEPRECATED - update docs/datastores.md and docs/config.md with migration guidance - add changelog highlight for v0.40 badger v1 support will be removed later in 2026. part of https://github.com/ipfs/kubo/issues/11186 * docs(changelog): link badger deprecation to #11186	2026-02-09 22:00:17 +01:00
Marcin Rataj	c6702eaf88	fix: allow dag import of 1MiB chunks wrapped in dag-pb (#11185) ... IPIP-499's unixfs-v1-2025 profile uses 1MiB chunks. with --raw-leaves=false, protobuf wrapping pushes blocks slightly over 1MiB. the previous 1MiB SoftBlockLimit rejected these blocks on dag import. raise SoftBlockLimit to 2MiB to match the bitswap spec, which requires implementations to support blocks up to 2MiB. - raise SoftBlockLimit to 2MiB per the bitswap spec - update error messages and help text - bump boxo to main with ipfs/boxo#1101 (raised ChunkSizeLimit/BlockSizeLimit, 256-byte overhead budget) - update sharness tests for 2MiB boundary - add test/cli boundary tests for block put, dag put, dag import, ipfs add (raw and wrapped leaves), and bitswap exchange including regression tests for the libp2p message size hard limit	2026-02-06 23:55:40 +01:00
Marcin Rataj	3a6b1ee122	feat(gateway): IPIP-0524 Gateway.AllowCodecConversion config option (#11090) ... * feat(gateway): IPIP-0524 Gateway.AllowCodecConversion config option Wire up boxo's AllowCodecConversion config to control codec conversion behavior per IPIP-0524. When false (default), the gateway returns 406 Not Acceptable if the requested format doesn't match the block's codec. Clients should fetch raw blocks (`?format=raw`) and convert client-side. Ref: https://github.com/ipfs/specs/pull/524 Ref: https://github.com/ipfs/boxo/pull/1077 Ref: https://github.com/ipfs/gateway-conformance/pull/254 * chore: update boxo for improved 406 codec conversion error boxo now returns an actionable hint when codec conversion is rejected: suggests fetching raw block with ?format=raw and converting client-side. * chore: bump boxo and gateway-conformance to v0.10 * docs: add IPLD Logical Format note to AllowCodecConversion	2026-02-06 02:21:35 +01:00
Marcin Rataj	67c89bbd7e	feat(config): add Import.* for CID Profiles from IPIP-499 (#11148) ... * feat(config): Import.* and unixfs-v1-2025 profile implements IPIP-499: add config options for controlling UnixFS DAG determinism and introduces `unixfs-v1-2025` and `unixfs-v0-2015` profiles for cross-implementation CID reproducibility. changes: - add Import.* fields: HAMTDirectorySizeEstimation, SymlinkMode, DAGLayout, IncludeEmptyDirectories, IncludeHidden - add validation for all Import.* config values - add unixfs-v1-2025 profile (recommended for new data) - add unixfs-v0-2015 profile (alias: legacy-cid-v0) - remove deprecated test-cid-v1 and test-cid-v1-wide profiles - wire Import.HAMTSizeEstimationMode() to boxo globals - update go.mod to use boxo with SizeEstimationMode support ref: https://specs.ipfs.tech/ipips/ipip-0499/ * feat(add): add --dereference-symlinks, --empty-dirs, --hidden CLI flags add CLI flags for controlling file collection behavior during ipfs add: - `--dereference-symlinks`: recursively resolve symlinks to their target content (replaces deprecated --dereference-args which only worked on CLI arguments). wired through go-ipfs-cmds to boxo's SerialFileOptions. - `--empty-dirs` / `-E`: include empty directories (default: true) - `--hidden` / `-H`: include hidden files (default: false) these flags are CLI-only and not wired to Import.* config options because go-ipfs-cmds library handles input file filtering before the directory tree is passed to kubo. removed unused Import.UnixFSSymlinkMode config option that was defined but never actually read by the CLI. also: - wire --trickle to Import.UnixFSDAGLayout config default - update go-ipfs-cmds to v0.15.1-0.20260117043932-17687e216294 - add SYMLINK HANDLING section to ipfs add help text - add CLI tests for all three flags ref: https://github.com/ipfs/specs/pull/499 * test(add): add CID profile tests and wire SizeEstimationMode add comprehensive test suite for UnixFS CID determinism per IPIP-499: - verify exact HAMT threshold boundary for both estimation modes: - v0-2015 (links): sum(name_len + cid_len) == 262144 - v1-2025 (block): serialized block size == 262144 - verify HAMT triggers at threshold + 1 byte for both profiles - add all deterministic CIDs for cross-implementation testing also wires SizeEstimationMode through CLI/API, allowing Import.UnixFSHAMTSizeEstimation config to take effect. bumps boxo to ipfs/boxo@6707376 which aligns HAMT threshold with JS implementation (uses > instead of >=), fixing CID determinism at the exact 256 KiB boundary. * feat(add): --dereference-symlinks now resolves all symlinks Previously, resolving symlinks required two flags: - --dereference-args: resolved symlinks passed as CLI arguments - --dereference-symlinks: resolved symlinks inside directories Now --dereference-symlinks handles both cases. Users only need one flag to fully dereference symlinks when adding files to IPFS. The deprecated --dereference-args still works for backwards compatibility but is no longer necessary. * chore: update boxo and improve changelog - update boxo to ebdaf07c (nil filter fix, thread-safety docs) - simplify changelog for IPIP-499 section - shorten test names, move context to comments * chore: update boxo to 5cf22196 * chore: apply suggestions from code review Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com> * test(add): verify balanced DAG layout produces uniform leaf depth add test that confirms kubo uses balanced layout (all leaves at same depth) rather than balanced-packed (varying depths). creates 45MiB file to trigger multi-level DAG and walks it to verify leaf depth uniformity. includes trickle subtest to validate test logic can detect varying depths. supports CAR export via DAG_LAYOUT_CAR_OUTPUT env var for test vectors. * chore(deps): update boxo to 6141039ad8ef switches to 6141039ad8 changes since 5cf22196ad0b: - refactor(unixfs): use arithmetic for exact block size calculation - refactor(unixfs): unify size tracking and make SizeEstimationMode immutable - feat(unixfs): optimize SizeEstimationBlock and add mode/mtime tests also clarifies that directory sharding globals affect both `ipfs add` and MFS. * test(cli): improve HAMT threshold tests with exact +1 byte verification - add UnixFSDataType() helper to directly check UnixFS type via protobuf - refactor threshold tests to use exact +1 byte calculations instead of +1 file - verify directory type directly (ft.TDirectory vs ft.THAMTShard) instead of inferring from link count - clean up helper function signatures by removing unused cidLength parameter * test(cli): consolidate profile tests into cid_profiles_test.go remove duplicate profile threshold tests from add_test.go since they are fully covered by the data-driven tests in cid_profiles_test.go. changes: - improve test names to describe what threshold is being tested - add inline documentation explaining each test's purpose - add byte-precise helper IPFSAddDeterministicBytes for threshold tests - remove ~200 lines of duplicated test code from add_test.go - keep non-profile tests (pinning, symlinks, hidden files) in add_test.go * chore: update to rebased boxo and go-ipfs-cmds PRs * docs: add HAMT threshold fix details to changelog * feat(mfs): use Import config for CID version and hash function make MFS commands (files cp, files write, files mkdir, files chcid) respect Import.CidVersion and Import.HashFunction config settings when CLI options are not explicitly provided. also add tests for: - files write respects Import.UnixFSRawLeaves=true - single-block file: files write produces same CID as ipfs add - updated comments clarifying CID parity with ipfs add * feat(files): wire Import.UnixFSChunker and UnixFSDirectoryMaxLinks to MFS `ipfs files` commands now respect these Import.* config options: - UnixFSChunker: configures chunk size for `files write` - UnixFSDirectoryMaxLinks: triggers HAMT sharding in `files mkdir` - UnixFSHAMTDirectorySizeEstimation: controls size estimation mode previously, MFS used hardcoded defaults ignoring user config. changes: - config/import.go: add UnixFSSplitterFunc() returning chunk.SplitterGen - core/node/core.go: pass chunker, maxLinks, sizeEstimationMode to mfs.NewRoot() via new boxo RootOption API - core/commands/files.go: pass maxLinks and sizeEstimationMode to mfs.Mkdir() and ensureContainingDirectoryExists(); document that UnixFSFileMaxLinks doesn't apply to files write (trickle DAG limitation) - test/cli/files_test.go: add tests for UnixFSDirectoryMaxLinks and UnixFSChunker, including CID parity test with `ipfs add --trickle` related: boxo@54e044f1b265 * feat(files): wire Import.UnixFSHAMTDirectoryMaxFanout and UnixFSHAMTDirectorySizeThreshold wire remaining HAMT config options to MFS root: - Import.UnixFSHAMTDirectoryMaxFanout via mfs.WithMaxHAMTFanout - Import.UnixFSHAMTDirectorySizeThreshold via mfs.WithHAMTShardingSize add CLI tests: - files mkdir respects Import.UnixFSHAMTDirectoryMaxFanout - files mkdir respects Import.UnixFSHAMTDirectorySizeThreshold - config change takes effect after daemon restart add UnixFSHAMTFanout() helper to test harness update boxo to ac97424d99ab90e097fc7c36f285988b596b6f05 * fix(mfs): single-block files in CIDv1 dirs now produce raw CIDs problem: `ipfs files write` in CIDv1 directories wrapped single-block files in dag-pb even when raw-leaves was enabled, producing different CIDs than `ipfs add --raw-leaves` for the same content. fix: boxo now collapses single-block ProtoNode wrappers (with no metadata) to RawNode in DagModifier.GetNode(). files with mtime/mode stay as dag-pb since raw blocks cannot store UnixFS metadata. also fixes sparse file writes where writing past EOF would lose data because expandSparse didn't update the internal node pointer. updates boxo to v0.36.1-0.20260203003133-7884ae23aaff updates t0250-files-api.sh test hashes to match new behavior * chore(test): use Go 1.22+ range-over-int syntax * chore: update boxo to c6829fe26860 - fix typo in files write help text - update boxo with CI fixes (gofumpt, race condition in test) * chore: update go-ipfs-cmds to 192ec9d15c1f includes binary content types fix: gzip, zip, vnd.ipld.car, vnd.ipld.raw, vnd.ipfs.ipns-record * chore: update boxo to 0a22cde9225c includes refactor of maxLinks check in addLinkChild (review feedback). * ci: fix helia-interop and improve caching skip '@helia/mfs - should have the same CID after creating a file' test until helia implements IPIP-499 (tracking: https://github.com/ipfs/helia/issues/941) the test fails because kubo now collapses single-block files to raw CIDs while helia explicitly uses reduceSingleLeafToSelf: false changes: - run aegir directly instead of helia-interop binary (binary ignores --grep flags) - cache node_modules keyed by @helia/interop version from npm registry - skip npm install on cache hit (matches ipfs-webui caching pattern) * chore: update boxo to 1e30b954 includes latest upstream changes from boxo main * chore: update go-ipfs-cmds to 1b2a641ed6f6 * chore: update boxo to f188f79fd412 switches to boxo@main after merging https://github.com/ipfs/boxo/pull/1088 * chore: update go-ipfs-cmds to af9bcbaf5709 switches to go-ipfs-cmds@master after merging https://github.com/ipfs/go-ipfs-cmds/pull/315 --------- Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com>	2026-02-04 22:03:15 +01:00
Marcin Rataj	ff4bb10989	chore: replace libp2p.io URL with Internet Archive (#11181) ... swap to last working snapshots until libp2p project fixes their websites Right now its either broken due to Fleek shutting down: https://web.archive.org/web/20260201232813/https://docs.libp2p.io/concepts/pubsub/overview/ or URLs changed by being moved to new place at: https://web.archive.org/web/20260204171903/https://libp2p.io/guides/circuit-relay/ we can undo/switch to real URLs once things stabilize for a few months	2026-02-04 18:50:58 +01:00
Marcin Rataj	7de7af0820	feat(dns): skip DNS lookups for AutoTLS hostnames (#11140) ... * feat(dns): resolve libp2p.direct addresses locally without network I/O p2p-forge hostnames encode IP addresses directly (e.g., 1-2-3-4.peerID.libp2p.direct -> 1.2.3.4), so DNS queries are wasteful. kubo now parses these IPs in-memory. - applies to both default libp2p.direct and custom AutoTLS.DomainSuffix - TXT queries still delegate to network for ACME DNS-01 compatibility - https://github.com/ipfs/kubo/pull/11140#discussion_r2683477754 use fallback to network DNS instead of returning errors when local parsing fails, ensuring forward compatibility with future DNS records - https://github.com/ipfs/kubo/pull/11140#discussion_r2683512408 add peerID validation using peer.Decode(), matching libp2p.direct server behavior, with fallback on invalid peerID - https://github.com/ipfs/kubo/pull/11140#discussion_r2683521930 document interaction with DNS.Resolvers in config.md - https://github.com/ipfs/kubo/pull/11140#discussion_r2683526647 add AutoTLS.SkipDNSLookup config flag to disable local resolution (useful for debugging or custom DNS override scenarios) - https://github.com/ipfs/kubo/pull/11140#discussion_r2683533462 add E2E test verifying libp2p.direct resolves locally even when DNS.Resolvers points to a broken server additional improvements: - use madns.BasicResolver interface instead of custom basicResolver - add compile-time interface checks for p2pForgeResolver and madns.Resolver - refactor tests: merge IPv4/IPv6, add helpers, use config.DefaultDomainSuffix - improve changelog to explain public good benefit (reducing DNS load) Fixes #11136	2026-01-30 17:20:56 +01:00
Marcin Rataj	824a47ae11	feat(pubsub): persistent validation and diagnostic commands (#11110) ... * feat(pubsub): persistent seqno validation and diagnostic commands - upgrade go-libp2p-pubsub to v0.15.0 - add persistent seqno validator using BasicSeqnoValidator stores max seen seqno per peer at /pubsub/seqno/<peerid> survives daemon restarts, addresses message cycling in large networks (#9665) - add `ipfs pubsub reset` command to clear validator state - add `ipfs diag datastore get/count` commands for datastore inspection requires daemon to be stopped, useful for debugging - change pubsub status from Deprecated to Experimental - add CLI tests for pubsub and diag datastore commands - remove flaky pubsub_msg_seen_cache_test.go (replaced by CLI tests) * fix(pubsub): improve reset command and add deprecation warnings - use batched delete for efficient bulk reset - check key existence before reporting deleted count - sync datastore after deletions to ensure persistence - show "no validator state found" when resetting non-existent peer - log deprecation warnings when using --enable-pubsub-experiment or --enable-namesys-pubsub CLI flags * refactor(test): add datastore helpers to test harness --------- Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com>	2026-01-16 00:27:09 +01:00
Marcin Rataj	edb7056747	feat(config): add Gateway.MaxRequestDuration option (#11138) ... * feat(config): add Gateway.MaxRequestDuration option exposes the previously hardcoded 1 hour gateway request deadline as a configurable option, allowing operators to adjust it to fit deployment needs. protects gateway from edge cases and slow client attacks. boxo: https://github.com/ipfs/boxo/pull/1079 * test(gateway): add MaxRequestDuration integration test verifies config is wired correctly and 504 is returned when exceeded * docs: add MaxRequestDuration to gateway production guide --------- Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com>	2026-01-15 19:31:16 +01:00
Marcin Rataj	447109df64	docs: clarify Routing.Type=custom as experimental (#11111) ... * docs: mark custom routing as experimental reorganize Routing.Type section for clarity, group production and experimental options, consolidate DHT explanation, add limitations section to delegated-routing.md documenting that HTTP-only routing cannot provide content reliably * chore(config): reorder Routing sections and improve callout formatting move DelegatedRouters after Type, add config option names to CAUTION headers * docs: address reviewer feedback on config.md - clarify that `auto` can be combined with custom URLs in `Routing.DelegatedRouters` - rename headers for consistency: `Routing.Routers.[name].Type`, `Routing.Routers.[name].Parameters`, `Routing.Methods` - replace deprecated Strategic Providing reference with `Provide.*` config - remove outdated caveat about 0.39 sweep limitation - wording: "likely suffer" → "will be most affected" * docs: remove redundant Summary section from delegated-routing.md the IMPORTANT callout and Motivation section already cover what users need to know. historical version info was noise for researchers trying to configure custom routing. addresses reviewer feedback from #11111. --------- Co-authored-by: Daniel Norman <2color@users.noreply.github.com> Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com>	2026-01-11 00:39:30 +01:00
Marcin Rataj	25ebab9dae	feat(p2p): add --foreground flag to listen and forward commands (#11099) ... * feat(p2p): add --foreground flag to listen and forward commands adds `-f/--foreground` option that keeps the command running until interrupted (SIGTERM/Ctrl+C) or closed via `ipfs p2p close`. the listener/forwarder is automatically removed when the command exits. useful for systemd services and scripts that need cleanup on exit. * docs: add p2p-tunnels.md with systemd examples - add dedicated docs/p2p-tunnels.md covering: - why p2p tunnels (NAT traversal, no public IP needed) - quick start with netcat - background and foreground modes - systemd integration with path-based activation - security considerations and troubleshooting - document Experimental.Libp2pStreamMounting in docs/config.md - simplify docs/experimental-features.md, link to new doc - add "Learn more" links to ipfs p2p listen/forward --help - update changelog entry with doc link - add cross-reference in misc/README.md * chore: reference kubo#5460 for p2p config Ref. https://github.com/ipfs/kubo/issues/5460 * fix(daemon): write api/gateway files only after HTTP server is ready fixes race condition where $IPFS_PATH/api and $IPFS_PATH/gateway files were written before the HTTP servers were ready to accept connections. this caused issues for tools like systemd path units that immediately try to connect when these files appear. changes: - add corehttp.ServeWithReady() that signals when server is ready - wait for ready signal before writing address files - use sync.WaitGroup.Go() (Go 1.25) for cleaner goroutine management - add TestAddressFileReady to verify both api and gateway files * fix(daemon): buffer errc channel and wait for all listeners - buffer error channel with len(listeners) to prevent deadlock when multiple servers write errors simultaneously - wait for ALL listeners to be ready before writing api/gateway file, not just the first one Feedback-from: https://github.com/ipfs/kubo/pull/11099#pullrequestreview-3593885839 * docs(changelog): improve p2p tunnel section clarity reframe to lead with user benefit and add example output * docs(p2p): remove obsolete race condition caveat the "First launch fails but restarts work" troubleshooting section described a race where the api file was written before the daemon was ready. this was fixed in 80b703a which ensures api/gateway files are only written after HTTP servers are ready to accept connections. --------- Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com>	2026-01-09 19:22:43 +01:00
Marcin Rataj	3e85793b58	docs: add production deployment guidance for gateway (#11117) ... - add "Running in Production" section to gateway.md - link to specs.ipfs.tech instead of github - update "Protocol Labs" to "IPFS Foundation" - add reverse proxy requirement to config.md PublicGateways section - reference reverse proxy, timeout, rate limiting, and CDN caveats	2026-01-09 18:26:32 +01:00
Marcin Rataj	23ba660ef0	fix(routing): use LegacyProvider for HTTP-only custom routing (#11112) ... * fix(routing): use LegacyProvider for HTTP-only custom routing when `Routing.Type=custom` with only HTTP routers and no DHT, fall back to LegacyProvider instead of SweepingProvider. SweepingProvider requires a DHT client which is unavailable in HTTP-only configurations, causing it to return NoopProvider and breaking provider record announcements to HTTP routers. fixes #11089 * test(routing): verify provide stat works with HTTP-only routing * docs(config): clarify SweepEnabled fallback for HTTP-only routing --------- Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com>	2026-01-09 18:23:31 +01:00
Marcin Rataj	c1e1cfebbb	docs: link sweep blogpost in Provide.DHT.SweepEnabled ... add link to Shipyard's "Provide Sweep: Solving the DHT Provide Bottleneck" blogpost and remove stale TODO placeholder	2025-11-28 23:19:03 +01:00
Marcin Rataj	af2e7e1953	docs: note sweep+accelerated DHT client limitation (#11084) ... document known 0.39 limitation where sweep provider may fail to estimate DHT size when accelerated client is still crawling the network, resulting in single-region mode without efficiency gains. also remove accelerated client recommendation from changelog since it may mislead users into enabling both together.	2025-11-28 23:10:56 +01:00
Marcin Rataj	823c11721d	Merge branch 'master' into merge-release-v0.39.0 ... resolve version.go conflict by keeping -dev version from master	2025-11-27 04:53:20 +01:00
Marcin Rataj	40ffd166ea	docs: restructure v0.39 changelog for clarity ... - rewrite overview to lead with user value (self-hosting on consumer hardware) - reorder highlights: provider features together, then UPnP, then housekeeping - simplify titles (drop "Amino", "Fixed", verbose descriptions) - link to Shipyard's sweep provider blogpost	2025-11-27 00:12:56 +01:00
Hector Sanjuan	73ab037d1d	feat: support GetClosesPeers (IPIP-476) and ExposeRoutingAPI by default (#10954) ... This allows Kubo to respond to the GetClosestPeers() http routing v1 endpoint as spec'ed here: https://github.com/ipfs/specs/pull/476 It is based on work from https://github.com/ipfs/boxo/pull/1021 We let IpfsNode implmement the contentRouter.Client interface with the new method. We use our WAN-DHT to get the closest peers. Additionally, Routing V1 HTTP API is exposed by default which enables light clients in browsers to use Kubo Gateway as delegated routing backend Co-authored-by: Marcin Rataj <lidel@lidel.org>	2025-11-19 10:51:56 +00:00
Marcin Rataj	cec7432043	feat: fast provide support in dag import (#11058) ... * fix(add): respect Provide config in fast-provide-root fast-provide-root should honor the same config settings as the regular provide system: - skip when Provide.Enabled is false - skip when Provide.DHT.Interval is 0 - respect Provide.Strategy (all/pinned/roots/mfs/combinations) This ensures fast-provide only runs when appropriate based on user configuration and the nature of the content being added (pinned vs unpinned, added to MFS or not). * feat(config): options to adjust global defaults Add Import.FastProvideRoot and Import.FastProvideWait configuration options to control default behavior of fast-provide-root and fast-provide-wait flags in ipfs add command. Users can now set global defaults in config while maintaining per-command flag overrides. - Add Import.FastProvideRoot (default: true) - Add Import.FastProvideWait (default: false) - Add ResolveBoolFromConfig helper for config resolution - Update docs with configuration details - Add log-based tests verifying actual behavior * refactor: extract fast-provide logic into reusable functions Extract fast-provide logic from add command into reusable components: - Add config.ShouldProvideForStrategy helper for strategy matching - Add ExecuteFastProvide function reusable across add and dag import commands - Move DefaultFastProvideTimeout constant to config/provide.go - Simplify add.go from 72 lines to 6 lines for fast-provide - Move fast-provide tests to dedicated TestAddFastProvide function Benefits: - cleaner API: callers only pass content characteristics - all strategy logic centralized in one place - better separation of concerns - easier to add fast-provide to other commands in future * feat(dag): add fast-provide support for dag import Adds --fast-provide-root and --fast-provide-wait flags to `ipfs dag import`, mirroring the fast-provide functionality available in `ipfs add`. Changes: - Add --fast-provide-root and --fast-provide-wait flags to dag import command - Implement fast-provide logic for all root CIDs in imported CAR files - Works even when --pin-roots=false (strategy checked internally) - Share ExecuteFastProvide implementation between add and dag import - Move ExecuteFastProvide to cmdenv package to avoid import cycles - Add logging when fast-provide is disabled - Conditional error handling: return error when wait=true, warn when wait=false - Update config docs to mention both ipfs add and ipfs dag import - Update changelog to use "provide" terminology and include dag import examples - Add comprehensive test coverage (TestDagImportFastProvide with 6 test cases) The fast-provide feature allows immediate DHT announcement of root CIDs for faster content discovery, bypassing the regular background queue. * docs: improve fast-provide documentation Refine documentation to better explain fast-provide and sweep provider working together, and highlight the performance improvement. Changelog: - add fast-provide to sweep provider features list - explain performance improvement: root CIDs discoverable in <1s vs 30+ seconds - note this uses optimistic DHT operations (faster with sweep provider) - simplify examples, point to --help for details Config docs: - fix: --fast-provide-roots should be --fast-provide-root (singular) - clarify Import.FastProvideRoot focuses on root CIDs while sweep handles all blocks - simplify Import.FastProvideWait description Command help: - ipfs add: explain sweep provider context upfront - ipfs dag import: add fast-provide explanation section - both explain the split: fast-provide for roots, sweep for all blocks * test: add tests for ShouldProvideForStrategy add tests covering all provide strategy combinations with focus on bitflag OR logic (the else-if bug fix). organized by behavior: - all strategy always provides - single strategies match only their flag - combined strategies use OR logic - zero strategy never provides * refactor: error cmd on error and wait=true change ExecuteFastProvide() to return error, enabling proper error propagation when --fast-provide-wait=true. in sync mode, provide failures now error the command as expected. in async mode (default), always returns nil with errors logged in background goroutine. also remove duplicate ExecuteFastProvide() from provide.go (75 lines), keeping single implementation in cmdenv/env.go for reuse across add and dag import commands. call sites simplified: - add.go: check and propagate error from ExecuteFastProvide - dag/import.go: return error from ForEach callback, remove confusing conditional error handling semantics: - precondition skips (DHT unavailable, etc): return nil (not failure) - async mode (wait=false): return nil, log errors in goroutine - sync mode (wait=true): return wrapped error on provide failure	2025-11-14 21:06:25 -08:00
Guillaume Michel	702c63b6db	feat: enable DHT Provide Sweep by default (#10955) ... Co-authored-by: Marcin Rataj <lidel@lidel.org> Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com>	2025-11-12 23:55:17 +01:00
Marcin Rataj	93f8897d7c	feat(config): optional Gateway.MaxRangeRequestFileSize (#10997) ... adds Gateway.MaxRangeRequestFileSize configuration to protect against CDN bugs where range requests over certain sizes return entire files instead of requested byte ranges, causing unexpected bandwidth costs. - default: 0 (no limit) - returns 501 Not Implemented for oversized range requests - protects against CDNs like Cloudflare that ignore range requests over 5GiB also introduces OptionalBytes type to reduce code duplication when handling byte-size configuration values, replacing manual string parsing with humanize.ParseBytes. migrates existing byte-size configs to use this new type. Fixes: https://github.com/ipfs/boxo/issues/856	2025-11-11 18:54:43 -08:00
Guillaume Michel	c2bf0f9515	feat(provider): resume cycle (#11031) ... * bump kad-dht: resume reprovide cycle * daemon: --provide-fresh-start flag * changelog * docs * go-fmt * chore: latest go-libp2p-kad-dht#1170 after conflict resolution, to confirm CI is still green * kad-dht: depend on latest master * move daemon flag to Provider.DHT.ResumeEnabled config * refactor: sweep provider datastore * bump kad-dht * bump kad-dht * bump kad-dht * make datastore keys constant * use kad-dht master * add emoji to changelog entry * go-fmt * bump kad-dht * test(provider): add tests for resume cycle feature validates Provide.DHT.ResumeEnabled behavior: - preserves cycle state when enabled (default) - resets cycle when disabled tests verify current_time_offset across restarts using JSON output --------- Co-authored-by: Marcin Rataj <lidel@lidel.org>	2025-10-29 11:07:46 +01:00
Marcin Rataj	ae78c7821c	fix: go-libp2p v0.44 with self-healing UPnP port mappings (#11032) ... * chore(deps): update go-libp2p to v0.44.0 - includes self-healing UPnP port mappings after router restarts - update go-netroute to v0.3.0 - update quic-go to v0.55.0 - add changelog entry for UPnP fix * docs: improve provide and UPnP clarity in changelog and docs - add alert polling rationale to changelog - add UPnP config note with default clarification - clarify sweep timing and prefix length explanations - add concrete examples for time offset and record holders - improve workers stats formatting - add See Also section to provide-stats.md * docs: add RISC-V prebuilt binaries to changelog and README - highlight linux-riscv64 availability with open hardware context - update README with arm64 builds, remove 32-bit examples	2025-10-25 01:13:34 +02:00
Guillaume Michel	16479ec692	feat(provide): detailed ipfs provide stat (#11019) ... * feat: provide stats * added N/A * format * workers stats alignment * ipfs provide stat --all --compact * consolidating compact stat * update column alignment * flags combinations errors * command description * change schedule AvgPrefixLen to float * changelog * alignments * provide stat description draft * rephrased provide-stats.md * linking provide-stats.md from command description * documentation test * fix: refactor provide stat command type handling - add extractSweepingProvider() helper to reduce nested type switching - extract lowWorkerThreshold constant for worker availability check - fix --lan error handling to work with buffered providers * docs: add clarifying comments * fix(commands): improve provide stat compact mode - prevent panic when both columns are empty - fix column alignment with UTF-8 characters - only track col0MaxWidth for first column (as intended) * test: add tests for ipfs provide stat command - test basic functionality, flags, JSON output - test legacy provider behavior - test integration with content scheduling - test disabled provider configurations - add parseSweepStats helper with t.Helper() * docs: improve provide command help text - update tagline to "Control and monitor content providing" - simplify help descriptions - make error messages more consistent - update tests to match new error messages * metrics rename ``` Next reprovide at: Next prefix: ``` updated to: ``` Next region prefix: Next region reprovide: ``` * docs: improve Provide system documentation clarity Enhance documentation for the Provide system to better explain how provider records work and the differences between sweep and legacy modes. Changes to docs/config.md: - Provide section: add clear explanation of provider records and their role - Provide.DHT: add provider record lifecycle and two provider systems overview - Provide.DHT.Interval: explain relationship to expiration, contrast sweep vs legacy behavior - Provide.DHT.SweepEnabled: rewrite to explain legacy problem, sweep solution, and efficiency gains - Monitoring section: prioritize command-line tools (ipfs provide stat) before Prometheus Changes to core/commands/provide.go: - ipfs provide stat help: add explanation of provider records, TTL expiration, and how sweep batching works Changes to docs/changelogs/v0.39.md: - Add context about why stats matter for monitoring provider health - Emphasize real-time monitoring workflow with watch command - Explain what users can observe (rates, queues, worker availability) * depend on latest kad-dht master * docs: nits --------- Co-authored-by: Marcin Rataj <lidel@lidel.org>	2025-10-23 20:29:36 +02:00
Guillaume Michel	241b723534	config: increase default Provide.DHT.MaxProvideConnsPerWorker (#11016) ... Increase default Provide.DHT.MaxProvideConnsPerWorker value to match the DHT replication factor (16 -> 20). A similar value is used in legacy systems (with and without accelerated DHT client).	2025-10-14 17:10:28 +02:00
Guillaume Michel	cf8194a8d1	docs: add sweeping provide worker count recommendation (#11001) ... Add recommentation for worker count for the sweeping provide system for users with millions of CIDs.	2025-10-01 16:01:27 +02:00
Guillaume Michel	776c21a6d6	fix: provider merge conflicts (#10989) ... Co-authored-by: Marcin Rataj <lidel@lidel.org>	2025-09-26 03:22:37 +02:00
Marcin Rataj	a688b7eeac	fix(mfs): add soft limit for --flush=false (#10985) ... * fix: add MFS operation limit for --flush=false adds a global counter that tracks consecutive MFS operations performed with --flush=false and fails with clear error after limit is reached. this prevents unbounded memory growth while avoiding the data corruption risks of auto-flushing. - adds Internal.MFSNoFlushLimit config - operations fail with actionable error at limit - counter resets on successful flush or any --flush=true operation - operations with --flush=true reset and don't count this commit removes automatic flush from https://github.com/ipfs/kubo/pull/10971 and instead errors to encourage users of --flush=false to develop a habit of calling 'ipfs files flush' periodically. boxo will no longer auto-flush (https://github.com/ipfs/boxo/pull/1041) to avoid corruption issues, and kubo applies the limit to 'ipfs files' commands instead. closes #10842 * test: add tests for MFSNoFlushLimit tests verify the new Internal.MFSNoFlushLimit config option: - default limit of 256 operations - custom limit configuration - counter reset on flush=true - counter reset on explicit flush command - limit=0 disables the feature - multiple MFS command types count towards limit * docs: explain why MFS operations fail instead of auto-flushing addresses feedback from https://github.com/ipfs/kubo/pull/10985#pullrequestreview-3256250970 - clarify that automatic flushing at limit was considered but rejected - explain the data corruption risks of auto-flushing - guide users who want auto-flush to use --flush=true (default) - document benefits of explicit failure for batch operations	2025-09-26 01:25:23 +02:00
Marcin Rataj	07f017f01d	docs(provide): validation and reprovide cycle visualization (#10977) ... * docs: improve slow reprovide warning messages simplify warning text and provide actionable solutions in order of preference * feat(config): add validation for Provide.DHT settings - validate interval doesn't exceed DHT record validity (48h) - validate worker counts and other parameters are within valid ranges - improve slow reprovide warning messages to reference config parameter - add tests for all validation cases * docs: add reprovide cycle visualization shows traffic patterns of legacy vs sweep vs accelerated DHT	2025-09-19 18:47:30 +02:00
Marcin Rataj	fa17b69c7d	fix(mfs): unbound cache growth with flush=false (#10971) ... * fix: prevent --flush=false in 'ipfs files rm' command the 'ipfs files rm' command always flushes for safety to ensure data integrity. this change adds an explicit error when users try to pass --flush=false, improving ux and preventing confusion. related to #10842 * fix: add MFS cache size limit to prevent unbounded growth - add Internal.MFSAutoflushThreshold config (experimental) - directories auto-flush when cache exceeds threshold with --flush=false - prevents high memory usage issue from #10842 - default: 256 entries per directory (matching HAMT shard size) - set to 0 to restore old behavior (risky, may cause errors) Closes #10842	2025-09-19 03:39:55 +02:00
Marcin Rataj	71e883440e	refactor(config): migration 17-to-18 to unify Provider/Reprovider into Provide.DHT (#10951) ... * refactor: consolidate Provider/Reprovider into unified Provide config - merge Provider and Reprovider configs into single Provide section - add fs-repo-17-to-18 migration for config consolidation - improve migration ergonomics with common package utilities - convert deprecated "flat" strategy to "all" during migration - improve Provide docs * docs: add total_provide_count metric guidance - document how to monitor provide success rates via prometheus metrics - add performance comparison section to changelog - explain how to evaluate sweep vs legacy provider effectiveness * fix: add OpenTelemetry meter provider for metrics - set up meter provider with Prometheus exporter in daemon - enables metrics from external libs like go-libp2p-kad-dht - fixes missing total_provide_count_total when SweepEnabled=true - update docs to reflect actual metric names --------- Co-authored-by: gammazero <11790789+gammazero@users.noreply.github.com> Co-authored-by: guillaumemichel <guillaume@michel.id> Co-authored-by: Daniel Norman <1992255+2color@users.noreply.github.com> Co-authored-by: Hector Sanjuan <code@hector.link>	2025-09-18 22:17:43 +02:00
Guillaume Michel	006f9dc704	feat: opt-in new Sweep provide system (#10834) ... * reprovide sweep draft * update reprovider dep * go mod tidy * fix provider type * change router type * dual reprovider * revert to provider.System * back to start * SweepingReprovider test * fix nil pointer deref * noop provider for nil dht * disabled initial network estimation * another iteration * suppress missing self addrs err * silence empty rt err on lan dht * comments * new attempt at integrating * reverting changes in core/node/libp2p/routing.go * removing SweepingProvider * make reprovider optional * add noop reprovider * update KeyChanFunc type alias * restore boxo KeyChanFunc * fix missing KeyChanFunc * test(sharness): PARALLEL=1 and timeout 30m running sequentially to see where timeout occurs * initialize MHStore * revert workflow debug * config * config docs * merged IpfsNode provider and reprovider * move Provider interface to from kad-dht to node * moved Provider interface from kad-dht to kubo/core/node * mod_tidy * Add Clear to Provider interface * use latest kad-dht commit * make linter happy * updated boxo provide interface * boxo PR fix * using latest kad-dht commit * use latest boxo release * fix fx * fx cyclic deps * fix merge issues * extended tests * don't provide LAN DHT * docs * restore dual dht provider * don't start provider before it is online * address linter * dual/provider fix * add delay in provider tests for dht bootstrap * add OfflineDelay parameter to config * remove increase number of workers in test * improved keystore gc process * fix: replace incorrect logger import in coreapi replaced github.com/labstack/gommon/log with the standard github.com/ipfs/go-log/v2 logger used throughout kubo. removed unused labstack dependency from go.mod files. * fix: remove duplicate WithDefault call in provider config * fix: use correct option method for burst workers * fix: improve error messages for experimental sweeping provider updated error messages to clearly indicate when commands are unavailable due to experimental sweeping provider being enabled via Reprovider.Sweep.Enabled=true * docs: remove obsolete KeyStoreGCInterval config removed from config.md as option no longer exists (removed in b540fba1a) updated keystore description to reflect gc happens at reprovide interval * docs: add TODO placeholder changelog for experimental sweeping DHT provider using v0.38-TODO.md name to avoid merge conflicts with master branch and allow CI tests to run. will be renamed to v0.38.md once config migration is added to the PR * fix: provideKeysRec go routine * clear keystore on close * fix: datastore prefix * fix: improve error handling in provideKeysRec - close errCh channel to distinguish between nil and pending errors - check for pending errors when provided.New closes - handle context cancellation during error send - prevent race condition where errors could be silently lost this ensures DAG walk errors are always propagated correctly * address gammazero's review * rename BurstProvider to LegacyProvider * use latest provider/keystore * boxo: make mfs StartProviding async * bump boxo * chore: update boxo to f2b4e12fb9a8ac138ccb82aae3b51ec51d9f631c - updated boxo dependency to specified commit - updated go.mod and go.sum files across all modules * use latest kad-dht/boxo * Buffered SweepingProvider wrapper * use latest kad-dht commit * allow no DHT router * use latest kad-dht & boxo --------- Co-authored-by: Marcin Rataj <lidel@lidel.org> Co-authored-by: gammazero <11790789+gammazero@users.noreply.github.com>	2025-09-18 17:22:04 +02:00
Marcin Rataj	3e1e7d17fb	feat(config): validate Import config at daemon startup (#10957) ... validates Import configuration fields to prevent invalid values: - CidVersion: must be 0 or 1 - UnixFSFileMaxLinks: must be positive - UnixFSDirectoryMaxLinks: must be non-negative - UnixFSHAMTDirectoryMaxFanout: power of 2, multiple of 8, ≤ 1024 - BatchMaxNodes/BatchMaxSize: must be positive - UnixFSChunker: validates format patterns - HashFunction: must be allowed by verifcid	2025-09-09 01:53:18 +02:00
Marcin Rataj	906ce802bf	feat(gateway): improved error page with retrieval state details (#10950) ... * feat(gateway): add DiagnosticServiceURL config - add Gateway.DiagnosticServiceURL to kubo config - pass diagnostic service URL to boxo gateway - document new config option in docs/config.md - default to https://check.ipfs.network * docs(changelog): add gateway error UX improvements to v0.38 - document improved 504 error pages with retrieval diagnostics - highlight new Gateway.DiagnosticServiceURL config option - include screenshot showing enhanced error page UX	2025-09-08 19:49:00 +02:00
Marcin Rataj	6971f9f557	docs(config): clarify Ipns.DelegatedPublishers auto behavior ... - explain how 'auto' resolution depends on Routing.Type - add command to inspect effective publishers - simplify publishing behavior explanation	2025-08-27 20:13:02 +02:00
Hector Sanjuan	846e219ead	feat: Reprovider.Strategy: rename "flat" to "all" (#10928) ... * Reprovider strategy: rename "flat" to "all". Value "flat" now parses to "all". Behaviour from "all" removed. Fixes #10864 which has detailed explanation. * core/node/provider.go: remove unused function mfsRootProvider It was used in the "all" strategy. * docs: improve reprovider.strategy=all changelog framing - highlight memory efficiency improvements - clarify this removes v0.28 workaround - update config.md memory requirements - fix announce-on profile typo * feat: deprecate Reprovider.Strategy=flat - add deprecation warning in daemon.go when flat strategy is detected - document that flat is deprecated in ParseReproviderStrategy comment - add explicit test case for flat -> all mapping - flat continues to work but users are warned to migrate to all --------- Co-authored-by: Marcin Rataj <lidel@lidel.org>	2025-08-21 19:02:29 +02:00
Marcin Rataj	426477ef78	docs: improve ipfs add --help (#10926) ... * fix(cmds): improve ipfs add --help clarity and organization - clarify --raw-leaves behavior with CIDv0/v1 defaults - add Import.* config references to all relevant options - replace deprecated 'ipfs object links' with 'ipfs ls' in examples - add section headers for better navigation (BASIC EXAMPLES, MFS INTEGRATION, etc.) - group related options logically (output control, CID & hashing, experimental features) - standardize experimental warnings format - add MerkleDAG docs link and clarify chunking behavior - fix incorrect Import.UnixFSHAMTThreshold reference addresses confusion from #10918 where --raw-leaves appeared to have no effect because CIDv1 automatically enables it by default * docs: fix typos in Import.* configuration documentation - fix 'chilcren' → 'children' - fix 'HAMT directory have' → 'HAMT directories have' - fix 'A HAMT is an structure' → 'A HAMT is a structure' * Update core/commands/add.go Co-authored-by: Daniel Norman <1992255+2color@users.noreply.github.com> --------- Co-authored-by: Daniel Norman <1992255+2color@users.noreply.github.com>	2025-08-21 14:57:30 +02:00
Marcin Rataj	ccb49de852	feat(config): AutoConf with "auto" placeholders (#10883) ... https://github.com/ipfs/kubo/pull/10883 https://github.com/ipshipyard/config.ipfs-mainnet.org/issues/3 --------- Co-authored-by: gammazero <gammazero@users.noreply.github.com>	2025-08-20 05:59:11 +02:00
Marcin Rataj	64c47c374a	feat(config): Gateway.RetrievalTimeout|MaxConcurrentRequests (#10905) ... * feat(gateway): concurrency and timeout limits Depends on https://github.com/ipfs/boxo/pull/994 * chore: boxo master with final boxo#994 this includes race-condition fixes from ipfs/boxo#994 and increased `DefaultMaxConcurrentRequests = 4096` * docs: concise config.md and changelog	2025-08-15 02:36:33 +02:00
Andrew Gillis	a22efea6f3	provider: clear provide queue when reprovide strategy changes (#10863) ... * provider: clear reprovide queue when reprovide strategy changes When the currently configured reprovide strategy does not match the previous strategy read from the datastore, then clear the reprovide queue and update the reprovide strategy that is stored in the datastore. Depends on https://github.com/ipfs/boxo/pull/978 Closes #10829 * Update docs/changelogs/v0.36.md Co-authored-by: Guillaume Michel <guillaumemichel@users.noreply.github.com> * update log message * update boxo * Move change log to v0.37.md * Add `provide clear` command to clear provide queue The `provide clear` command clears all items from the provide queue and prints out the number of items removed from the queue. The `quiet` option tells the command not to print output. * refactor(cmds): ipfs provide clear moving to new namespace to avoid conflicts, and also document other commands * docs: clarify Reprovider.Strategy * chore: remove undesired md link	2025-07-16 01:28:27 -07:00
Marcin Rataj	8589296c88	Merge remote-tracking branch 'origin/master' into merge-release-v0.36.0	2025-07-14 20:12:04 +02:00
Marcin Rataj	12144dbeed	chore: update links in markdown	2025-07-14 17:56:50 +02:00
Marcin Rataj	b4e6769a56	docs(config): add network exposure considerations (#10856) ... this adds Security section in effort to clarify how each port should be evaluated from the security perspective	2025-07-11 18:33:03 +02:00
Hector Sanjuan	eb6cc02c06	feat(config): connmgr: expose silence period (#10827)	2025-06-17 15:55:57 +02:00
Andrew Gillis	0cf1b22536	bitswap/client: configurable broadcast reduction (#10825) ... * Configure bitswap braodcast reduction Add new config items to `Internal.Bitswap` to allow configuration of bitswap broadcast reduction behavior. Broadcast reduction behavior is enabled by default, and uses settings that should be suitable for most installations of kubo. * update sharness metrics test * Explicit defaults for broadcast reduction configuration * Update docs/config.md	2025-06-17 04:35:21 -07:00
Marcin Rataj	b45ed8a3d7	feat: HTTP retrieval enabled by default (#10836)	2025-06-17 01:48:46 +02:00
sukun	d70f6421d4	feat: AutoTLS with AutoNATv2 client (#10835) ... * update autotls; libp2p to use autonatv2 * chore: p2p-forge@main with autonat fix https://github.com/ipshipyard/p2p-forge/pull/70 --------- Co-authored-by: Marcin Rataj <lidel@lidel.org>	2025-06-17 00:29:50 +02:00
Andrew Gillis	bb91273015	Minor wording fixes in docs (#10822)	2025-05-30 15:14:37 -07:00
Marcin Rataj	f84fb2849b	fix(fuse): ipns error handling and friendly errors (#10807) ... * fix(fusei/ux): check if paths exist, print err * fix(fuse): ipns 'could not resolve' error type changed when code got extracted to boxo, but it was not caught because of FUSE tests do not cover IPNS in online mode Closes #8095 Closes #2167 Closes #3013 * docs: clarify opt-in	2025-05-15 23:43:43 +02:00
Marcin Rataj	2ab3f58c99	fix(config): wire up Provider.Enabled flag (#10804) ... * fix(config): explicit Provider.Enabled flag Adds missing config option described in https://github.com/ipfs/kubo/issues/10803 * refactor: remove Experimental.StrategicProviding removing experiment, replaced with Provider.Enabled * test(cli): routing [re]provide updated and added tests for manually triggering provide and reprovide and making them respect global configuration flag to avoid inconsistent behaviors * docs: improve DelegatedRouters * refactor: default DefaultProviderWorkerCount=16 - simplified default for both - 16 is safer for non-accelerated DHT client - acceletated DHT performs better without limit anyway - updated docs	2025-05-15 19:19:18 +02:00
Sergey Gorbunov	7c844bacea	feat(fuse): Expose MFS as FUSE mount point (#10781) ... * Add MFS command line options, extend existing mount functions for MFS, set defaults. * Directory listing and file stat. * Add a read-only MFS view. * Add mkdir and interface checks. * Add remove and rename functionality. * Implement all required write interfaces. * Adjust mount functions for other architechtures. * Merge branch 'master' into feat/10710-mfs-fuse-mount * Write a basic read/write test. * Write more basic tests, add a mutex to the file object, fix modtime. * Add a concurrency test, remove mutexes from file and directory structures. * Refactor naming(mfdir -> mfsdir) and add documentation. * Add CID retrieval through ipfs_cid xattr. * Add docs, add xattr listing, fix bugs for mv and stat, refactor. * Add MFS command line options, extend existing mount functions for MFS, set defaults. * docs phrasing * docs: Mounts.MFS * docs: warn about lazy-loaded DAGs * test: TEST_FUSE=1 ./t0030-mount.sh -v --------- Co-authored-by: Guillaume Michel <guillaumemichel@users.noreply.github.com> Co-authored-by: guillaumemichel <guillaume@michel.id> Co-authored-by: Marcin Rataj <lidel@lidel.org>	2025-05-06 21:55:53 +02:00