164 Commits

Author	SHA1	Message	Date
Marcin Rataj	4a7472f1ac	fix: improve ipfs name put for IPNS record republishing (#11199) ... `name put` rejected republishing the exact same record because the sequence check used `>=` which blocked the common use case of fetching a third-party record and putting it back to refresh DHT availability. allow putting identical records (same bytes) while still rejecting different records with the same or lower sequence number. also add a success message on put (suppressible with `--quiet`), and clarify the error message to say "IPNS record" and reference `ipfs name put --force`. Closes #11197 (cherry picked from commit 3ba73501fe)	2026-02-17 03:42:09 +01:00
Marcin Rataj	6a008fc74c	refactor: apply go fix modernizers from Go 1.26 (#11190) ... * chore: apply go fix modernizers from Go 1.26 automated refactoring: interface{} to any, slices.Contains, and other idiomatic updates. * feat(ci): add `go fix` check to Go analysis workflow ensures Go 1.26 modernizers are applied, fails CI if `go fix ./...` produces any changes (similar to existing `go fmt` enforcement)	2026-02-11 01:01:32 +01:00
Marcin Rataj	36c29c55f0	feat: update to Go 1.26 (#11189) ... * feat: update to Go 1.26 replace deprecated httputil.NewSingleHostReverseProxy (Director) with ReverseProxy.Rewrite, switch math/rand to math/rand/v2 in production code, update Dockerfile base image. * fix test to accept response with HTTP status of 307 and 308 where 302 and 301 are expected --------- Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com>	2026-02-11 00:08:28 +01:00
Marcin Rataj	c6702eaf88	fix: allow dag import of 1MiB chunks wrapped in dag-pb (#11185) ... IPIP-499's unixfs-v1-2025 profile uses 1MiB chunks. with --raw-leaves=false, protobuf wrapping pushes blocks slightly over 1MiB. the previous 1MiB SoftBlockLimit rejected these blocks on dag import. raise SoftBlockLimit to 2MiB to match the bitswap spec, which requires implementations to support blocks up to 2MiB. - raise SoftBlockLimit to 2MiB per the bitswap spec - update error messages and help text - bump boxo to main with ipfs/boxo#1101 (raised ChunkSizeLimit/BlockSizeLimit, 256-byte overhead budget) - update sharness tests for 2MiB boundary - add test/cli boundary tests for block put, dag put, dag import, ipfs add (raw and wrapped leaves), and bitswap exchange including regression tests for the libp2p message size hard limit	2026-02-06 23:55:40 +01:00
Daniel Norman	f57d13c2c2	feat: swarm addrs autonat command (#11184) ... * feat: add swarm addrs autonat command fixes #11171 by adding a self service way to debug public reachability with autonat * test: add test for ipfs swarm addr autonat command * docs: add ipfs swarm addrs autonat to changelog * test: update failing test * fix: swarm addrs autonat bugfixes and cleanup - fix help text to show capitalized reachability values (Public, Private, Unknown) matching actual output from network.Reachability.String() - default Reachability to "Unknown" instead of empty string when the host interface assertion fails - extract multiaddrsToStrings and writeAddrSection helpers to deduplicate repeated conversion loops and text formatting blocks --------- Co-authored-by: Marcin Rataj <lidel@lidel.org>	2026-02-06 20:33:15 +01:00
Marcin Rataj	67c89bbd7e	feat(config): add Import.* for CID Profiles from IPIP-499 (#11148) ... * feat(config): Import.* and unixfs-v1-2025 profile implements IPIP-499: add config options for controlling UnixFS DAG determinism and introduces `unixfs-v1-2025` and `unixfs-v0-2015` profiles for cross-implementation CID reproducibility. changes: - add Import.* fields: HAMTDirectorySizeEstimation, SymlinkMode, DAGLayout, IncludeEmptyDirectories, IncludeHidden - add validation for all Import.* config values - add unixfs-v1-2025 profile (recommended for new data) - add unixfs-v0-2015 profile (alias: legacy-cid-v0) - remove deprecated test-cid-v1 and test-cid-v1-wide profiles - wire Import.HAMTSizeEstimationMode() to boxo globals - update go.mod to use boxo with SizeEstimationMode support ref: https://specs.ipfs.tech/ipips/ipip-0499/ * feat(add): add --dereference-symlinks, --empty-dirs, --hidden CLI flags add CLI flags for controlling file collection behavior during ipfs add: - `--dereference-symlinks`: recursively resolve symlinks to their target content (replaces deprecated --dereference-args which only worked on CLI arguments). wired through go-ipfs-cmds to boxo's SerialFileOptions. - `--empty-dirs` / `-E`: include empty directories (default: true) - `--hidden` / `-H`: include hidden files (default: false) these flags are CLI-only and not wired to Import.* config options because go-ipfs-cmds library handles input file filtering before the directory tree is passed to kubo. removed unused Import.UnixFSSymlinkMode config option that was defined but never actually read by the CLI. also: - wire --trickle to Import.UnixFSDAGLayout config default - update go-ipfs-cmds to v0.15.1-0.20260117043932-17687e216294 - add SYMLINK HANDLING section to ipfs add help text - add CLI tests for all three flags ref: https://github.com/ipfs/specs/pull/499 * test(add): add CID profile tests and wire SizeEstimationMode add comprehensive test suite for UnixFS CID determinism per IPIP-499: - verify exact HAMT threshold boundary for both estimation modes: - v0-2015 (links): sum(name_len + cid_len) == 262144 - v1-2025 (block): serialized block size == 262144 - verify HAMT triggers at threshold + 1 byte for both profiles - add all deterministic CIDs for cross-implementation testing also wires SizeEstimationMode through CLI/API, allowing Import.UnixFSHAMTSizeEstimation config to take effect. bumps boxo to ipfs/boxo@6707376 which aligns HAMT threshold with JS implementation (uses > instead of >=), fixing CID determinism at the exact 256 KiB boundary. * feat(add): --dereference-symlinks now resolves all symlinks Previously, resolving symlinks required two flags: - --dereference-args: resolved symlinks passed as CLI arguments - --dereference-symlinks: resolved symlinks inside directories Now --dereference-symlinks handles both cases. Users only need one flag to fully dereference symlinks when adding files to IPFS. The deprecated --dereference-args still works for backwards compatibility but is no longer necessary. * chore: update boxo and improve changelog - update boxo to ebdaf07c (nil filter fix, thread-safety docs) - simplify changelog for IPIP-499 section - shorten test names, move context to comments * chore: update boxo to 5cf22196 * chore: apply suggestions from code review Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com> * test(add): verify balanced DAG layout produces uniform leaf depth add test that confirms kubo uses balanced layout (all leaves at same depth) rather than balanced-packed (varying depths). creates 45MiB file to trigger multi-level DAG and walks it to verify leaf depth uniformity. includes trickle subtest to validate test logic can detect varying depths. supports CAR export via DAG_LAYOUT_CAR_OUTPUT env var for test vectors. * chore(deps): update boxo to 6141039ad8ef switches to 6141039ad8 changes since 5cf22196ad0b: - refactor(unixfs): use arithmetic for exact block size calculation - refactor(unixfs): unify size tracking and make SizeEstimationMode immutable - feat(unixfs): optimize SizeEstimationBlock and add mode/mtime tests also clarifies that directory sharding globals affect both `ipfs add` and MFS. * test(cli): improve HAMT threshold tests with exact +1 byte verification - add UnixFSDataType() helper to directly check UnixFS type via protobuf - refactor threshold tests to use exact +1 byte calculations instead of +1 file - verify directory type directly (ft.TDirectory vs ft.THAMTShard) instead of inferring from link count - clean up helper function signatures by removing unused cidLength parameter * test(cli): consolidate profile tests into cid_profiles_test.go remove duplicate profile threshold tests from add_test.go since they are fully covered by the data-driven tests in cid_profiles_test.go. changes: - improve test names to describe what threshold is being tested - add inline documentation explaining each test's purpose - add byte-precise helper IPFSAddDeterministicBytes for threshold tests - remove ~200 lines of duplicated test code from add_test.go - keep non-profile tests (pinning, symlinks, hidden files) in add_test.go * chore: update to rebased boxo and go-ipfs-cmds PRs * docs: add HAMT threshold fix details to changelog * feat(mfs): use Import config for CID version and hash function make MFS commands (files cp, files write, files mkdir, files chcid) respect Import.CidVersion and Import.HashFunction config settings when CLI options are not explicitly provided. also add tests for: - files write respects Import.UnixFSRawLeaves=true - single-block file: files write produces same CID as ipfs add - updated comments clarifying CID parity with ipfs add * feat(files): wire Import.UnixFSChunker and UnixFSDirectoryMaxLinks to MFS `ipfs files` commands now respect these Import.* config options: - UnixFSChunker: configures chunk size for `files write` - UnixFSDirectoryMaxLinks: triggers HAMT sharding in `files mkdir` - UnixFSHAMTDirectorySizeEstimation: controls size estimation mode previously, MFS used hardcoded defaults ignoring user config. changes: - config/import.go: add UnixFSSplitterFunc() returning chunk.SplitterGen - core/node/core.go: pass chunker, maxLinks, sizeEstimationMode to mfs.NewRoot() via new boxo RootOption API - core/commands/files.go: pass maxLinks and sizeEstimationMode to mfs.Mkdir() and ensureContainingDirectoryExists(); document that UnixFSFileMaxLinks doesn't apply to files write (trickle DAG limitation) - test/cli/files_test.go: add tests for UnixFSDirectoryMaxLinks and UnixFSChunker, including CID parity test with `ipfs add --trickle` related: boxo@54e044f1b265 * feat(files): wire Import.UnixFSHAMTDirectoryMaxFanout and UnixFSHAMTDirectorySizeThreshold wire remaining HAMT config options to MFS root: - Import.UnixFSHAMTDirectoryMaxFanout via mfs.WithMaxHAMTFanout - Import.UnixFSHAMTDirectorySizeThreshold via mfs.WithHAMTShardingSize add CLI tests: - files mkdir respects Import.UnixFSHAMTDirectoryMaxFanout - files mkdir respects Import.UnixFSHAMTDirectorySizeThreshold - config change takes effect after daemon restart add UnixFSHAMTFanout() helper to test harness update boxo to ac97424d99ab90e097fc7c36f285988b596b6f05 * fix(mfs): single-block files in CIDv1 dirs now produce raw CIDs problem: `ipfs files write` in CIDv1 directories wrapped single-block files in dag-pb even when raw-leaves was enabled, producing different CIDs than `ipfs add --raw-leaves` for the same content. fix: boxo now collapses single-block ProtoNode wrappers (with no metadata) to RawNode in DagModifier.GetNode(). files with mtime/mode stay as dag-pb since raw blocks cannot store UnixFS metadata. also fixes sparse file writes where writing past EOF would lose data because expandSparse didn't update the internal node pointer. updates boxo to v0.36.1-0.20260203003133-7884ae23aaff updates t0250-files-api.sh test hashes to match new behavior * chore(test): use Go 1.22+ range-over-int syntax * chore: update boxo to c6829fe26860 - fix typo in files write help text - update boxo with CI fixes (gofumpt, race condition in test) * chore: update go-ipfs-cmds to 192ec9d15c1f includes binary content types fix: gzip, zip, vnd.ipld.car, vnd.ipld.raw, vnd.ipfs.ipns-record * chore: update boxo to 0a22cde9225c includes refactor of maxLinks check in addLinkChild (review feedback). * ci: fix helia-interop and improve caching skip '@helia/mfs - should have the same CID after creating a file' test until helia implements IPIP-499 (tracking: https://github.com/ipfs/helia/issues/941) the test fails because kubo now collapses single-block files to raw CIDs while helia explicitly uses reduceSingleLeafToSelf: false changes: - run aegir directly instead of helia-interop binary (binary ignores --grep flags) - cache node_modules keyed by @helia/interop version from npm registry - skip npm install on cache hit (matches ipfs-webui caching pattern) * chore: update boxo to 1e30b954 includes latest upstream changes from boxo main * chore: update go-ipfs-cmds to 1b2a641ed6f6 * chore: update boxo to f188f79fd412 switches to boxo@main after merging https://github.com/ipfs/boxo/pull/1088 * chore: update go-ipfs-cmds to af9bcbaf5709 switches to go-ipfs-cmds@master after merging https://github.com/ipfs/go-ipfs-cmds/pull/315 --------- Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com>	2026-02-04 22:03:15 +01:00
Andrew Gillis	77ed3dd0ef	feat(rpc): Content-Type headers and IPNS record get/put (#11067) ... * fix http header when compress enabled for get command Closes #2376 * fix(rpc): set Content-Type for ipfs get based on output format - set application/x-tar when outputting tar (default and --archive) - set application/gzip when compression is enabled (--compress) - update go-ipfs-cmds with Tar encoding type and RFC 6713 compliant MIME types (application/gzip instead of application/x-gzip) * test(rpc): add Content-Type header tests for ipfs get * feat(rpc): add Content-Type headers for binary responses set proper Content-Type headers for RPC endpoints that return binary data: - `dag export`: application/vnd.ipld.car - `block get`: application/vnd.ipld.raw - `diag profile`: application/zip - `get`: application/x-tar or application/gzip (already worked, migrated to new API) uses the new OctetStream encoding type and SetContentType() method from go-ipfs-cmds to specify custom MIME types for binary responses. refs: https://github.com/ipfs/kubo/issues/2376 * feat(rpc): add `ipfs name get` command for IPNS record retrieval add dedicated command to retrieve raw signed IPNS records from the routing system. returns protobuf-encoded IPNS record with Content-Type `application/vnd.ipfs.ipns-record`. this provides a more convenient alternative to `ipfs routing get /ipns/<name>` which returns JSON with base64-encoded data. the raw output can be piped directly to `ipfs name inspect`: ipfs name get <name> | ipfs name inspect spec: https://specs.ipfs.tech/ipns/ipns-record/ * feat(rpc): add `ipfs name put` command for IPNS record storage adds `ipfs name put` to complement `ipfs name get`, allowing users to store IPNS records obtained from external sources without needing the private key. useful for backup, restore, and debugging workflows. the command validates records by default (signature, sequence number). use `--force` to bypass validation for testing how routing handles malformed or outdated records. also reorganizes test/cli files: - rename http_rpc_* -> rpc_* to match existing convention - merge name_get_put_test.go into name_test.go - add file header comments documenting test purposes * chore(deps): update go-ipfs-cmds to latest master includes SetContentType() for dynamic Content-Type headers --------- Co-authored-by: Marcin Rataj <lidel@lidel.org>	2026-01-30 23:41:55 +01:00
Marcin Rataj	7de7af0820	feat(dns): skip DNS lookups for AutoTLS hostnames (#11140) ... * feat(dns): resolve libp2p.direct addresses locally without network I/O p2p-forge hostnames encode IP addresses directly (e.g., 1-2-3-4.peerID.libp2p.direct -> 1.2.3.4), so DNS queries are wasteful. kubo now parses these IPs in-memory. - applies to both default libp2p.direct and custom AutoTLS.DomainSuffix - TXT queries still delegate to network for ACME DNS-01 compatibility - https://github.com/ipfs/kubo/pull/11140#discussion_r2683477754 use fallback to network DNS instead of returning errors when local parsing fails, ensuring forward compatibility with future DNS records - https://github.com/ipfs/kubo/pull/11140#discussion_r2683512408 add peerID validation using peer.Decode(), matching libp2p.direct server behavior, with fallback on invalid peerID - https://github.com/ipfs/kubo/pull/11140#discussion_r2683521930 document interaction with DNS.Resolvers in config.md - https://github.com/ipfs/kubo/pull/11140#discussion_r2683526647 add AutoTLS.SkipDNSLookup config flag to disable local resolution (useful for debugging or custom DNS override scenarios) - https://github.com/ipfs/kubo/pull/11140#discussion_r2683533462 add E2E test verifying libp2p.direct resolves locally even when DNS.Resolvers points to a broken server additional improvements: - use madns.BasicResolver interface instead of custom basicResolver - add compile-time interface checks for p2pForgeResolver and madns.Resolver - refactor tests: merge IPv4/IPv6, add helpers, use config.DefaultDomainSuffix - improve changelog to explain public good benefit (reducing DNS load) Fixes #11136	2026-01-30 17:20:56 +01:00
segfault_bits	c1fd4d70f5	feat(cli): ls --long (#11103) ... * Implements the -l/--long flag for the ipfs ls command to display Unix-style file permissions and modification times, similar to the traditional ls -l. When the --long flag is used, the output includes: - File mode/permissions in Unix format (e.g., -rw-r--r--, drwxr-xr-x) - File hash (CID) - File size (when --size is also specified) - Modification time in human-readable format - File name The permission string implementation handles all file types and special bits: - File types: regular (-), directory (d), symlink (l), named pipe (p), socket (s), character device (c), block device (b) - Special permission bits: setuid (s/S), setgid (s/S), sticky (t/T) - Lowercase when execute bit is set, uppercase when not set The timestamp format follows Unix ls conventions: - Recent files (within 6 months): "Jan 02 15:04" - Older files: "Jan 02 2006" Signed-off-by: sneax <paladesh600@gmail.com> * fix(ls): correct --long flag header order and help text - fix header column order: was "Mode Hash Size Name ModTime" but data outputs "Mode Hash Size ModTime Name", now headers match data order - remove redundant if/else branch in directory output that had identical code in both branches - add example output to help text showing format with mode, hash, size, mtime, and name columns - document that files without preserved metadata show '----------' for mode and '-' for mtime - add changelog entry for v0.40 * test(ls): add format stability tests for --long flag add tests to prevent formatting regressions in ipfs ls --long output: unit tests (core/commands/ls_test.go): - TestFormatMode: 20 cases covering all file types (regular, dir, symlink, pipe, socket, block/char devices) and special permission bits (setuid, setgid, sticky with/without execute) - TestFormatModTime: zero time, old time (year format), future time, format length consistency integration tests (test/cli/ls_test.go): - explicit full output comparison with deterministic CIDs to catch any formatting changes - header column order verification for --long with --size=true/false - files without preserved metadata (---------- and - placeholders) - directory output (trailing slash, d prefix in mode) requested in: https://github.com/ipfs/kubo/pull/11103#issuecomment-3745043561 * fix(ls): improve --long flag docs and fix minor issues - improved godocs for formatMode and formatModTime functions - fixed permBit signature: char rune → char byte (avoids unnecessary cast) - clarified help text: mode/mtime are optional UnixFS metadata - documented that times are displayed in UTC - fixed flaky time test by using 1 month ago instead of 1 hour - removed hardcoded CID assertion that would break on DAG changes * fix(ls): show "-" for missing mode in --long output display "-" instead of "----------" when mode metadata is not preserved. this avoids ambiguity with Unix mode 0000 and matches how missing mtime is already displayed. follows common Unix tool conventions (ps, netstat) where "-" indicates "not available". --------- Signed-off-by: sneax <paladesh600@gmail.com> Co-authored-by: Marcin Rataj <lidel@lidel.org>	2026-01-16 02:27:32 +01:00
Marcin Rataj	824a47ae11	feat(pubsub): persistent validation and diagnostic commands (#11110) ... * feat(pubsub): persistent seqno validation and diagnostic commands - upgrade go-libp2p-pubsub to v0.15.0 - add persistent seqno validator using BasicSeqnoValidator stores max seen seqno per peer at /pubsub/seqno/<peerid> survives daemon restarts, addresses message cycling in large networks (#9665) - add `ipfs pubsub reset` command to clear validator state - add `ipfs diag datastore get/count` commands for datastore inspection requires daemon to be stopped, useful for debugging - change pubsub status from Deprecated to Experimental - add CLI tests for pubsub and diag datastore commands - remove flaky pubsub_msg_seen_cache_test.go (replaced by CLI tests) * fix(pubsub): improve reset command and add deprecation warnings - use batched delete for efficient bulk reset - check key existence before reporting deleted count - sync datastore after deletions to ensure persistence - show "no validator state found" when resetting non-existent peer - log deprecation warnings when using --enable-pubsub-experiment or --enable-namesys-pubsub CLI flags * refactor(test): add datastore helpers to test harness --------- Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com>	2026-01-16 00:27:09 +01:00
Marcin Rataj	edb7056747	feat(config): add Gateway.MaxRequestDuration option (#11138) ... * feat(config): add Gateway.MaxRequestDuration option exposes the previously hardcoded 1 hour gateway request deadline as a configurable option, allowing operators to adjust it to fit deployment needs. protects gateway from edge cases and slow client attacks. boxo: https://github.com/ipfs/boxo/pull/1079 * test(gateway): add MaxRequestDuration integration test verifies config is wired correctly and 504 is returned when exceeded * docs: add MaxRequestDuration to gateway production guide --------- Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com>	2026-01-15 19:31:16 +01:00
Lucas Molas	39c609b3db	feat(mfs): chroot command to change the root (#8648) ... Co-authored-by: Marcin Rataj <lidel@lidel.org>	2026-01-13 21:47:36 +01:00
Andrew Gillis	56bf782cc6	test: fix flaky ipfswatch test (#11142) ... Gracefully shutdown the ipfs node to ensure data is saved. Forceful shutdown is done if there is an error sending the interrupt signal to the process, such as on Windows.	2026-01-13 02:38:44 -08:00
Marcin Rataj	25ebab9dae	feat(p2p): add --foreground flag to listen and forward commands (#11099) ... * feat(p2p): add --foreground flag to listen and forward commands adds `-f/--foreground` option that keeps the command running until interrupted (SIGTERM/Ctrl+C) or closed via `ipfs p2p close`. the listener/forwarder is automatically removed when the command exits. useful for systemd services and scripts that need cleanup on exit. * docs: add p2p-tunnels.md with systemd examples - add dedicated docs/p2p-tunnels.md covering: - why p2p tunnels (NAT traversal, no public IP needed) - quick start with netcat - background and foreground modes - systemd integration with path-based activation - security considerations and troubleshooting - document Experimental.Libp2pStreamMounting in docs/config.md - simplify docs/experimental-features.md, link to new doc - add "Learn more" links to ipfs p2p listen/forward --help - update changelog entry with doc link - add cross-reference in misc/README.md * chore: reference kubo#5460 for p2p config Ref. https://github.com/ipfs/kubo/issues/5460 * fix(daemon): write api/gateway files only after HTTP server is ready fixes race condition where $IPFS_PATH/api and $IPFS_PATH/gateway files were written before the HTTP servers were ready to accept connections. this caused issues for tools like systemd path units that immediately try to connect when these files appear. changes: - add corehttp.ServeWithReady() that signals when server is ready - wait for ready signal before writing address files - use sync.WaitGroup.Go() (Go 1.25) for cleaner goroutine management - add TestAddressFileReady to verify both api and gateway files * fix(daemon): buffer errc channel and wait for all listeners - buffer error channel with len(listeners) to prevent deadlock when multiple servers write errors simultaneously - wait for ALL listeners to be ready before writing api/gateway file, not just the first one Feedback-from: https://github.com/ipfs/kubo/pull/11099#pullrequestreview-3593885839 * docs(changelog): improve p2p tunnel section clarity reframe to lead with user benefit and add example output * docs(p2p): remove obsolete race condition caveat the "First launch fails but restarts work" troubleshooting section described a race where the api file was written before the daemon was ready. this was fixed in 80b703a which ensures api/gateway files are only written after HTTP servers are ready to accept connections. --------- Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com>	2026-01-09 19:22:43 +01:00
Marcin Rataj	5288946fd1	feat(cli): improve ipfs dag stat output UX (#11097) ... - add TTY auto-detection for progress display (matching `dag export`) - use single-line progress with carriage return instead of flooding - show human-readable sizes alongside raw bytes in summary - update --progress flag to be auto-detected by default progress format: `Fetched/Processed N blocks, M bytes (X MB)` summary format: `Total Size: 99 (99 B)`	2026-01-09 18:41:10 +01:00
Marcin Rataj	23ba660ef0	fix(routing): use LegacyProvider for HTTP-only custom routing (#11112) ... * fix(routing): use LegacyProvider for HTTP-only custom routing when `Routing.Type=custom` with only HTTP routers and no DHT, fall back to LegacyProvider instead of SweepingProvider. SweepingProvider requires a DHT client which is unavailable in HTTP-only configurations, causing it to return NoopProvider and breaking provider record announcements to HTTP routers. fixes #11089 * test(routing): verify provide stat works with HTTP-only routing * docs(config): clarify SweepEnabled fallback for HTTP-only routing --------- Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com>	2026-01-09 18:23:31 +01:00
Andrew Gillis	aa3c88dcdd	shutdown daemon after test (#11135)	2026-01-07 20:51:19 -08:00
Marcin Rataj	1301710a91	fix(ci): parallelize gotest, cleanup output, flakiness (#11113) ... * ci: parallelize gotest by separating test/cli into own job split the Go Test workflow into two parallel jobs: - `unit-tests`: runs unit tests (excluding test/cli) - `cli-tests`: runs test/cli end-to-end tests test/cli takes ~3 minutes (~50% of total gotest time), so running it in parallel should reduce wall-clock CI time by ~1.5-2.5 minutes. both jobs produce JUnit XML and HTML reports for consistent debugging. * ci(gotest): reduce noise on test timeout panics add GOTRACEBACK=single to show only one goroutine stack instead of all when a test timeout panic occurs. this makes CI output much cleaner when tests hang. * fix(ci): prevent stderr from corrupting test JSON output - remove 2>&1 which mixed "go: downloading" stderr messages into JSON - add JSON validation before parsing - print failed test names for easier debugging * ci(gotest): use gotestsum for human-readable test output - replace per-package coverage loop with single gotestsum invocation - both unit-tests and cli-tests now show human-readable output - simplified coverage collection (single -coverprofile, no gocovmerge) - clarified step names to indicate they run tests * ci: fix codecov uploads by adding token - add CODECOV_TOKEN to gotest.yml and sharness.yml - update codecov-action to v5.5.2 - add fail_ci_if_error: false for robustness codecov stopped receiving coverage data ~1 year ago when they started requiring tokens for public repos * refactor(make): add test_unit and test_cli targets - add `make test_unit` for unit tests with coverage (used by CI) - add `make test_cli` for CLI integration tests (used by CI) - only disable colors when CI env var is set (local dev gets colors) - remove legacy targets: test_go_test, test_go_short, test_go_race, test_go_expensive - update gotest.yml to use make targets instead of inline commands - add test artifacts to .gitignore * fix(ci): move client/rpc tests to cli-tests job client/rpc tests use test/cli/harness which requires the ipfs binary. Move them from test_unit to test_cli where the binary is built. also: - update gotestsum to v1.13.0 - simplify workflow step names * fix(ci): use build tags when listing test packages go list needs build tags to properly exclude packages like fuse/mfs when running with TEST_FUSE=0 (nofuse tag). * fix(ci): move test/integration to cli-tests job test/integration tests need the ipfs binary, move them from test_unit to test_cli. * fix(test): fix flaky kubo-as-a-library and GetClosestPeers tests kubo-as-a-library: use `Bootstrap()` instead of raw `Swarm().Connect()` to fix race condition between swarm connection and bitswap peer discovery. `Bootstrap()` properly integrates peers into the routing system, ensuring bitswap learns about connected peers synchronously. GetClosestPeers: simplify retry logic using `EventuallyWithT` with 10-minute timeout. tests all 4 routing types (`auto`, `autoclient`, `dht`, `dhtclient`) against real bootstrap peers with patient polling. * fix(example): use bidirectional Swarm().Connect() for reliable bitswap - connect nodes bidirectionally (A→B and B→A) to simulate mutual peering - mutual peering protects connection from resource manager culling - use port 0 for random available ports (avoids CI conflicts) - enable LoopbackAddressesOnLanDHT for local testing - move retry logic to test file using require.Eventually * fix(ci): add test_examples target and parallel example-tests job - add `make test_examples` target to mk/golang.mk for consistency with test_unit/test_cli - move example tests to separate parallel CI job (example-tests) - example: use Bootstrap() with autoconf.FallbackBootstrapPeers for reliable bitswap - example: increase context timeout to 10 minutes - test: add 60s per-request timeout to GetClosestPeers (server has 30s routing timeout) - test: reduce EventuallyWithT to 3 minutes (locally passes in under 1 minute) * fix(ci): improve test targets, exclusion patterns, and artifact naming - define COVERPKG_EXCLUDE and UNIT_EXCLUDE as documented variables - use grep -vE with single regex instead of multiple grep -v calls - add mkdir -p before rm to ensure directories exist - add DEPS_GO dependency to test_cli target - make CLI test timeout configurable via TEST_CLI_TIMEOUT (default 10m) - fix test_examples cleanup on failure using subshell - reduce GetClosestPeers test wait time from 3m to 2m - rename artifacts to match job names: unit-tests-{junit,html}, cli-tests-{junit,html} - update cli-tests upload-artifact from v5 to v6 * fix(ci): fix unit test exclusion and speed up example test - fix UNIT_EXCLUDE regex to match client/rpc at end of path - remove public bootstrap peers from example (only connect to nodeA) - example test now runs in ~3s instead of timing out * fix(test): fix flaky TestAddMultipleGCLive race condition added time.Sleep after spawning GC goroutines to ensure they reach GCLock() before the test proceeds. without this, the adder's maybePauseForGC() might check GCRequested() before GC has even requested the lock, causing the lock to not be released and GC to block indefinitely. this matches the existing pattern in TestAddGCLive which already had this sleep. also replaced context.Background() with t.Context() in both TestAddMultipleGCLive and TestAddGCLive for proper test lifecycle management. * fix(example): use test harness settings for reliable CI the kubo-as-a-library example was flaky on CI. applied test-harness-like settings that match what transports_test.go uses: - TCP-only on 127.0.0.1 with random port (no QUIC/UDP) - explicitly disable non-TCP transports (QUIC, Relay, WebTransport, etc) - use NilRouterOption (no routing) since we connect peers directly - bitswap works with directly connected peers without DHT lookups - 2-minute context timeout - streaming output in test for debugging	2026-01-08 05:07:08 +01:00
Marcin Rataj	91d392d9ae	chore: update go-libp2p 0.46 (#11105) ... * fix: update go-libp2p to v0.46.0 - reduced WebRTC log noise (go-libp2p#3426) - fixed mDNS discovery on Windows/macOS (go-libp2p#3434) - includes quic-go v0.57.1 (v0.56.0 + v0.57.0) * fix(example): kubo-as-a-library test timeout - use custom ports (4010/4011) to avoid conflicts with default 4001 - add 2-minute context timeout to fail fast - get peer addresses dynamically instead of hardcoding wrong port - wait for peer connection synchronously instead of fire-and-forget - update comments to reference autoconf.FallbackBootstrapPeers * chore: update p2p-forge to v0.7.0 * fix(test): wait for DHT readiness in GetClosestPeers test the test was failing for `routing_type=auto` because it only waited for swarm connections but not for the DHT routing table to be populated. added a separate probe loop that waits for GetClosestPeers to succeed before running the actual test assertions.	2025-12-16 19:03:08 +01:00
Andrew Gillis	ab44726177	fix(ipfswatch): loading datastore plugins (#11078) ... * ipfswatch: fix loading datastore plugins * test: add CLI tests for ipfswatch --------- Co-authored-by: Marcin Rataj <lidel@lidel.org>	2025-12-10 02:50:59 +01:00
vlerdman	d29c0b9c01	Add bytes progress tracker for ipfs pin add (#11074) ... * Add bytes progress tracker for ipfs pin add * upgrade to boxo that has ipfs/boxo#1071	2025-12-02 17:11:22 -08:00
Hector Sanjuan	73ab037d1d	feat: support GetClosesPeers (IPIP-476) and ExposeRoutingAPI by default (#10954) ... This allows Kubo to respond to the GetClosestPeers() http routing v1 endpoint as spec'ed here: https://github.com/ipfs/specs/pull/476 It is based on work from https://github.com/ipfs/boxo/pull/1021 We let IpfsNode implmement the contentRouter.Client interface with the new method. We use our WAN-DHT to get the closest peers. Additionally, Routing V1 HTTP API is exposed by default which enables light clients in browsers to use Kubo Gateway as delegated routing backend Co-authored-by: Marcin Rataj <lidel@lidel.org>	2025-11-19 10:51:56 +00:00
Marcin Rataj	597f2b827d	test: add regression tests for config secrets protection (#11061)	2025-11-17 19:10:40 +01:00
Marcin Rataj	1404861086	test: add regression tests for API.Authorizations (#11060)	2025-11-17 18:52:05 +01:00
Marcin Rataj	c7eda21d68	test: verifyWorkerRun and helptext (#11063)	2025-11-17 18:51:33 +01:00
Marcin Rataj	cec7432043	feat: fast provide support in dag import (#11058) ... * fix(add): respect Provide config in fast-provide-root fast-provide-root should honor the same config settings as the regular provide system: - skip when Provide.Enabled is false - skip when Provide.DHT.Interval is 0 - respect Provide.Strategy (all/pinned/roots/mfs/combinations) This ensures fast-provide only runs when appropriate based on user configuration and the nature of the content being added (pinned vs unpinned, added to MFS or not). * feat(config): options to adjust global defaults Add Import.FastProvideRoot and Import.FastProvideWait configuration options to control default behavior of fast-provide-root and fast-provide-wait flags in ipfs add command. Users can now set global defaults in config while maintaining per-command flag overrides. - Add Import.FastProvideRoot (default: true) - Add Import.FastProvideWait (default: false) - Add ResolveBoolFromConfig helper for config resolution - Update docs with configuration details - Add log-based tests verifying actual behavior * refactor: extract fast-provide logic into reusable functions Extract fast-provide logic from add command into reusable components: - Add config.ShouldProvideForStrategy helper for strategy matching - Add ExecuteFastProvide function reusable across add and dag import commands - Move DefaultFastProvideTimeout constant to config/provide.go - Simplify add.go from 72 lines to 6 lines for fast-provide - Move fast-provide tests to dedicated TestAddFastProvide function Benefits: - cleaner API: callers only pass content characteristics - all strategy logic centralized in one place - better separation of concerns - easier to add fast-provide to other commands in future * feat(dag): add fast-provide support for dag import Adds --fast-provide-root and --fast-provide-wait flags to `ipfs dag import`, mirroring the fast-provide functionality available in `ipfs add`. Changes: - Add --fast-provide-root and --fast-provide-wait flags to dag import command - Implement fast-provide logic for all root CIDs in imported CAR files - Works even when --pin-roots=false (strategy checked internally) - Share ExecuteFastProvide implementation between add and dag import - Move ExecuteFastProvide to cmdenv package to avoid import cycles - Add logging when fast-provide is disabled - Conditional error handling: return error when wait=true, warn when wait=false - Update config docs to mention both ipfs add and ipfs dag import - Update changelog to use "provide" terminology and include dag import examples - Add comprehensive test coverage (TestDagImportFastProvide with 6 test cases) The fast-provide feature allows immediate DHT announcement of root CIDs for faster content discovery, bypassing the regular background queue. * docs: improve fast-provide documentation Refine documentation to better explain fast-provide and sweep provider working together, and highlight the performance improvement. Changelog: - add fast-provide to sweep provider features list - explain performance improvement: root CIDs discoverable in <1s vs 30+ seconds - note this uses optimistic DHT operations (faster with sweep provider) - simplify examples, point to --help for details Config docs: - fix: --fast-provide-roots should be --fast-provide-root (singular) - clarify Import.FastProvideRoot focuses on root CIDs while sweep handles all blocks - simplify Import.FastProvideWait description Command help: - ipfs add: explain sweep provider context upfront - ipfs dag import: add fast-provide explanation section - both explain the split: fast-provide for roots, sweep for all blocks * test: add tests for ShouldProvideForStrategy add tests covering all provide strategy combinations with focus on bitflag OR logic (the else-if bug fix). organized by behavior: - all strategy always provides - single strategies match only their flag - combined strategies use OR logic - zero strategy never provides * refactor: error cmd on error and wait=true change ExecuteFastProvide() to return error, enabling proper error propagation when --fast-provide-wait=true. in sync mode, provide failures now error the command as expected. in async mode (default), always returns nil with errors logged in background goroutine. also remove duplicate ExecuteFastProvide() from provide.go (75 lines), keeping single implementation in cmdenv/env.go for reuse across add and dag import commands. call sites simplified: - add.go: check and propagate error from ExecuteFastProvide - dag/import.go: return error from ForEach callback, remove confusing conditional error handling semantics: - precondition skips (DHT unavailable, etc): return nil (not failure) - async mode (wait=false): return nil, log errors in goroutine - sync mode (wait=true): return wrapped error on provide failure	2025-11-14 21:06:25 -08:00
Hector Sanjuan	d45c615e73	feat(telemetry): collect high level provide DHT sweep settings (#11056) ... * telemetry: collect provideDHTSweepEnabled Fixes #11055. * telemetry: track custom Provide.DHT.Interval and MaxWorkers collects whether users customize Interval and MaxWorkers from defaults to help identify if defaults need adjustment * docs: improve telemetry documentation structure and clarity restructure docs/telemetry.md into meaningful sections (routing & discovery, content providing, network configuration), add exact config field paths for all tracked settings, and establish code as source of truth by linking from LogEvent struct while removing redundant field comments --------- Co-authored-by: Marcin Rataj <lidel@lidel.org>	2025-11-13 01:28:15 +01:00
Guillaume Michel	702c63b6db	feat: enable DHT Provide Sweep by default (#10955) ... Co-authored-by: Marcin Rataj <lidel@lidel.org> Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com>	2025-11-12 23:55:17 +01:00
Marcin Rataj	f067a86170	fix(go-log@2.9/go-libp2p@0.45): dynamic log level control and tail (#11039) ... This fix restores dynamic log level control and tail for go-libp2p loggers Updated to: https://github.com/libp2p/go-libp2p/releases/tag/v0.45.0 https://github.com/ipfs/go-log/releases/tag/v2.9.0 these changes restore dynamic log level control and tail for go-libp2p subsystems after the migration to slog, fixing the regression introduced in https://github.com/libp2p/go-libp2p/pull/3364 Fixes https://github.com/ipfs/kubo/issues/11035 For details why and how, see explainer in https://github.com/ipfs/go-log/releases/tag/v2.9.0	2025-11-06 16:00:56 +01:00
Guillaume Michel	c2bf0f9515	feat(provider): resume cycle (#11031) ... * bump kad-dht: resume reprovide cycle * daemon: --provide-fresh-start flag * changelog * docs * go-fmt * chore: latest go-libp2p-kad-dht#1170 after conflict resolution, to confirm CI is still green * kad-dht: depend on latest master * move daemon flag to Provider.DHT.ResumeEnabled config * refactor: sweep provider datastore * bump kad-dht * bump kad-dht * bump kad-dht * make datastore keys constant * use kad-dht master * add emoji to changelog entry * go-fmt * bump kad-dht * test(provider): add tests for resume cycle feature validates Provide.DHT.ResumeEnabled behavior: - preserves cycle state when enabled (default) - resets cycle when disabled tests verify current_time_offset across restarts using JSON output --------- Co-authored-by: Marcin Rataj <lidel@lidel.org>	2025-10-29 11:07:46 +01:00
Guillaume Michel	16479ec692	feat(provide): detailed ipfs provide stat (#11019) ... * feat: provide stats * added N/A * format * workers stats alignment * ipfs provide stat --all --compact * consolidating compact stat * update column alignment * flags combinations errors * command description * change schedule AvgPrefixLen to float * changelog * alignments * provide stat description draft * rephrased provide-stats.md * linking provide-stats.md from command description * documentation test * fix: refactor provide stat command type handling - add extractSweepingProvider() helper to reduce nested type switching - extract lowWorkerThreshold constant for worker availability check - fix --lan error handling to work with buffered providers * docs: add clarifying comments * fix(commands): improve provide stat compact mode - prevent panic when both columns are empty - fix column alignment with UTF-8 characters - only track col0MaxWidth for first column (as intended) * test: add tests for ipfs provide stat command - test basic functionality, flags, JSON output - test legacy provider behavior - test integration with content scheduling - test disabled provider configurations - add parseSweepStats helper with t.Helper() * docs: improve provide command help text - update tagline to "Control and monitor content providing" - simplify help descriptions - make error messages more consistent - update tests to match new error messages * metrics rename ``` Next reprovide at: Next prefix: ``` updated to: ``` Next region prefix: Next region reprovide: ``` * docs: improve Provide system documentation clarity Enhance documentation for the Provide system to better explain how provider records work and the differences between sweep and legacy modes. Changes to docs/config.md: - Provide section: add clear explanation of provider records and their role - Provide.DHT: add provider record lifecycle and two provider systems overview - Provide.DHT.Interval: explain relationship to expiration, contrast sweep vs legacy behavior - Provide.DHT.SweepEnabled: rewrite to explain legacy problem, sweep solution, and efficiency gains - Monitoring section: prioritize command-line tools (ipfs provide stat) before Prometheus Changes to core/commands/provide.go: - ipfs provide stat help: add explanation of provider records, TTL expiration, and how sweep batching works Changes to docs/changelogs/v0.39.md: - Add context about why stats matter for monitoring provider health - Emphasize real-time monitoring workflow with watch command - Explain what users can observe (rates, queues, worker availability) * depend on latest kad-dht master * docs: nits --------- Co-authored-by: Marcin Rataj <lidel@lidel.org>	2025-10-23 20:29:36 +02:00
Marcin Rataj	f4834e797d	fix: migrations for Windows (#11010) ... * test: add migration tests for Windows and macOS - add dedicated CI workflow for migration tests on Windows/macOS - workflow triggers on migration-related file changes only * build: remove redundant go version checks - remove GO_MIN_VERSION and check_go_version scripts - go.mod already enforces minimum version (go 1.25) - fixes make build on Windows * fix: windows migration panic by reading config into memory fixes migration panic on Windows when upgrading from v0.37 to v0.38 by reading the entire config file into memory before performing atomic operations. this avoids file locking issues on Windows where open files cannot be renamed. also fixes: - TestRepoDir to set USERPROFILE on Windows (not just HOME) - CLI migration tests to sanitize directory names (remove colons) minimal fix that solves the "panic: error can't be dealt with transactionally: Access is denied" error without adding unnecessary platform-specific complexity. * fix: set PATH for CLI migration tests in CI the CLI tests need the built ipfs binary to be in PATH * fix: use ipfs shutdown for graceful daemon termination in tests replaces platform-specific signal handling with ipfs shutdown command which works consistently across all platforms including Windows * fix: isolate PATH modifications in parallel migration tests tests running in parallel with t.Parallel() were interfering with each other through global PATH modifications via os.Setenv(). this caused tests to download real migration binaries instead of using mocks, leading to Windows failures due to path separator issues in external tools. now each test builds its own custom PATH and passes it explicitly to commands, preventing interference between parallel tests. * chore: improve error messages in WithBackup * fix: Windows CI migration test failures - add .exe extension to mock migration binaries on Windows - handle repo lock file properly in mock migration binary - ensure lock is created and removed to prevent conflicts * refactor: align atomicfile error handling with fs-repo-migrations - check close error in Abort() before attempting removal - leave temp file on rename failure for debugging (like fs-repo-15-to-16) - improves consistency with external migration implementations * fix: use req.Context in repo migrate to avoid double-lock The repo migrate command was calling cctx.Context() which has a hidden side effect: it lazily constructs the IPFS node by calling GetNode(), which opens the repository and acquires repo.lock. When migrations then tried to acquire the same lock, it failed with "lock is already held by us" because go4.org/lock tracks locks per-process in a global map. The fix uses req.Context instead, which is a plain context.Context with no side effects. This provides what migrations need (cancellation handling) without triggering node construction or repo opening. Context types explained: - req.Context: Standard Go context for request lifetime, cancellation, and timeouts. No side effects. - cctx.Context(): Kubo-specific method that lazily constructs the full IPFS node (opens repo, acquires lock, initializes subsystems). Returns the node's internal context. Why req.Context is correct here: - Migrations work on raw filesystem (only need ConfigRoot path) - Command has SetDoesNotUseRepo(true) - doesn't need running node - Migrations handle their own locking via lockfile.Lock() - Need cancellation support but not node lifecycle The bug only appeared with embedded migrations (v16+) because they run in-process. External migrations (pre-v16) were separate processes, so each had isolated state. Sequential migrations (forward then backward) in the same process exposed this latent double-lock issue. Also adds repo.lock acquisition to RunEmbeddedMigrations to prevent concurrent migration access, and removes the now-unnecessary daemon lock check from the migrate command handler. * fix: use req.Context for migrations and autoconf in daemon startup daemon.go was incorrectly using cctx.Context() in two critical places: 1. Line 337: migrations call - cctx.Context() triggers GetNode() which opens the repo and acquires repo.lock BEFORE migrations run, causing "lock is already held by us" errors when migrations try to lock 2. Line 390: autoconf client.Start() - uses context for HTTP timeouts and background updater lifecycle, doesn't need node construction Both now use req.Context (plain Go context) which provides: - request lifetime and cancellation - no side effects (doesn't construct node or open repo) - correct lifecycle for HTTP requests and background goroutines	2025-10-08 18:02:04 +02:00
Marcin Rataj	a688b7eeac	fix(mfs): add soft limit for --flush=false (#10985) ... * fix: add MFS operation limit for --flush=false adds a global counter that tracks consecutive MFS operations performed with --flush=false and fails with clear error after limit is reached. this prevents unbounded memory growth while avoiding the data corruption risks of auto-flushing. - adds Internal.MFSNoFlushLimit config - operations fail with actionable error at limit - counter resets on successful flush or any --flush=true operation - operations with --flush=true reset and don't count this commit removes automatic flush from https://github.com/ipfs/kubo/pull/10971 and instead errors to encourage users of --flush=false to develop a habit of calling 'ipfs files flush' periodically. boxo will no longer auto-flush (https://github.com/ipfs/boxo/pull/1041) to avoid corruption issues, and kubo applies the limit to 'ipfs files' commands instead. closes #10842 * test: add tests for MFSNoFlushLimit tests verify the new Internal.MFSNoFlushLimit config option: - default limit of 256 operations - custom limit configuration - counter reset on flush=true - counter reset on explicit flush command - limit=0 disables the feature - multiple MFS command types count towards limit * docs: explain why MFS operations fail instead of auto-flushing addresses feedback from https://github.com/ipfs/kubo/pull/10985#pullrequestreview-3256250970 - clarify that automatic flushing at limit was considered but rejected - explain the data corruption risks of auto-flushing - guide users who want auto-flush to use --flush=true (default) - document benefits of explicit failure for batch operations	2025-09-26 01:25:23 +02:00
Marcin Rataj	1107ac42af	feat: limit pin names to 255 bytes (#10981) ... adds validation to ensure pin names don't exceed 255 bytes across all commands that accept pin names. this prevents issues with filesystem limitations and improves compatibility. affected commands: - ipfs pin add --name - ipfs add --pin-name - ipfs pin ls --name (filter) - ipfs pin remote add --name - ipfs pin remote ls --name (filter) - ipfs pin remote rm --name (filter)	2025-09-25 23:54:04 +02:00
Guillaume Michel	6fcbba4b4a	fix: allow custom http provide when libp2p node is offline (#10974) ... * feat: allow custom http provide when offline * refactor: improve offline HTTP provider handling and tests - fixed comment/function name mismatch - added mock server test for HTTP provide success - clarified test names for offline scenarios * test: simplify single-node provider tests use h.NewNode().Init() instead of NewNodes(1) for cleaner test setup * fix: allow SweepingProvider to work with HTTP-only routing when no DHT is available but HTTP routers are configured for providing, return NoopProvider instead of failing. this allows the daemon to start and HTTP-based providing to work through the routing system. moved HTTP provider detection to config package as HasHTTPProviderConfigured() for better code organization and reusability. this fix is important as SweepingProvider will become the new default in the future. --------- Co-authored-by: Marcin Rataj <lidel@lidel.org>	2025-09-19 18:55:42 +02:00
Marcin Rataj	f6a9b347cb	fix(cmds): cleanup unicode identify strings (#9465) ... preserve private use characters as specified in https://github.com/libp2p/specs/pull/491 enforce 128 rune limit on untrusted peer data	2025-09-19 04:46:38 +02:00
Marcin Rataj	90740dca21	feat: ipfs-webui v4.9.0 with retrieval diagnostics (#10969) ... * fix(webui): show helpful errors for incompatible configurations - show error when Gateway.NoFetch=true and WebUI is not available locally - show error when Gateway.DeserializedResponses=false (incompatible) - add tests for both error scenarios * chore(webui): update to v4.9.0 https://github.com/ipfs/ipfs-webui/releases/tag/v4.9.0 * docs: add WebUI v4.9.0 update to v0.38 changelog - highlight new diagnostics screen for troubleshooting - include screenshots of key features in table format - add local access URL for WebUI - update TOC with new sections	2025-09-19 04:16:21 +02:00
Marcin Rataj	fa17b69c7d	fix(mfs): unbound cache growth with flush=false (#10971) ... * fix: prevent --flush=false in 'ipfs files rm' command the 'ipfs files rm' command always flushes for safety to ensure data integrity. this change adds an explicit error when users try to pass --flush=false, improving ux and preventing confusion. related to #10842 * fix: add MFS cache size limit to prevent unbounded growth - add Internal.MFSAutoflushThreshold config (experimental) - directories auto-flush when cache exceeds threshold with --flush=false - prevents high memory usage issue from #10842 - default: 256 entries per directory (matching HAMT shard size) - set to 0 to restore old behavior (risky, may cause errors) Closes #10842	2025-09-19 03:39:55 +02:00
Marcin Rataj	d37b92bfcd	fix: ipfs pin ls <cid> --names (#10970) ... * fix: use CheckIfPinnedWithType for pin ls with names updates to use CheckIfPinnedWithType method from https://github.com/ipfs/boxo/pull/1035, enabling efficient pin name retrieval for 'ipfs pin ls <cid> --names' - uses new CheckIfPinnedWithType from boxo for type-specific pin checks - pin names are now returned when listing specific CIDs with --names flag * test: add CLI tests for pin ls with names tests cover: - pin ls with specific CIDs returning names - pin ls without CID listing all pins with names - pin ls with --type and --names combinations - JSON output with and without names - pin update preserving names - error cases (invalid CID, unpinned CID) * docs: add pin name improvements to v0.38 changelog covers fix for ipfs pin ls --names with specific CIDs and RPC pin name leak fix * fix(rpc): support pin names in Add() passes the Name field from PinAddSettings to the API request adds test to verify pin names work via RPC * test: add coverage for pin names functionality - test special characters, unicode, long names - test concurrent operations - test persistence across daemon restarts - test garbage collection preservation - fix indirect pin test logic * chore: boxo@main with boxo#1039 * fix(pin): improve pin ls robustness and validation - add nil check for n.Pinning with early fail-fast validation - use pin.StringToMode() for consistent type validation - add edge case tests for invalid types and unpinned CIDs	2025-09-19 03:17:45 +02:00
Marcin Rataj	71e883440e	refactor(config): migration 17-to-18 to unify Provider/Reprovider into Provide.DHT (#10951) ... * refactor: consolidate Provider/Reprovider into unified Provide config - merge Provider and Reprovider configs into single Provide section - add fs-repo-17-to-18 migration for config consolidation - improve migration ergonomics with common package utilities - convert deprecated "flat" strategy to "all" during migration - improve Provide docs * docs: add total_provide_count metric guidance - document how to monitor provide success rates via prometheus metrics - add performance comparison section to changelog - explain how to evaluate sweep vs legacy provider effectiveness * fix: add OpenTelemetry meter provider for metrics - set up meter provider with Prometheus exporter in daemon - enables metrics from external libs like go-libp2p-kad-dht - fixes missing total_provide_count_total when SweepEnabled=true - update docs to reflect actual metric names --------- Co-authored-by: gammazero <11790789+gammazero@users.noreply.github.com> Co-authored-by: guillaumemichel <guillaume@michel.id> Co-authored-by: Daniel Norman <1992255+2color@users.noreply.github.com> Co-authored-by: Hector Sanjuan <code@hector.link>	2025-09-18 22:17:43 +02:00
Guillaume Michel	006f9dc704	feat: opt-in new Sweep provide system (#10834) ... * reprovide sweep draft * update reprovider dep * go mod tidy * fix provider type * change router type * dual reprovider * revert to provider.System * back to start * SweepingReprovider test * fix nil pointer deref * noop provider for nil dht * disabled initial network estimation * another iteration * suppress missing self addrs err * silence empty rt err on lan dht * comments * new attempt at integrating * reverting changes in core/node/libp2p/routing.go * removing SweepingProvider * make reprovider optional * add noop reprovider * update KeyChanFunc type alias * restore boxo KeyChanFunc * fix missing KeyChanFunc * test(sharness): PARALLEL=1 and timeout 30m running sequentially to see where timeout occurs * initialize MHStore * revert workflow debug * config * config docs * merged IpfsNode provider and reprovider * move Provider interface to from kad-dht to node * moved Provider interface from kad-dht to kubo/core/node * mod_tidy * Add Clear to Provider interface * use latest kad-dht commit * make linter happy * updated boxo provide interface * boxo PR fix * using latest kad-dht commit * use latest boxo release * fix fx * fx cyclic deps * fix merge issues * extended tests * don't provide LAN DHT * docs * restore dual dht provider * don't start provider before it is online * address linter * dual/provider fix * add delay in provider tests for dht bootstrap * add OfflineDelay parameter to config * remove increase number of workers in test * improved keystore gc process * fix: replace incorrect logger import in coreapi replaced github.com/labstack/gommon/log with the standard github.com/ipfs/go-log/v2 logger used throughout kubo. removed unused labstack dependency from go.mod files. * fix: remove duplicate WithDefault call in provider config * fix: use correct option method for burst workers * fix: improve error messages for experimental sweeping provider updated error messages to clearly indicate when commands are unavailable due to experimental sweeping provider being enabled via Reprovider.Sweep.Enabled=true * docs: remove obsolete KeyStoreGCInterval config removed from config.md as option no longer exists (removed in b540fba1a) updated keystore description to reflect gc happens at reprovide interval * docs: add TODO placeholder changelog for experimental sweeping DHT provider using v0.38-TODO.md name to avoid merge conflicts with master branch and allow CI tests to run. will be renamed to v0.38.md once config migration is added to the PR * fix: provideKeysRec go routine * clear keystore on close * fix: datastore prefix * fix: improve error handling in provideKeysRec - close errCh channel to distinguish between nil and pending errors - check for pending errors when provided.New closes - handle context cancellation during error send - prevent race condition where errors could be silently lost this ensures DAG walk errors are always propagated correctly * address gammazero's review * rename BurstProvider to LegacyProvider * use latest provider/keystore * boxo: make mfs StartProviding async * bump boxo * chore: update boxo to f2b4e12fb9a8ac138ccb82aae3b51ec51d9f631c - updated boxo dependency to specified commit - updated go.mod and go.sum files across all modules * use latest kad-dht/boxo * Buffered SweepingProvider wrapper * use latest kad-dht commit * allow no DHT router * use latest kad-dht & boxo --------- Co-authored-by: Marcin Rataj <lidel@lidel.org> Co-authored-by: gammazero <11790789+gammazero@users.noreply.github.com>	2025-09-18 17:22:04 +02:00
Marcin Rataj	20f8151628	fix: enforce identity CID size limits (#10949) ... * fix: enforce identity CID size limits - validate --inline-limit against verifcid.MaxDigestSize - add error when --hash=identity exceeds size limit - add tests for identity CID overflow scenarios - update help text to show maximum inline limit This prevents creation of unbounded identity CIDs by enforcing the 128-byte limit defined in https://github.com/ipfs/boxo/pull/1018 Fixes #6011 IPIP: https://github.com/ipfs/specs/pull/512	2025-09-09 20:22:21 +02:00
Hector Sanjuan	15f723a15e	fix: disable telemetry in test profile (#10931) ... * Tests: disable telemetry in tests by default Disable the plugin in cli tests and sharness by default. Enable only in telemetry tests. There are cases when tests get stuck or get killed and leave daemons hanging around. We don't want to be getting telemetry from those. * sharness: attempt to fix * sharness: add missing --bool flag * fix(ci): add omitempty to Plugin.Config field The sharness problem is that when the telemetry plugin is configured initially with 'ipfs config --bool', it creates a structure without the 'Config: null' field, but when the config is copied and replaced, it expects the structure to be preserved. Adding omitempty ensures the Config field is omitted from JSON when nil, making the config structure consistent between initial creation and replacement operations. --------- Co-authored-by: Marcin Rataj <lidel@lidel.org>	2025-08-24 14:30:35 +02:00
Andrew Gillis	ae068a8061	fix: harness tests random panic (#10933) ... * fix: harness tests random panic Connecting nodes in parallel can cause TLS handshake failures. For each node, connect to the other nodes serially. It is not necessary to connect in parallel as it does not save any significant time. Closes #10932	2025-08-22 13:57:35 -07:00
Hector Sanjuan	846e219ead	feat: Reprovider.Strategy: rename "flat" to "all" (#10928) ... * Reprovider strategy: rename "flat" to "all". Value "flat" now parses to "all". Behaviour from "all" removed. Fixes #10864 which has detailed explanation. * core/node/provider.go: remove unused function mfsRootProvider It was used in the "all" strategy. * docs: improve reprovider.strategy=all changelog framing - highlight memory efficiency improvements - clarify this removes v0.28 workaround - update config.md memory requirements - fix announce-on profile typo * feat: deprecate Reprovider.Strategy=flat - add deprecation warning in daemon.go when flat strategy is detected - document that flat is deprecated in ParseReproviderStrategy comment - add explicit test case for flat -> all mapping - flat continues to work but users are warned to migrate to all --------- Co-authored-by: Marcin Rataj <lidel@lidel.org>	2025-08-21 19:02:29 +02:00
Marcin Rataj	ccb49de852	feat(config): AutoConf with "auto" placeholders (#10883) ... https://github.com/ipfs/kubo/pull/10883 https://github.com/ipshipyard/config.ipfs-mainnet.org/issues/3 --------- Co-authored-by: gammazero <gammazero@users.noreply.github.com>	2025-08-20 05:59:11 +02:00
Marcin Rataj	4bafb22b76	fix(ci): make NewRandPort thread-safe (#10921) ... * chore: disable AutoTLS in TCP-only transport tests Tests were failing intermittently. Disabling AutoTLS when WebSocket transport is disabled appears to resolve the issue. * fix: make NewRandPort thread-safe Track allocated ports globally to prevent conflicts when tests run in parallel.	2025-08-18 22:12:22 +02:00
Hector Sanjuan	4255cc3889	feat: telemetry plugin (#10866) ... * Initial pass at Telemetry plugin Currently, IP Shipyard, with the help of Probelab, monitor and extract Amino/IPFS public network metrics with the use of DHT crawlers and bootstrappers (via peerlog plugin). For example, we log all peer IDs seen and their AgentVersion/Addresses obtained from the `identify` protocol, which provides insights into protocol usage, total number of peers etc. We would like to increase the ability to obtain more insights from the network by collecting some more information in the future, but also to give users more control over this collection (i.e. opt-out). The information collected will not allow unique identification of anyone and is only used for aggregation. Now, this PR explores a way of moving in this direction: * A new "telemetry" fx plugin is in charge of dealing with telemetry * The FX plugin allows to plug and make decisions / take actions during the setup phase: * We can inspect whether we are using Private Networks before the libp2p.Host has been initialized. * We can send telemetry after the libp2p Host is initialized. * Everything is self-contained. Custom builds can remove the plugin altogether without needing to surgically edit the code. As for behaviour: * The user can opt-in/out via EnvVar, file in the repo path or plugin configuration. * Users on private networks or with custom bootstrappers are detected, offered a wall of text explaining why we need telemetry and invited to opt-in. Opt-out happens otherwise on a timeout (with no input). Their preferences are stored. * Users on standard settings are opted-in by default. This is the status quo in Kubo already, except they don't get a chance to opt out. The telemetry libp2p protocol is yet to be defined, but expect something similar to identify, with a protobuf being pushed to bootstrappers or to a specific telemetry node that we define. In the case of pnets, this will be done with a temporary peer. * checkpoint * telemetry plugin: second pass * On first run it generates a UUID and shows a message to the user. * UUID is persistend to "telemetry_uuid" * Sends telemetry 1 minute after boot and every 24h * LogEvent is the thing containing all the telemetry that is sent * Opt-out possible via env-var or plugin configuration * Telemetry: add changelog and environment variable documentation * docs: improved daemon message making it more obvious nothing was sent yet and that user had 15m to out-out plus some debug logs that confirm opt-out * refactor: rename IPFS_TELEMETRY_MODE to IPFS_TELEMETRY * fix: add User-Agent header to telemetry requests --------- Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com> Co-authored-by: Marcin Rataj <lidel@lidel.org>	2025-08-18 20:46:05 +02:00
Andrew Gillis	187fce7a6d	fix typos in docs and comments (#10920)	2025-08-18 08:43:25 -07:00
Marcin Rataj	64c47c374a	feat(config): Gateway.RetrievalTimeout|MaxConcurrentRequests (#10905) ... * feat(gateway): concurrency and timeout limits Depends on https://github.com/ipfs/boxo/pull/994 * chore: boxo master with final boxo#994 this includes race-condition fixes from ipfs/boxo#994 and increased `DefaultMaxConcurrentRequests = 4096` * docs: concise config.md and changelog	2025-08-15 02:36:33 +02:00