* chore: apply go fix modernizers from Go 1.26
automated refactoring: interface{} to any, slices.Contains,
and other idiomatic updates.
* feat(ci): add `go fix` check to Go analysis workflow
ensures Go 1.26 modernizers are applied, fails CI if `go fix ./...`
produces any changes (similar to existing `go fmt` enforcement)
* feat: update to Go 1.26
replace deprecated httputil.NewSingleHostReverseProxy (Director)
with ReverseProxy.Rewrite, switch math/rand to math/rand/v2 in
production code, update Dockerfile base image.
* fix test to accept response with HTTP status of 307 and 308 where 302 and 301 are expected
---------
Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com>
* docs: loud deprecation of badger v1 datastore
badger v1 (go-ds-badger) has not been maintained by its upstream
maintainers for years and has known bugs (startup timeouts, shutdown
hangs, fd exhaustion). make the deprecation loud and unavoidable:
- print ERROR log line and bordered stderr message on every daemon
start when a badger-based datastore is detected
- mark badgerds and badgerds-measure init profiles as DEPRECATED
- update docs/datastores.md and docs/config.md with migration guidance
- add changelog highlight for v0.40
badger v1 support will be removed later in 2026.
part of https://github.com/ipfs/kubo/issues/11186
* docs(changelog): link badger deprecation to #11186
* feat(gateway): IPIP-0524 Gateway.AllowCodecConversion config option
Wire up boxo's AllowCodecConversion config to control codec conversion
behavior per IPIP-0524. When false (default), the gateway returns
406 Not Acceptable if the requested format doesn't match the block's
codec.
Clients should fetch raw blocks (`?format=raw`) and convert client-side.
Ref: https://github.com/ipfs/specs/pull/524
Ref: https://github.com/ipfs/boxo/pull/1077
Ref: https://github.com/ipfs/gateway-conformance/pull/254
* chore: update boxo for improved 406 codec conversion error
boxo now returns an actionable hint when codec conversion is rejected:
suggests fetching raw block with ?format=raw and converting client-side.
* chore: bump boxo and gateway-conformance to v0.10
* docs: add IPLD Logical Format note to AllowCodecConversion
* feat(config): Import.* and unixfs-v1-2025 profile
implements IPIP-499: add config options for controlling UnixFS DAG
determinism and introduces `unixfs-v1-2025` and `unixfs-v0-2015`
profiles for cross-implementation CID reproducibility.
changes:
- add Import.* fields: HAMTDirectorySizeEstimation, SymlinkMode,
DAGLayout, IncludeEmptyDirectories, IncludeHidden
- add validation for all Import.* config values
- add unixfs-v1-2025 profile (recommended for new data)
- add unixfs-v0-2015 profile (alias: legacy-cid-v0)
- remove deprecated test-cid-v1 and test-cid-v1-wide profiles
- wire Import.HAMTSizeEstimationMode() to boxo globals
- update go.mod to use boxo with SizeEstimationMode support
ref: https://specs.ipfs.tech/ipips/ipip-0499/
* feat(add): add --dereference-symlinks, --empty-dirs, --hidden CLI flags
add CLI flags for controlling file collection behavior during ipfs add:
- `--dereference-symlinks`: recursively resolve symlinks to their target
content (replaces deprecated --dereference-args which only worked on
CLI arguments). wired through go-ipfs-cmds to boxo's SerialFileOptions.
- `--empty-dirs` / `-E`: include empty directories (default: true)
- `--hidden` / `-H`: include hidden files (default: false)
these flags are CLI-only and not wired to Import.* config options because
go-ipfs-cmds library handles input file filtering before the directory
tree is passed to kubo. removed unused Import.UnixFSSymlinkMode config
option that was defined but never actually read by the CLI.
also:
- wire --trickle to Import.UnixFSDAGLayout config default
- update go-ipfs-cmds to v0.15.1-0.20260117043932-17687e216294
- add SYMLINK HANDLING section to ipfs add help text
- add CLI tests for all three flags
ref: https://github.com/ipfs/specs/pull/499
* test(add): add CID profile tests and wire SizeEstimationMode
add comprehensive test suite for UnixFS CID determinism per IPIP-499:
- verify exact HAMT threshold boundary for both estimation modes:
- v0-2015 (links): sum(name_len + cid_len) == 262144
- v1-2025 (block): serialized block size == 262144
- verify HAMT triggers at threshold + 1 byte for both profiles
- add all deterministic CIDs for cross-implementation testing
also wires SizeEstimationMode through CLI/API, allowing
Import.UnixFSHAMTSizeEstimation config to take effect.
bumps boxo to ipfs/boxo@6707376 which aligns HAMT threshold with
JS implementation (uses > instead of >=), fixing CID determinism
at the exact 256 KiB boundary.
* feat(add): --dereference-symlinks now resolves all symlinks
Previously, resolving symlinks required two flags:
- --dereference-args: resolved symlinks passed as CLI arguments
- --dereference-symlinks: resolved symlinks inside directories
Now --dereference-symlinks handles both cases. Users only need one flag
to fully dereference symlinks when adding files to IPFS.
The deprecated --dereference-args still works for backwards compatibility
but is no longer necessary.
* chore: update boxo and improve changelog
- update boxo to ebdaf07c (nil filter fix, thread-safety docs)
- simplify changelog for IPIP-499 section
- shorten test names, move context to comments
* chore: update boxo to 5cf22196
* chore: apply suggestions from code review
Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com>
* test(add): verify balanced DAG layout produces uniform leaf depth
add test that confirms kubo uses balanced layout (all leaves at same
depth) rather than balanced-packed (varying depths). creates 45MiB file
to trigger multi-level DAG and walks it to verify leaf depth uniformity.
includes trickle subtest to validate test logic can detect varying depths.
supports CAR export via DAG_LAYOUT_CAR_OUTPUT env var for test vectors.
* chore(deps): update boxo to 6141039ad8ef
switches to 6141039ad8
changes since 5cf22196ad0b:
- refactor(unixfs): use arithmetic for exact block size calculation
- refactor(unixfs): unify size tracking and make SizeEstimationMode immutable
- feat(unixfs): optimize SizeEstimationBlock and add mode/mtime tests
also clarifies that directory sharding globals affect both `ipfs add` and MFS.
* test(cli): improve HAMT threshold tests with exact +1 byte verification
- add UnixFSDataType() helper to directly check UnixFS type via protobuf
- refactor threshold tests to use exact +1 byte calculations instead of +1 file
- verify directory type directly (ft.TDirectory vs ft.THAMTShard) instead of
inferring from link count
- clean up helper function signatures by removing unused cidLength parameter
* test(cli): consolidate profile tests into cid_profiles_test.go
remove duplicate profile threshold tests from add_test.go since they
are fully covered by the data-driven tests in cid_profiles_test.go.
changes:
- improve test names to describe what threshold is being tested
- add inline documentation explaining each test's purpose
- add byte-precise helper IPFSAddDeterministicBytes for threshold tests
- remove ~200 lines of duplicated test code from add_test.go
- keep non-profile tests (pinning, symlinks, hidden files) in add_test.go
* chore: update to rebased boxo and go-ipfs-cmds PRs
* docs: add HAMT threshold fix details to changelog
* feat(mfs): use Import config for CID version and hash function
make MFS commands (files cp, files write, files mkdir, files chcid)
respect Import.CidVersion and Import.HashFunction config settings
when CLI options are not explicitly provided.
also add tests for:
- files write respects Import.UnixFSRawLeaves=true
- single-block file: files write produces same CID as ipfs add
- updated comments clarifying CID parity with ipfs add
* feat(files): wire Import.UnixFSChunker and UnixFSDirectoryMaxLinks to MFS
`ipfs files` commands now respect these Import.* config options:
- UnixFSChunker: configures chunk size for `files write`
- UnixFSDirectoryMaxLinks: triggers HAMT sharding in `files mkdir`
- UnixFSHAMTDirectorySizeEstimation: controls size estimation mode
previously, MFS used hardcoded defaults ignoring user config.
changes:
- config/import.go: add UnixFSSplitterFunc() returning chunk.SplitterGen
- core/node/core.go: pass chunker, maxLinks, sizeEstimationMode to
mfs.NewRoot() via new boxo RootOption API
- core/commands/files.go: pass maxLinks and sizeEstimationMode to
mfs.Mkdir() and ensureContainingDirectoryExists(); document that
UnixFSFileMaxLinks doesn't apply to files write (trickle DAG limitation)
- test/cli/files_test.go: add tests for UnixFSDirectoryMaxLinks and
UnixFSChunker, including CID parity test with `ipfs add --trickle`
related: boxo@54e044f1b265
* feat(files): wire Import.UnixFSHAMTDirectoryMaxFanout and UnixFSHAMTDirectorySizeThreshold
wire remaining HAMT config options to MFS root:
- Import.UnixFSHAMTDirectoryMaxFanout via mfs.WithMaxHAMTFanout
- Import.UnixFSHAMTDirectorySizeThreshold via mfs.WithHAMTShardingSize
add CLI tests:
- files mkdir respects Import.UnixFSHAMTDirectoryMaxFanout
- files mkdir respects Import.UnixFSHAMTDirectorySizeThreshold
- config change takes effect after daemon restart
add UnixFSHAMTFanout() helper to test harness
update boxo to ac97424d99ab90e097fc7c36f285988b596b6f05
* fix(mfs): single-block files in CIDv1 dirs now produce raw CIDs
problem: `ipfs files write` in CIDv1 directories wrapped single-block
files in dag-pb even when raw-leaves was enabled, producing different
CIDs than `ipfs add --raw-leaves` for the same content.
fix: boxo now collapses single-block ProtoNode wrappers (with no
metadata) to RawNode in DagModifier.GetNode(). files with mtime/mode
stay as dag-pb since raw blocks cannot store UnixFS metadata.
also fixes sparse file writes where writing past EOF would lose data
because expandSparse didn't update the internal node pointer.
updates boxo to v0.36.1-0.20260203003133-7884ae23aaff
updates t0250-files-api.sh test hashes to match new behavior
* chore(test): use Go 1.22+ range-over-int syntax
* chore: update boxo to c6829fe26860
- fix typo in files write help text
- update boxo with CI fixes (gofumpt, race condition in test)
* chore: update go-ipfs-cmds to 192ec9d15c1f
includes binary content types fix: gzip, zip, vnd.ipld.car, vnd.ipld.raw,
vnd.ipfs.ipns-record
* chore: update boxo to 0a22cde9225c
includes refactor of maxLinks check in addLinkChild (review feedback).
* ci: fix helia-interop and improve caching
skip '@helia/mfs - should have the same CID after creating a file' test
until helia implements IPIP-499 (tracking: https://github.com/ipfs/helia/issues/941)
the test fails because kubo now collapses single-block files to raw CIDs
while helia explicitly uses reduceSingleLeafToSelf: false
changes:
- run aegir directly instead of helia-interop binary (binary ignores --grep flags)
- cache node_modules keyed by @helia/interop version from npm registry
- skip npm install on cache hit (matches ipfs-webui caching pattern)
* chore: update boxo to 1e30b954
includes latest upstream changes from boxo main
* chore: update go-ipfs-cmds to 1b2a641ed6f6
* chore: update boxo to f188f79fd412
switches to boxo@main after merging https://github.com/ipfs/boxo/pull/1088
* chore: update go-ipfs-cmds to af9bcbaf5709
switches to go-ipfs-cmds@master after merging https://github.com/ipfs/go-ipfs-cmds/pull/315
---------
Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com>
* feat(dns): resolve libp2p.direct addresses locally without network I/O
p2p-forge hostnames encode IP addresses directly (e.g., 1-2-3-4.peerID.libp2p.direct -> 1.2.3.4),
so DNS queries are wasteful. kubo now parses these IPs in-memory.
- applies to both default libp2p.direct and custom AutoTLS.DomainSuffix
- TXT queries still delegate to network for ACME DNS-01 compatibility
- https://github.com/ipfs/kubo/pull/11140#discussion_r2683477754
use fallback to network DNS instead of returning errors when local
parsing fails, ensuring forward compatibility with future DNS records
- https://github.com/ipfs/kubo/pull/11140#discussion_r2683512408
add peerID validation using peer.Decode(), matching libp2p.direct
server behavior, with fallback on invalid peerID
- https://github.com/ipfs/kubo/pull/11140#discussion_r2683521930
document interaction with DNS.Resolvers in config.md
- https://github.com/ipfs/kubo/pull/11140#discussion_r2683526647
add AutoTLS.SkipDNSLookup config flag to disable local resolution
(useful for debugging or custom DNS override scenarios)
- https://github.com/ipfs/kubo/pull/11140#discussion_r2683533462
add E2E test verifying libp2p.direct resolves locally even when
DNS.Resolvers points to a broken server
additional improvements:
- use madns.BasicResolver interface instead of custom basicResolver
- add compile-time interface checks for p2pForgeResolver and madns.Resolver
- refactor tests: merge IPv4/IPv6, add helpers, use config.DefaultDomainSuffix
- improve changelog to explain public good benefit (reducing DNS load)
Fixes#11136
* feat(config): add Gateway.MaxRequestDuration option
exposes the previously hardcoded 1 hour gateway request deadline as a
configurable option, allowing operators to adjust it to fit deployment
needs. protects gateway from edge cases and slow client attacks.
boxo: https://github.com/ipfs/boxo/pull/1079
* test(gateway): add MaxRequestDuration integration test
verifies config is wired correctly and 504 is returned when exceeded
* docs: add MaxRequestDuration to gateway production guide
---------
Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com>
This allows Kubo to respond to the GetClosestPeers() http routing v1 endpoint
as spec'ed here: https://github.com/ipfs/specs/pull/476
It is based on work from https://github.com/ipfs/boxo/pull/1021
We let IpfsNode implmement the contentRouter.Client interface with the new
method. We use our WAN-DHT to get the closest peers.
Additionally, Routing V1 HTTP API is exposed by default which enables light clients in browsers to use Kubo Gateway as delegated routing backend
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* fix(add): respect Provide config in fast-provide-root
fast-provide-root should honor the same config settings as the regular
provide system:
- skip when Provide.Enabled is false
- skip when Provide.DHT.Interval is 0
- respect Provide.Strategy (all/pinned/roots/mfs/combinations)
This ensures fast-provide only runs when appropriate based on user
configuration and the nature of the content being added (pinned vs
unpinned, added to MFS or not).
* feat(config): options to adjust global defaults
Add Import.FastProvideRoot and Import.FastProvideWait configuration options
to control default behavior of fast-provide-root and fast-provide-wait flags
in ipfs add command. Users can now set global defaults in config while
maintaining per-command flag overrides.
- Add Import.FastProvideRoot (default: true)
- Add Import.FastProvideWait (default: false)
- Add ResolveBoolFromConfig helper for config resolution
- Update docs with configuration details
- Add log-based tests verifying actual behavior
* refactor: extract fast-provide logic into reusable functions
Extract fast-provide logic from add command into reusable components:
- Add config.ShouldProvideForStrategy helper for strategy matching
- Add ExecuteFastProvide function reusable across add and dag import commands
- Move DefaultFastProvideTimeout constant to config/provide.go
- Simplify add.go from 72 lines to 6 lines for fast-provide
- Move fast-provide tests to dedicated TestAddFastProvide function
Benefits:
- cleaner API: callers only pass content characteristics
- all strategy logic centralized in one place
- better separation of concerns
- easier to add fast-provide to other commands in future
* feat(dag): add fast-provide support for dag import
Adds --fast-provide-root and --fast-provide-wait flags to `ipfs dag import`,
mirroring the fast-provide functionality available in `ipfs add`.
Changes:
- Add --fast-provide-root and --fast-provide-wait flags to dag import command
- Implement fast-provide logic for all root CIDs in imported CAR files
- Works even when --pin-roots=false (strategy checked internally)
- Share ExecuteFastProvide implementation between add and dag import
- Move ExecuteFastProvide to cmdenv package to avoid import cycles
- Add logging when fast-provide is disabled
- Conditional error handling: return error when wait=true, warn when wait=false
- Update config docs to mention both ipfs add and ipfs dag import
- Update changelog to use "provide" terminology and include dag import examples
- Add comprehensive test coverage (TestDagImportFastProvide with 6 test cases)
The fast-provide feature allows immediate DHT announcement of root CIDs
for faster content discovery, bypassing the regular background queue.
* docs: improve fast-provide documentation
Refine documentation to better explain fast-provide and sweep provider working
together, and highlight the performance improvement.
Changelog:
- add fast-provide to sweep provider features list
- explain performance improvement: root CIDs discoverable in <1s vs 30+ seconds
- note this uses optimistic DHT operations (faster with sweep provider)
- simplify examples, point to --help for details
Config docs:
- fix: --fast-provide-roots should be --fast-provide-root (singular)
- clarify Import.FastProvideRoot focuses on root CIDs while sweep handles all blocks
- simplify Import.FastProvideWait description
Command help:
- ipfs add: explain sweep provider context upfront
- ipfs dag import: add fast-provide explanation section
- both explain the split: fast-provide for roots, sweep for all blocks
* test: add tests for ShouldProvideForStrategy
add tests covering all provide strategy combinations with focus on
bitflag OR logic (the else-if bug fix). organized by behavior:
- all strategy always provides
- single strategies match only their flag
- combined strategies use OR logic
- zero strategy never provides
* refactor: error cmd on error and wait=true
change ExecuteFastProvide() to return error, enabling proper error
propagation when --fast-provide-wait=true. in sync mode, provide
failures now error the command as expected. in async mode (default),
always returns nil with errors logged in background goroutine.
also remove duplicate ExecuteFastProvide() from provide.go (75 lines),
keeping single implementation in cmdenv/env.go for reuse across add
and dag import commands.
call sites simplified:
- add.go: check and propagate error from ExecuteFastProvide
- dag/import.go: return error from ForEach callback, remove confusing
conditional error handling
semantics:
- precondition skips (DHT unavailable, etc): return nil (not failure)
- async mode (wait=false): return nil, log errors in goroutine
- sync mode (wait=true): return wrapped error on provide failure
adds Gateway.MaxRangeRequestFileSize configuration to protect against CDN bugs
where range requests over certain sizes return entire files instead of requested
byte ranges, causing unexpected bandwidth costs.
- default: 0 (no limit)
- returns 501 Not Implemented for oversized range requests
- protects against CDNs like Cloudflare that ignore range requests over 5GiB
also introduces OptionalBytes type to reduce code duplication when handling
byte-size configuration values, replacing manual string parsing with humanize.ParseBytes.
migrates existing byte-size configs to use this new type.
Fixes: https://github.com/ipfs/boxo/issues/856
Increase default Provide.DHT.MaxProvideConnsPerWorker value to match the
DHT replication factor (16 -> 20). A similar value is used in legacy
systems (with and without accelerated DHT client).
* fix: add MFS operation limit for --flush=false
adds a global counter that tracks consecutive MFS operations performed
with --flush=false and fails with clear error after limit is reached.
this prevents unbounded memory growth while avoiding the data corruption
risks of auto-flushing.
- adds Internal.MFSNoFlushLimit config
- operations fail with actionable error at limit
- counter resets on successful flush or any --flush=true operation
- operations with --flush=true reset and don't count
this commit removes automatic flush from https://github.com/ipfs/kubo/pull/10971
and instead errors to encourage users of --flush=false to develop a habit
of calling 'ipfs files flush' periodically.
boxo will no longer auto-flush (https://github.com/ipfs/boxo/pull/1041) to
avoid corruption issues, and kubo applies the limit to 'ipfs files' commands
instead.
closes#10842
* test: add tests for MFSNoFlushLimit
tests verify the new Internal.MFSNoFlushLimit config option:
- default limit of 256 operations
- custom limit configuration
- counter reset on flush=true
- counter reset on explicit flush command
- limit=0 disables the feature
- multiple MFS command types count towards limit
* docs: explain why MFS operations fail instead of auto-flushing
addresses feedback from https://github.com/ipfs/kubo/pull/10985#pullrequestreview-3256250970
- clarify that automatic flushing at limit was considered but rejected
- explain the data corruption risks of auto-flushing
- guide users who want auto-flush to use --flush=true (default)
- document benefits of explicit failure for batch operations
* feat: allow custom http provide when offline
* refactor: improve offline HTTP provider handling and tests
- fixed comment/function name mismatch
- added mock server test for HTTP provide success
- clarified test names for offline scenarios
* test: simplify single-node provider tests
use h.NewNode().Init() instead of NewNodes(1) for cleaner test setup
* fix: allow SweepingProvider to work with HTTP-only routing
when no DHT is available but HTTP routers are configured for providing,
return NoopProvider instead of failing. this allows the daemon to start
and HTTP-based providing to work through the routing system.
moved HTTP provider detection to config package as HasHTTPProviderConfigured()
for better code organization and reusability.
this fix is important as SweepingProvider will become the new default in the future.
---------
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* docs: improve slow reprovide warning messages
simplify warning text and provide actionable solutions in order of preference
* feat(config): add validation for Provide.DHT settings
- validate interval doesn't exceed DHT record validity (48h)
- validate worker counts and other parameters are within valid ranges
- improve slow reprovide warning messages to reference config parameter
- add tests for all validation cases
* docs: add reprovide cycle visualization
shows traffic patterns of legacy vs sweep vs accelerated DHT
* fix: prevent --flush=false in 'ipfs files rm' command
the 'ipfs files rm' command always flushes for safety to ensure
data integrity. this change adds an explicit error when users
try to pass --flush=false, improving ux and preventing confusion.
related to #10842
* fix: add MFS cache size limit to prevent unbounded growth
- add Internal.MFSAutoflushThreshold config (experimental)
- directories auto-flush when cache exceeds threshold with --flush=false
- prevents high memory usage issue from #10842
- default: 256 entries per directory (matching HAMT shard size)
- set to 0 to restore old behavior (risky, may cause errors)
Closes#10842
* refactor: consolidate Provider/Reprovider into unified Provide config
- merge Provider and Reprovider configs into single Provide section
- add fs-repo-17-to-18 migration for config consolidation
- improve migration ergonomics with common package utilities
- convert deprecated "flat" strategy to "all" during migration
- improve Provide docs
* docs: add total_provide_count metric guidance
- document how to monitor provide success rates via prometheus metrics
- add performance comparison section to changelog
- explain how to evaluate sweep vs legacy provider effectiveness
* fix: add OpenTelemetry meter provider for metrics
- set up meter provider with Prometheus exporter in daemon
- enables metrics from external libs like go-libp2p-kad-dht
- fixes missing total_provide_count_total when SweepEnabled=true
- update docs to reflect actual metric names
---------
Co-authored-by: gammazero <11790789+gammazero@users.noreply.github.com>
Co-authored-by: guillaumemichel <guillaume@michel.id>
Co-authored-by: Daniel Norman <1992255+2color@users.noreply.github.com>
Co-authored-by: Hector Sanjuan <code@hector.link>
* reprovide sweep draft
* update reprovider dep
* go mod tidy
* fix provider type
* change router type
* dual reprovider
* revert to provider.System
* back to start
* SweepingReprovider test
* fix nil pointer deref
* noop provider for nil dht
* disabled initial network estimation
* another iteration
* suppress missing self addrs err
* silence empty rt err on lan dht
* comments
* new attempt at integrating
* reverting changes in core/node/libp2p/routing.go
* removing SweepingProvider
* make reprovider optional
* add noop reprovider
* update KeyChanFunc type alias
* restore boxo KeyChanFunc
* fix missing KeyChanFunc
* test(sharness): PARALLEL=1 and timeout 30m
running sequentially to see where timeout occurs
* initialize MHStore
* revert workflow debug
* config
* config docs
* merged IpfsNode provider and reprovider
* move Provider interface to from kad-dht to node
* moved Provider interface from kad-dht to kubo/core/node
* mod_tidy
* Add Clear to Provider interface
* use latest kad-dht commit
* make linter happy
* updated boxo provide interface
* boxo PR fix
* using latest kad-dht commit
* use latest boxo release
* fix fx
* fx cyclic deps
* fix merge issues
* extended tests
* don't provide LAN DHT
* docs
* restore dual dht provider
* don't start provider before it is online
* address linter
* dual/provider fix
* add delay in provider tests for dht bootstrap
* add OfflineDelay parameter to config
* remove increase number of workers in test
* improved keystore gc process
* fix: replace incorrect logger import in coreapi
replaced github.com/labstack/gommon/log with the standard
github.com/ipfs/go-log/v2 logger used throughout kubo.
removed unused labstack dependency from go.mod files.
* fix: remove duplicate WithDefault call in provider config
* fix: use correct option method for burst workers
* fix: improve error messages for experimental sweeping provider
updated error messages to clearly indicate when commands are unavailable
due to experimental sweeping provider being enabled via Reprovider.Sweep.Enabled=true
* docs: remove obsolete KeyStoreGCInterval config
removed from config.md as option no longer exists (removed in b540fba1a)
updated keystore description to reflect gc happens at reprovide interval
* docs: add TODO placeholder changelog for experimental sweeping DHT provider
using v0.38-TODO.md name to avoid merge conflicts with master branch
and allow CI tests to run. will be renamed to v0.38.md once config
migration is added to the PR
* fix: provideKeysRec go routine
* clear keystore on close
* fix: datastore prefix
* fix: improve error handling in provideKeysRec
- close errCh channel to distinguish between nil and pending errors
- check for pending errors when provided.New closes
- handle context cancellation during error send
- prevent race condition where errors could be silently lost
this ensures DAG walk errors are always propagated correctly
* address gammazero's review
* rename BurstProvider to LegacyProvider
* use latest provider/keystore
* boxo: make mfs StartProviding async
* bump boxo
* chore: update boxo to f2b4e12fb9a8ac138ccb82aae3b51ec51d9f631c
- updated boxo dependency to specified commit
- updated go.mod and go.sum files across all modules
* use latest kad-dht/boxo
* Buffered SweepingProvider wrapper
* use latest kad-dht commit
* allow no DHT router
* use latest kad-dht & boxo
---------
Co-authored-by: Marcin Rataj <lidel@lidel.org>
Co-authored-by: gammazero <11790789+gammazero@users.noreply.github.com>
validates Import configuration fields to prevent invalid values:
- CidVersion: must be 0 or 1
- UnixFSFileMaxLinks: must be positive
- UnixFSDirectoryMaxLinks: must be non-negative
- UnixFSHAMTDirectoryMaxFanout: power of 2, multiple of 8, ≤ 1024
- BatchMaxNodes/BatchMaxSize: must be positive
- UnixFSChunker: validates format patterns
- HashFunction: must be allowed by verifcid
* Tests: disable telemetry in tests by default
Disable the plugin in cli tests and sharness by default. Enable only in
telemetry tests.
There are cases when tests get stuck or get killed and leave daemons hanging around. We don't want to be getting telemetry from those.
* sharness: attempt to fix
* sharness: add missing --bool flag
* fix(ci): add omitempty to Plugin.Config field
The sharness problem is that when the telemetry plugin is configured
initially with 'ipfs config --bool', it creates a structure without
the 'Config: null' field, but when the config is copied and replaced,
it expects the structure to be preserved.
Adding omitempty ensures the Config field is omitted from JSON when
nil, making the config structure consistent between initial creation
and replacement operations.
---------
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* Reprovider strategy: rename "flat" to "all".
Value "flat" now parses to "all". Behaviour from "all" removed.
Fixes#10864 which has detailed explanation.
* core/node/provider.go: remove unused function mfsRootProvider
It was used in the "all" strategy.
* docs: improve reprovider.strategy=all changelog framing
- highlight memory efficiency improvements
- clarify this removes v0.28 workaround
- update config.md memory requirements
- fix announce-on profile typo
* feat: deprecate Reprovider.Strategy=flat
- add deprecation warning in daemon.go when flat strategy is detected
- document that flat is deprecated in ParseReproviderStrategy comment
- add explicit test case for flat -> all mapping
- flat continues to work but users are warned to migrate to all
---------
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* feat(gateway): concurrency and timeout limits
Depends on https://github.com/ipfs/boxo/pull/994
* chore: boxo master with final boxo#994
this includes race-condition fixes from ipfs/boxo#994
and increased `DefaultMaxConcurrentRequests = 4096`
* docs: concise config.md and changelog
* Provide according to strategy
Updates boxo to a version with the changes from https://github.com/ipfs/boxo/pull/976, which decentralize the providing responsibilities (from a central providing.Exchange to blockstore, pinner, mfs).
The changes consist in initializing the Pinner, MFS and the blockstore with the provider.System, which is created first.
Since the provider.System is created first, the reproviding KeyChanFunc is set
later when we can create it once we have the Pinner, MFS and the blockstore.
Some additional work applies to the Add() workflow. Normally, blocks would get provided at the Blockstore or the Pinner, but when adding blocks AND a "pinned" strategy is used, the blockstore does not provide, and the
pinner does not traverse the DAG (and thus doesn't provide either), so we need to provide directly from the Adder. This is resolved by wrapping the DAGService in a "providingDAGService" which provides every added block, when using the "pinned" strategy.
`ipfs --offline add` when the ONLINE daemon is running will now announce blocks per the chosen strategy, where before it did not announce them. This is documented in the changelog. A couple of releases ago, adding with `ipfs --offline add` was faster, but this is no longer the case so we are not incurring in any penalties by sticking to the fact that the daemon is online and has a providing strategy that we follow.
Co-authored-by: gammazero <11790789+gammazero@users.noreply.github.com>
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* feat(add): add support for naming pinned CID
Signed-off-by: kapil <kapilsareen584@gmail.com>
* fix(add): no double pinning and simplify pin-name
- modify PinRoot to accept name parameter, eliminating double pinning
- remove automatic filename fallback logic for cleaner behavior
- only create named pins when explicitly requested via --pin-name=value
- replace NoPinName constant with idiomatic empty string literals
- Update help text and tests to reflect explicit-only behavior
* docs: changelog
* chore: lint
* test: negative case for empty pin-name
* chore: gofmt
---------
Signed-off-by: kapil <kapilsareen584@gmail.com>
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* Configure bitswap braodcast reduction
Add new config items to `Internal.Bitswap` to allow configuration of bitswap broadcast reduction behavior. Broadcast reduction behavior is enabled by default, and uses settings that should be suitable for most installations of kubo.
* update sharness metrics test
* Explicit defaults for broadcast reduction configuration
* Update docs/config.md
* fix(config): explicit Provider.Enabled flag
Adds missing config option described in
https://github.com/ipfs/kubo/issues/10803
* refactor: remove Experimental.StrategicProviding
removing experiment, replaced with Provider.Enabled
* test(cli): routing [re]provide
updated and added tests for manually triggering provide and reprovide
and making them respect global configuration flag to avoid
inconsistent behaviors
* docs: improve DelegatedRouters
* refactor: default DefaultProviderWorkerCount=16
- simplified default for both
- 16 is safer for non-accelerated DHT client
- acceletated DHT performs better without limit anyway - updated docs
* Add MFS command line options, extend existing mount functions for MFS, set defaults.
* Directory listing and file stat.
* Add a read-only MFS view.
* Add mkdir and interface checks.
* Add remove and rename functionality.
* Implement all required write interfaces.
* Adjust mount functions for other architechtures.
* Merge branch 'master' into feat/10710-mfs-fuse-mount
* Write a basic read/write test.
* Write more basic tests, add a mutex to the file object, fix modtime.
* Add a concurrency test, remove mutexes from file and directory structures.
* Refactor naming(mfdir -> mfsdir) and add documentation.
* Add CID retrieval through ipfs_cid xattr.
* Add docs, add xattr listing, fix bugs for mv and stat, refactor.
* Add MFS command line options, extend existing mount functions for MFS, set defaults.
* docs phrasing
* docs: Mounts.MFS
* docs: warn about lazy-loaded DAGs
* test: TEST_FUSE=1 ./t0030-mount.sh -v
---------
Co-authored-by: Guillaume Michel <guillaumemichel@users.noreply.github.com>
Co-authored-by: guillaumemichel <guillaume@michel.id>
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* Feat: http retrieval as experimental feature
This introduces the http-retrieval capability as an experimental feature.
It can be enabled in the configuration `Experimental.HTTPRetrieval.Enabled = true`.
Documentation and changelog to be added later.
* refactor: HTTPRetrieval.Enabled as Flag
* docs(config): HTTPRetrieval section
* refactor: reusable MockHTTPContentRouter
* feat: HTTPRetrieval.TLSInsecureSkipVerify
allows self-signed certificates in tests
* feat(config): HTTPRetrieval.MaxBlockSize
* test: end-to-end HTTPRetrieval.Enabled
this spawns two http services on localhost:
1. HTTP router that returns HTTP provider when /routing/v1/providers/cid i queried
2. HTTP provider that returns a block when /ipfs/cid is queried
3. Configures Kubo to use (1) instead of cid.contact
this seems to work (running test with DEBUG=true shows (1) was queried
for the test CID and returned multiaddr of (2), but Kubo never requested
test CID block from (2) – needs investigation
* fix: enable /routing/v1/peers for non-cid.contact
we artificially limited every delegated routing endpoint because of
cid.contact being limited to one endpoint
* feat: Routing.DelegatedRouters
make it easy to override the hardcoded implicit HTTP routeur URL
without having to set the entire custom Router.Routers and
Router.Methods
(http_retrieval_client_test.go still needs to be fixed in future commit)
* test: flag remaining work
* docs: review feedback
* refactor: providerQueryMgr with bitswapNetworks
this fixes two regressions:
(1) introduced in https://github.com/ipfs/kubo/issues/10717
where we only used bitswapLib2p query manager
(this is why E2E did not act on http provider)
(2) introduced in https://github.com/ipfs/kubo/pull/10765
where it was not possible to set binary peerID in IgnoreProviders
(we changed to []string)
* refactor: Bitswap.Libp2pEnabled
replaces Bitswap.Enabled with Bitswap.Libp2pEnabled
adds tests that confirm it is possible to disable libp2p bitswap fully
and only keep http in client mode
also, removes the need for passing empty blockstore in client-only mode
* docs: changelog
---------
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* feat: add Bitswap configuration and related tests
* fix: update Bitswap function to use 'provide' parameter for server enablement
* docs: update changelog for Bitswap functionality changes
* fix: update Bitswap server enablement logic and improve related tests
* fix: rename BitswapConfig to Bitswap and update references
* docs: config and changelog
* fix: `ipfs cat` panic when `Bitswap.Enabled=false`
Fixes panic described in:
https://github.com/ipfs/kubo/pull/10782#discussion_r2069116219
---------
Co-authored-by: gystemd <gystemd@gmail.com>
Co-authored-by: gammazero <11790789+gammazero@users.noreply.github.com>
Co-authored-by: Giulio Piva <giulio.piva@dedicated.world>
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* datastore: metrics optional and off by default
When ipfs is initialized, the datastore metrics wrapper is not configured by default as it previously was. To enable datastore metrics during initialization, specifying the appropriate `--profile` option.
To enable datastore metrics tracking wrapper, initialize with datastore profile name + "-measure" suffix. For example:
```
ipfs init --profile flatfs-measure
```
Closes#10767
* fix sharness tests for new datastore dafaults
* Add sharness test to check metrics added by flatfs-measure profile
* Document updated metrics in changelog
* update config doc with new profiles
* docs(changelog): separate section
* initialize non-measure pebbleds with FormatMajorVersion config
* docs: fix typos, add docs link
---------
Co-authored-by: gammazero <gammazero@users.noreply.github.com>
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* Upgrade to pebble v2.0.3
- Configure latest pebble database format at init
- Do not automatically ratchet database format if set in config
- Daemon messge about new available pebble format
- Document pebble config with formatMajorVersion
- Add warning to users running badger, nudging them to switch to flatfs or pebble
- docs: explain Pebble's `FormatMajorVersion`
- Use pebbleds instead of badgerds in t0060-daemon.sh
- Print badgerds warning message to stderr
* adjust ipfs stats provide
* update boxo dep
* bump boxo
* fixing tests
* docs/chore: mark stat reprovide as experimental
* docs: Provider.Strategy
explicitly document it is not used - without this legacy users will have
it in their config and be very confused
---------
Co-authored-by: Marcin Rataj <lidel@lidel.org>
Adds `Routing.IgnoreProviders`.
This requires initializing a custom providerQueryManager and using it instead
of the default created internally in Bitswap. Since the default is created
with some internal default configuration options (MaxProviders), this hardcodes it.
* feat(libp2p): enable shared TCP listeners
* docs: switch mentions of /ws to /tcp/4001
* feat: AutoTLS.AutoWSS
This adds AutoTLS.AutoWSS flag that is set to true by default.
It will check if Addresses.Swarm contain explicit /ws listener,
and if not found, it will append one per every /tcp listener
This way existing TCP ports are reused without any extra configuration,
but we don't break user's who have custom / explicit /ws listener
already.
I also moved logger around, to include Addresses.Swarm inspection
results in `autotls` logger.
* chore: go-libp2p v0.38.1
https://github.com/libp2p/go-libp2p/releases/tag/v0.38.0https://github.com/libp2p/go-libp2p/releases/tag/v0.38.1
* docs: AutoTLS.AutoWSS and go-libp2p v0.38.x
* chore: p2p-forge/client v0.2.0
https://github.com/ipshipyard/p2p-forge/releases/tag/v0.2.0
* fix: disable libp2p.ShareTCPListener() in PNET
* chore(ci): timeout sharness after 15m
average successful run is <9 minutes, no need to wait for 20
https://github.com/ipfs/kubo/actions/workflows/sharness.yml?query=is%3Asuccess
---------
Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com>
Co-authored-by: Marcin Rataj <lidel@lidel.org>
