* fix: add MFS operation limit for --flush=false
adds a global counter that tracks consecutive MFS operations performed
with --flush=false and fails with clear error after limit is reached.
this prevents unbounded memory growth while avoiding the data corruption
risks of auto-flushing.
- adds Internal.MFSNoFlushLimit config
- operations fail with actionable error at limit
- counter resets on successful flush or any --flush=true operation
- operations with --flush=true reset and don't count
this commit removes automatic flush from https://github.com/ipfs/kubo/pull/10971
and instead errors to encourage users of --flush=false to develop a habit
of calling 'ipfs files flush' periodically.
boxo will no longer auto-flush (https://github.com/ipfs/boxo/pull/1041) to
avoid corruption issues, and kubo applies the limit to 'ipfs files' commands
instead.
closes#10842
* test: add tests for MFSNoFlushLimit
tests verify the new Internal.MFSNoFlushLimit config option:
- default limit of 256 operations
- custom limit configuration
- counter reset on flush=true
- counter reset on explicit flush command
- limit=0 disables the feature
- multiple MFS command types count towards limit
* docs: explain why MFS operations fail instead of auto-flushing
addresses feedback from https://github.com/ipfs/kubo/pull/10985#pullrequestreview-3256250970
- clarify that automatic flushing at limit was considered but rejected
- explain the data corruption risks of auto-flushing
- guide users who want auto-flush to use --flush=true (default)
- document benefits of explicit failure for batch operations
(cherry picked from commit a688b7eeac)
* Filestore: provide Filestore nodes
When strategy is set to "all" (the blockstore does all the providing when a
block is written), no providing was happening to Filestore blocks that were
not written to the underlying blockstore (so, the DAG leaves, as they live in
the filesystem directly). This fixes that.
* docs: clarify filestore and urlstore fix in changelog
both filestore (local file references) and urlstore (HTTP/HTTPS URL
references) blocks are now properly provided shortly after initial add
(cherry picked from commit f63887ae96)
* fix: SweepingProvider shouldn't error when missing DHT
* fix: prevent panic when SweepingProvider has no DHT
when SweepingProvider is enabled but no DHT is available (e.g., Routing.Type=none),
the daemon would panic with a nil pointer dereference in ResettableKeystore.ResetCids.
this fix:
- returns NoopProvider when no DHT implementation is available
- skips keystore initialization for NoopProvider to avoid unnecessary operations
- allows nodes to run without DHT when using HTTP-only routing or offline mode
the panic occurred because initKeyStore tried to access a nil keystore when
SweepingProvider returned nil for the keystore parameter. by checking if the
provider is NoopProvider and skipping keystore operations, we avoid the panic
while maintaining correct behavior for all other provider types.
cc #10974#10975
---------
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* feat: allow custom http provide when offline
* refactor: improve offline HTTP provider handling and tests
- fixed comment/function name mismatch
- added mock server test for HTTP provide success
- clarified test names for offline scenarios
* test: simplify single-node provider tests
use h.NewNode().Init() instead of NewNodes(1) for cleaner test setup
* fix: allow SweepingProvider to work with HTTP-only routing
when no DHT is available but HTTP routers are configured for providing,
return NoopProvider instead of failing. this allows the daemon to start
and HTTP-based providing to work through the routing system.
moved HTTP provider detection to config package as HasHTTPProviderConfigured()
for better code organization and reusability.
this fix is important as SweepingProvider will become the new default in the future.
---------
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* docs: improve slow reprovide warning messages
simplify warning text and provide actionable solutions in order of preference
* feat(config): add validation for Provide.DHT settings
- validate interval doesn't exceed DHT record validity (48h)
- validate worker counts and other parameters are within valid ranges
- improve slow reprovide warning messages to reference config parameter
- add tests for all validation cases
* docs: add reprovide cycle visualization
shows traffic patterns of legacy vs sweep vs accelerated DHT
* ci: optimize build workflows
- use go version from go.mod instead of hardcoding
- group platforms by OS for parallel builds
- remove legacy try-build targets
* fix: checkout before setup-go in all workflows
setup-go needs go.mod to be present, so checkout must happen first
* chore: remove deprecated // +build syntax
go 1.17+ uses //go:build, the old syntax is no longer needed
* simplify: remove nofuse tag from CI workflows
- workflows now rely on platform build constraints
- keep make nofuse target for manual builds
- remove unused appveyor.yml
* ci: remove legacy travis variable and fix gateway-conformance
- remove TRAVIS env variable from 4 workflows
- fix gateway-conformance checkout path to match working-directory
- replace deprecated cache-go-action with built-in setup-go caching
* fix(webui): show helpful errors for incompatible configurations
- show error when Gateway.NoFetch=true and WebUI is not available locally
- show error when Gateway.DeserializedResponses=false (incompatible)
- add tests for both error scenarios
* chore(webui): update to v4.9.0
https://github.com/ipfs/ipfs-webui/releases/tag/v4.9.0
* docs: add WebUI v4.9.0 update to v0.38 changelog
- highlight new diagnostics screen for troubleshooting
- include screenshots of key features in table format
- add local access URL for WebUI
- update TOC with new sections
* fix: prevent --flush=false in 'ipfs files rm' command
the 'ipfs files rm' command always flushes for safety to ensure
data integrity. this change adds an explicit error when users
try to pass --flush=false, improving ux and preventing confusion.
related to #10842
* fix: add MFS cache size limit to prevent unbounded growth
- add Internal.MFSAutoflushThreshold config (experimental)
- directories auto-flush when cache exceeds threshold with --flush=false
- prevents high memory usage issue from #10842
- default: 256 entries per directory (matching HAMT shard size)
- set to 0 to restore old behavior (risky, may cause errors)
Closes#10842
* fix: use CheckIfPinnedWithType for pin ls with names
updates to use CheckIfPinnedWithType method from https://github.com/ipfs/boxo/pull/1035,
enabling efficient pin name retrieval for 'ipfs pin ls <cid> --names'
- uses new CheckIfPinnedWithType from boxo for type-specific pin checks
- pin names are now returned when listing specific CIDs with --names flag
* test: add CLI tests for pin ls with names
tests cover:
- pin ls with specific CIDs returning names
- pin ls without CID listing all pins with names
- pin ls with --type and --names combinations
- JSON output with and without names
- pin update preserving names
- error cases (invalid CID, unpinned CID)
* docs: add pin name improvements to v0.38 changelog
covers fix for ipfs pin ls --names with specific CIDs
and RPC pin name leak fix
* fix(rpc): support pin names in Add()
passes the Name field from PinAddSettings to the API request
adds test to verify pin names work via RPC
* test: add coverage for pin names functionality
- test special characters, unicode, long names
- test concurrent operations
- test persistence across daemon restarts
- test garbage collection preservation
- fix indirect pin test logic
* chore: boxo@main with boxo#1039
* fix(pin): improve pin ls robustness and validation
- add nil check for n.Pinning with early fail-fast validation
- use pin.StringToMode() for consistent type validation
- add edge case tests for invalid types and unpinned CIDs
* refactor: consolidate Provider/Reprovider into unified Provide config
- merge Provider and Reprovider configs into single Provide section
- add fs-repo-17-to-18 migration for config consolidation
- improve migration ergonomics with common package utilities
- convert deprecated "flat" strategy to "all" during migration
- improve Provide docs
* docs: add total_provide_count metric guidance
- document how to monitor provide success rates via prometheus metrics
- add performance comparison section to changelog
- explain how to evaluate sweep vs legacy provider effectiveness
* fix: add OpenTelemetry meter provider for metrics
- set up meter provider with Prometheus exporter in daemon
- enables metrics from external libs like go-libp2p-kad-dht
- fixes missing total_provide_count_total when SweepEnabled=true
- update docs to reflect actual metric names
---------
Co-authored-by: gammazero <11790789+gammazero@users.noreply.github.com>
Co-authored-by: guillaumemichel <guillaume@michel.id>
Co-authored-by: Daniel Norman <1992255+2color@users.noreply.github.com>
Co-authored-by: Hector Sanjuan <code@hector.link>
* reprovide sweep draft
* update reprovider dep
* go mod tidy
* fix provider type
* change router type
* dual reprovider
* revert to provider.System
* back to start
* SweepingReprovider test
* fix nil pointer deref
* noop provider for nil dht
* disabled initial network estimation
* another iteration
* suppress missing self addrs err
* silence empty rt err on lan dht
* comments
* new attempt at integrating
* reverting changes in core/node/libp2p/routing.go
* removing SweepingProvider
* make reprovider optional
* add noop reprovider
* update KeyChanFunc type alias
* restore boxo KeyChanFunc
* fix missing KeyChanFunc
* test(sharness): PARALLEL=1 and timeout 30m
running sequentially to see where timeout occurs
* initialize MHStore
* revert workflow debug
* config
* config docs
* merged IpfsNode provider and reprovider
* move Provider interface to from kad-dht to node
* moved Provider interface from kad-dht to kubo/core/node
* mod_tidy
* Add Clear to Provider interface
* use latest kad-dht commit
* make linter happy
* updated boxo provide interface
* boxo PR fix
* using latest kad-dht commit
* use latest boxo release
* fix fx
* fx cyclic deps
* fix merge issues
* extended tests
* don't provide LAN DHT
* docs
* restore dual dht provider
* don't start provider before it is online
* address linter
* dual/provider fix
* add delay in provider tests for dht bootstrap
* add OfflineDelay parameter to config
* remove increase number of workers in test
* improved keystore gc process
* fix: replace incorrect logger import in coreapi
replaced github.com/labstack/gommon/log with the standard
github.com/ipfs/go-log/v2 logger used throughout kubo.
removed unused labstack dependency from go.mod files.
* fix: remove duplicate WithDefault call in provider config
* fix: use correct option method for burst workers
* fix: improve error messages for experimental sweeping provider
updated error messages to clearly indicate when commands are unavailable
due to experimental sweeping provider being enabled via Reprovider.Sweep.Enabled=true
* docs: remove obsolete KeyStoreGCInterval config
removed from config.md as option no longer exists (removed in b540fba1a)
updated keystore description to reflect gc happens at reprovide interval
* docs: add TODO placeholder changelog for experimental sweeping DHT provider
using v0.38-TODO.md name to avoid merge conflicts with master branch
and allow CI tests to run. will be renamed to v0.38.md once config
migration is added to the PR
* fix: provideKeysRec go routine
* clear keystore on close
* fix: datastore prefix
* fix: improve error handling in provideKeysRec
- close errCh channel to distinguish between nil and pending errors
- check for pending errors when provided.New closes
- handle context cancellation during error send
- prevent race condition where errors could be silently lost
this ensures DAG walk errors are always propagated correctly
* address gammazero's review
* rename BurstProvider to LegacyProvider
* use latest provider/keystore
* boxo: make mfs StartProviding async
* bump boxo
* chore: update boxo to f2b4e12fb9a8ac138ccb82aae3b51ec51d9f631c
- updated boxo dependency to specified commit
- updated go.mod and go.sum files across all modules
* use latest kad-dht/boxo
* Buffered SweepingProvider wrapper
* use latest kad-dht commit
* allow no DHT router
* use latest kad-dht & boxo
---------
Co-authored-by: Marcin Rataj <lidel@lidel.org>
Co-authored-by: gammazero <11790789+gammazero@users.noreply.github.com>
* fix: enforce identity CID size limits
- validate --inline-limit against verifcid.MaxDigestSize
- add error when --hash=identity exceeds size limit
- add tests for identity CID overflow scenarios
- update help text to show maximum inline limit
This prevents creation of unbounded identity CIDs by enforcing
the 128-byte limit defined in https://github.com/ipfs/boxo/pull/1018Fixes#6011
IPIP: https://github.com/ipfs/specs/pull/512
validates Import configuration fields to prevent invalid values:
- CidVersion: must be 0 or 1
- UnixFSFileMaxLinks: must be positive
- UnixFSDirectoryMaxLinks: must be non-negative
- UnixFSHAMTDirectoryMaxFanout: power of 2, multiple of 8, ≤ 1024
- BatchMaxNodes/BatchMaxSize: must be positive
- UnixFSChunker: validates format patterns
- HashFunction: must be allowed by verifcid
* Reprovider strategy: rename "flat" to "all".
Value "flat" now parses to "all". Behaviour from "all" removed.
Fixes#10864 which has detailed explanation.
* core/node/provider.go: remove unused function mfsRootProvider
It was used in the "all" strategy.
* docs: improve reprovider.strategy=all changelog framing
- highlight memory efficiency improvements
- clarify this removes v0.28 workaround
- update config.md memory requirements
- fix announce-on profile typo
* feat: deprecate Reprovider.Strategy=flat
- add deprecation warning in daemon.go when flat strategy is detected
- document that flat is deprecated in ParseReproviderStrategy comment
- add explicit test case for flat -> all mapping
- flat continues to work but users are warned to migrate to all
---------
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* fix(cmds): improve ipfs add --help clarity and organization
- clarify --raw-leaves behavior with CIDv0/v1 defaults
- add Import.* config references to all relevant options
- replace deprecated 'ipfs object links' with 'ipfs ls' in examples
- add section headers for better navigation (BASIC EXAMPLES, MFS INTEGRATION, etc.)
- group related options logically (output control, CID & hashing, experimental features)
- standardize experimental warnings format
- add MerkleDAG docs link and clarify chunking behavior
- fix incorrect Import.UnixFSHAMTThreshold reference
addresses confusion from #10918 where --raw-leaves appeared to have no effect
because CIDv1 automatically enables it by default
* docs: fix typos in Import.* configuration documentation
- fix 'chilcren' → 'children'
- fix 'HAMT directory have' → 'HAMT directories have'
- fix 'A HAMT is an structure' → 'A HAMT is a structure'
* Update core/commands/add.go
Co-authored-by: Daniel Norman <1992255+2color@users.noreply.github.com>
---------
Co-authored-by: Daniel Norman <1992255+2color@users.noreply.github.com>
test was expecting immediate GC lock acquisition after pipe close,
but timing wasn't guaranteed. replaced blocking wait with 5-second
timeout to handle timing variations while still detecting deadlocks.
* feat(gateway): concurrency and timeout limits
Depends on https://github.com/ipfs/boxo/pull/994
* chore: boxo master with final boxo#994
this includes race-condition fixes from ipfs/boxo#994
and increased `DefaultMaxConcurrentRequests = 4096`
* docs: concise config.md and changelog
* feat(ipns): Add a parameter in name.publish to change the sequence number
* test: monotonic name publish --sequence
* docs: `name publish --sequence`
* chore: boxo main with PR 962
---------
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* fix(relay): feed connected peers to AutoRelay discovery
Feed all connected swarm peers to AutoRelay as potential relay
candidates. This allows peers from HTTP routing and manual connections
to serve as relays, not just DHT-discovered peers.
Fixes#10899
* docs: changelog
* feat: update log level command to show log levels
* test: add log level tests
* update TestCommands test
* docs: relation to GOLOG_LOG_LEVEL
* chore: update to latest go-log
* fix: do not output single subsystem name in CLI
* test: explicit subsystem request dont output subsystem
* LevelFromString renamed to Parse
* Modify `ipfs log level`
* Denote default level with sdubsystem name '(defult)'.
* make "*" an dalias for "all". Test to make sure both work the same.
* Fix command stream result handling
Do not close the stream prematurely in a deferred function. Also, avoid possible shadowing errors.
Closes#9007
* test(cli): test/cli/cid_test.go
includes regression tests for https://github.com/ipfs/kubo/issues/9007
and better coverage than the old test/sharness/t0290-cid.sh
(identified bug in basemoji)
* fix: base256emoji in cid bases --prefix
changed the character display logic from ASCII range check to
unicode.IsPrint() to properly handle Unicode characters including
emojis.
* docs: changelog
---------
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* fix: `ipfs cid format` without repo
these commands should work without daemon or repo
but were missing SetDoesNotUseRepo(true)
* test: test/cli/commands_without_repo_test.go
* Provide according to strategy
Updates boxo to a version with the changes from https://github.com/ipfs/boxo/pull/976, which decentralize the providing responsibilities (from a central providing.Exchange to blockstore, pinner, mfs).
The changes consist in initializing the Pinner, MFS and the blockstore with the provider.System, which is created first.
Since the provider.System is created first, the reproviding KeyChanFunc is set
later when we can create it once we have the Pinner, MFS and the blockstore.
Some additional work applies to the Add() workflow. Normally, blocks would get provided at the Blockstore or the Pinner, but when adding blocks AND a "pinned" strategy is used, the blockstore does not provide, and the
pinner does not traverse the DAG (and thus doesn't provide either), so we need to provide directly from the Adder. This is resolved by wrapping the DAGService in a "providingDAGService" which provides every added block, when using the "pinned" strategy.
`ipfs --offline add` when the ONLINE daemon is running will now announce blocks per the chosen strategy, where before it did not announce them. This is documented in the changelog. A couple of releases ago, adding with `ipfs --offline add` was faster, but this is no longer the case so we are not incurring in any penalties by sticking to the fact that the daemon is online and has a providing strategy that we follow.
Co-authored-by: gammazero <11790789+gammazero@users.noreply.github.com>
Co-authored-by: Marcin Rataj <lidel@lidel.org>
- Move `ipfs stat reprovide` to `ipfs provide stat`
- Mark `ipfs stat provide` as deprecated and replaces by `ipfs provide stat`
- Mark `ipfs stat reprovide` as deprecated and replaces by `ipfs provide stat`
- Remove redundant code from deprecated subcommands
Closes#10869
* feat(add): add support for naming pinned CID
Signed-off-by: kapil <kapilsareen584@gmail.com>
* fix(add): no double pinning and simplify pin-name
- modify PinRoot to accept name parameter, eliminating double pinning
- remove automatic filename fallback logic for cleaner behavior
- only create named pins when explicitly requested via --pin-name=value
- replace NoPinName constant with idiomatic empty string literals
- Update help text and tests to reflect explicit-only behavior
* docs: changelog
* chore: lint
* test: negative case for empty pin-name
* chore: gofmt
---------
Signed-off-by: kapil <kapilsareen584@gmail.com>
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* refactor: remove goprocess
The `goprocess` package is no longer needed. It can be replaces by modern `context` and `context.AfterFunc`.
* mod tidy
* log unmount errors on shutdown
* Do not log non-mounted errors on shutdown
* Use WaitGroup associated with IPFS node to wait for services to whutdown
* Prefer explicit Close to context.ArterFunc
* Do not use node-level WaitGroup
* Unmount for non-supported platforms
* fix return values
* test: daemon shuts down gracefully
make sure ongoing operations dont block shutdown
* test(cli): add TestFUSE
* test: smarter RequiresFUSE
opportunistically run FUSE tests if env has fusermount
and TEST_FUSE was not explicitly set
* docs: changelog
---------
Co-authored-by: gammazero <gammazero@users.noreply.github.com>
Co-authored-by: Marcin Rataj <lidel@lidel.org>
No behaviour changes.
Currently we are using ProvideManyRouter for Bitswap, which is only meant to
use ContentDiscovery. This makes things more clear in that there is a
designated ContentDiscovery instance.
After boxo v0.33.1, this is a recommended step to fix http retrieval bugs.
Having a single ConnectEventManager prevents misdirected operations in the
network.Router to change the Connectedness state in a way that the counterpart
(httpnet or bsnet) can later correct.
* provider: clear reprovide queue when reprovide strategy changes
When the currently configured reprovide strategy does not match the previous strategy read from the datastore, then clear the reprovide queue and update the reprovide strategy that is stored in the datastore.
Depends on https://github.com/ipfs/boxo/pull/978Closes#10829
* Update docs/changelogs/v0.36.md
Co-authored-by: Guillaume Michel <guillaumemichel@users.noreply.github.com>
* update log message
* update boxo
* Move change log to v0.37.md
* Add `provide clear` command to clear provide queue
The `provide clear` command clears all items from the provide queue and prints out the number of items removed from the queue. The `quiet` option tells the command not to print output.
* refactor(cmds): ipfs provide clear
moving to new namespace to avoid conflicts, and also document other
commands
* docs: clarify Reprovider.Strategy
* chore: remove undesired md link
this wires up https://github.com/ipfs/boxo/pull/971
to make sure explicitly allowlisted hosts have
their own metric label
if we ever need more flexibility here, this can be exposed as
a separate config
The `ipfs config edit` command did not correctly handle the `EDITOR` environment variable correctly when its value contains flags and arguments, i.e. `EDITOR=emacs -nw`. The command was treating the entire value of `$EDITOR` as the name of the editor command. This has been fixed to parse the value of `$EDITOR` into separate args, respecting shell quoting.
Closes#9375
(cherry picked from commit 4195a1da9f)
* Replace use of go-car v1 with go-car/v2
The `dag export` commend needed to be updated to use go-car/v2. This removes the last dependency on go-car v1.
(cherry picked from commit ace0b0b987)
* Configure bitswap braodcast reduction
Add new config items to `Internal.Bitswap` to allow configuration of bitswap broadcast reduction behavior. Broadcast reduction behavior is enabled by default, and uses settings that should be suitable for most installations of kubo.
* update sharness metrics test
* Explicit defaults for broadcast reduction configuration
* Update docs/config.md
