* fix: add MFS operation limit for --flush=false
adds a global counter that tracks consecutive MFS operations performed
with --flush=false and fails with clear error after limit is reached.
this prevents unbounded memory growth while avoiding the data corruption
risks of auto-flushing.
- adds Internal.MFSNoFlushLimit config
- operations fail with actionable error at limit
- counter resets on successful flush or any --flush=true operation
- operations with --flush=true reset and don't count
this commit removes automatic flush from https://github.com/ipfs/kubo/pull/10971
and instead errors to encourage users of --flush=false to develop a habit
of calling 'ipfs files flush' periodically.
boxo will no longer auto-flush (https://github.com/ipfs/boxo/pull/1041) to
avoid corruption issues, and kubo applies the limit to 'ipfs files' commands
instead.
closes#10842
* test: add tests for MFSNoFlushLimit
tests verify the new Internal.MFSNoFlushLimit config option:
- default limit of 256 operations
- custom limit configuration
- counter reset on flush=true
- counter reset on explicit flush command
- limit=0 disables the feature
- multiple MFS command types count towards limit
* docs: explain why MFS operations fail instead of auto-flushing
addresses feedback from https://github.com/ipfs/kubo/pull/10985#pullrequestreview-3256250970
- clarify that automatic flushing at limit was considered but rejected
- explain the data corruption risks of auto-flushing
- guide users who want auto-flush to use --flush=true (default)
- document benefits of explicit failure for batch operations
(cherry picked from commit a688b7eeac)
* Filestore: provide Filestore nodes
When strategy is set to "all" (the blockstore does all the providing when a
block is written), no providing was happening to Filestore blocks that were
not written to the underlying blockstore (so, the DAG leaves, as they live in
the filesystem directly). This fixes that.
* docs: clarify filestore and urlstore fix in changelog
both filestore (local file references) and urlstore (HTTP/HTTPS URL
references) blocks are now properly provided shortly after initial add
(cherry picked from commit f63887ae96)
* docs: improve slow reprovide warning messages
simplify warning text and provide actionable solutions in order of preference
* feat(config): add validation for Provide.DHT settings
- validate interval doesn't exceed DHT record validity (48h)
- validate worker counts and other parameters are within valid ranges
- improve slow reprovide warning messages to reference config parameter
- add tests for all validation cases
* docs: add reprovide cycle visualization
shows traffic patterns of legacy vs sweep vs accelerated DHT
* fix(webui): show helpful errors for incompatible configurations
- show error when Gateway.NoFetch=true and WebUI is not available locally
- show error when Gateway.DeserializedResponses=false (incompatible)
- add tests for both error scenarios
* chore(webui): update to v4.9.0
https://github.com/ipfs/ipfs-webui/releases/tag/v4.9.0
* docs: add WebUI v4.9.0 update to v0.38 changelog
- highlight new diagnostics screen for troubleshooting
- include screenshots of key features in table format
- add local access URL for WebUI
- update TOC with new sections
* fix: prevent --flush=false in 'ipfs files rm' command
the 'ipfs files rm' command always flushes for safety to ensure
data integrity. this change adds an explicit error when users
try to pass --flush=false, improving ux and preventing confusion.
related to #10842
* fix: add MFS cache size limit to prevent unbounded growth
- add Internal.MFSAutoflushThreshold config (experimental)
- directories auto-flush when cache exceeds threshold with --flush=false
- prevents high memory usage issue from #10842
- default: 256 entries per directory (matching HAMT shard size)
- set to 0 to restore old behavior (risky, may cause errors)
Closes#10842
* fix: use CheckIfPinnedWithType for pin ls with names
updates to use CheckIfPinnedWithType method from https://github.com/ipfs/boxo/pull/1035,
enabling efficient pin name retrieval for 'ipfs pin ls <cid> --names'
- uses new CheckIfPinnedWithType from boxo for type-specific pin checks
- pin names are now returned when listing specific CIDs with --names flag
* test: add CLI tests for pin ls with names
tests cover:
- pin ls with specific CIDs returning names
- pin ls without CID listing all pins with names
- pin ls with --type and --names combinations
- JSON output with and without names
- pin update preserving names
- error cases (invalid CID, unpinned CID)
* docs: add pin name improvements to v0.38 changelog
covers fix for ipfs pin ls --names with specific CIDs
and RPC pin name leak fix
* fix(rpc): support pin names in Add()
passes the Name field from PinAddSettings to the API request
adds test to verify pin names work via RPC
* test: add coverage for pin names functionality
- test special characters, unicode, long names
- test concurrent operations
- test persistence across daemon restarts
- test garbage collection preservation
- fix indirect pin test logic
* chore: boxo@main with boxo#1039
* fix(pin): improve pin ls robustness and validation
- add nil check for n.Pinning with early fail-fast validation
- use pin.StringToMode() for consistent type validation
- add edge case tests for invalid types and unpinned CIDs
* refactor: consolidate Provider/Reprovider into unified Provide config
- merge Provider and Reprovider configs into single Provide section
- add fs-repo-17-to-18 migration for config consolidation
- improve migration ergonomics with common package utilities
- convert deprecated "flat" strategy to "all" during migration
- improve Provide docs
* docs: add total_provide_count metric guidance
- document how to monitor provide success rates via prometheus metrics
- add performance comparison section to changelog
- explain how to evaluate sweep vs legacy provider effectiveness
* fix: add OpenTelemetry meter provider for metrics
- set up meter provider with Prometheus exporter in daemon
- enables metrics from external libs like go-libp2p-kad-dht
- fixes missing total_provide_count_total when SweepEnabled=true
- update docs to reflect actual metric names
---------
Co-authored-by: gammazero <11790789+gammazero@users.noreply.github.com>
Co-authored-by: guillaumemichel <guillaume@michel.id>
Co-authored-by: Daniel Norman <1992255+2color@users.noreply.github.com>
Co-authored-by: Hector Sanjuan <code@hector.link>
* reprovide sweep draft
* update reprovider dep
* go mod tidy
* fix provider type
* change router type
* dual reprovider
* revert to provider.System
* back to start
* SweepingReprovider test
* fix nil pointer deref
* noop provider for nil dht
* disabled initial network estimation
* another iteration
* suppress missing self addrs err
* silence empty rt err on lan dht
* comments
* new attempt at integrating
* reverting changes in core/node/libp2p/routing.go
* removing SweepingProvider
* make reprovider optional
* add noop reprovider
* update KeyChanFunc type alias
* restore boxo KeyChanFunc
* fix missing KeyChanFunc
* test(sharness): PARALLEL=1 and timeout 30m
running sequentially to see where timeout occurs
* initialize MHStore
* revert workflow debug
* config
* config docs
* merged IpfsNode provider and reprovider
* move Provider interface to from kad-dht to node
* moved Provider interface from kad-dht to kubo/core/node
* mod_tidy
* Add Clear to Provider interface
* use latest kad-dht commit
* make linter happy
* updated boxo provide interface
* boxo PR fix
* using latest kad-dht commit
* use latest boxo release
* fix fx
* fx cyclic deps
* fix merge issues
* extended tests
* don't provide LAN DHT
* docs
* restore dual dht provider
* don't start provider before it is online
* address linter
* dual/provider fix
* add delay in provider tests for dht bootstrap
* add OfflineDelay parameter to config
* remove increase number of workers in test
* improved keystore gc process
* fix: replace incorrect logger import in coreapi
replaced github.com/labstack/gommon/log with the standard
github.com/ipfs/go-log/v2 logger used throughout kubo.
removed unused labstack dependency from go.mod files.
* fix: remove duplicate WithDefault call in provider config
* fix: use correct option method for burst workers
* fix: improve error messages for experimental sweeping provider
updated error messages to clearly indicate when commands are unavailable
due to experimental sweeping provider being enabled via Reprovider.Sweep.Enabled=true
* docs: remove obsolete KeyStoreGCInterval config
removed from config.md as option no longer exists (removed in b540fba1a)
updated keystore description to reflect gc happens at reprovide interval
* docs: add TODO placeholder changelog for experimental sweeping DHT provider
using v0.38-TODO.md name to avoid merge conflicts with master branch
and allow CI tests to run. will be renamed to v0.38.md once config
migration is added to the PR
* fix: provideKeysRec go routine
* clear keystore on close
* fix: datastore prefix
* fix: improve error handling in provideKeysRec
- close errCh channel to distinguish between nil and pending errors
- check for pending errors when provided.New closes
- handle context cancellation during error send
- prevent race condition where errors could be silently lost
this ensures DAG walk errors are always propagated correctly
* address gammazero's review
* rename BurstProvider to LegacyProvider
* use latest provider/keystore
* boxo: make mfs StartProviding async
* bump boxo
* chore: update boxo to f2b4e12fb9a8ac138ccb82aae3b51ec51d9f631c
- updated boxo dependency to specified commit
- updated go.mod and go.sum files across all modules
* use latest kad-dht/boxo
* Buffered SweepingProvider wrapper
* use latest kad-dht commit
* allow no DHT router
* use latest kad-dht & boxo
---------
Co-authored-by: Marcin Rataj <lidel@lidel.org>
Co-authored-by: gammazero <11790789+gammazero@users.noreply.github.com>
* fix: enforce identity CID size limits
- validate --inline-limit against verifcid.MaxDigestSize
- add error when --hash=identity exceeds size limit
- add tests for identity CID overflow scenarios
- update help text to show maximum inline limit
This prevents creation of unbounded identity CIDs by enforcing
the 128-byte limit defined in https://github.com/ipfs/boxo/pull/1018Fixes#6011
IPIP: https://github.com/ipfs/specs/pull/512
validates Import configuration fields to prevent invalid values:
- CidVersion: must be 0 or 1
- UnixFSFileMaxLinks: must be positive
- UnixFSDirectoryMaxLinks: must be non-negative
- UnixFSHAMTDirectoryMaxFanout: power of 2, multiple of 8, ≤ 1024
- BatchMaxNodes/BatchMaxSize: must be positive
- UnixFSChunker: validates format patterns
- HashFunction: must be allowed by verifcid
* chore: simplify and update
* docs: streamline release checklist
- reduce from 128 to 113 lines while preserving all information
- add clarity on command execution context (which branch/directory)
- improve dangerous operation warnings (point of no return)
- consolidate prerequisites into checklist format
- clarify cherry-pick -x rationale (traceability and deduplication)
- explain why release branches are kept (patch releases)
- add fetch step to prevent stale branch issues
- simplify social media into single optional item
- add STOP checkpoint after tag push for docker verification
* docs: update release checklist last updated reference
* docs: remove GOPATH requirement from release checklist
mkreleaselog now works from any directory where kubo is checked out
* docs: clarify merge process in release checklist
add step to merge master into merge-release branch first
and specify "Create a merge commit" option explicitly
* docs: address PR #10870 feedback
- update forum example to v0.37.0
- add social media post examples from v0.36.0
- add example for creating next release issue
* docs: update blog example to kubo v0.36.0 commit
* Reprovider strategy: rename "flat" to "all".
Value "flat" now parses to "all". Behaviour from "all" removed.
Fixes#10864 which has detailed explanation.
* core/node/provider.go: remove unused function mfsRootProvider
It was used in the "all" strategy.
* docs: improve reprovider.strategy=all changelog framing
- highlight memory efficiency improvements
- clarify this removes v0.28 workaround
- update config.md memory requirements
- fix announce-on profile typo
* feat: deprecate Reprovider.Strategy=flat
- add deprecation warning in daemon.go when flat strategy is detected
- document that flat is deprecated in ParseReproviderStrategy comment
- add explicit test case for flat -> all mapping
- flat continues to work but users are warned to migrate to all
---------
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* fix(cmds): improve ipfs add --help clarity and organization
- clarify --raw-leaves behavior with CIDv0/v1 defaults
- add Import.* config references to all relevant options
- replace deprecated 'ipfs object links' with 'ipfs ls' in examples
- add section headers for better navigation (BASIC EXAMPLES, MFS INTEGRATION, etc.)
- group related options logically (output control, CID & hashing, experimental features)
- standardize experimental warnings format
- add MerkleDAG docs link and clarify chunking behavior
- fix incorrect Import.UnixFSHAMTThreshold reference
addresses confusion from #10918 where --raw-leaves appeared to have no effect
because CIDv1 automatically enables it by default
* docs: fix typos in Import.* configuration documentation
- fix 'chilcren' → 'children'
- fix 'HAMT directory have' → 'HAMT directories have'
- fix 'A HAMT is an structure' → 'A HAMT is a structure'
* Update core/commands/add.go
Co-authored-by: Daniel Norman <1992255+2color@users.noreply.github.com>
---------
Co-authored-by: Daniel Norman <1992255+2color@users.noreply.github.com>
* Initial pass at Telemetry plugin
Currently, IP Shipyard, with the help of Probelab, monitor and extract
Amino/IPFS public network metrics with the use of DHT crawlers and
bootstrappers (via peerlog plugin). For example, we log all peer IDs seen and
their AgentVersion/Addresses obtained from the `identify` protocol, which
provides insights into protocol usage, total number of peers etc.
We would like to increase the ability to obtain more insights from the network
by collecting some more information in the future, but also to give users more
control over this collection (i.e. opt-out). The information collected will
not allow unique identification of anyone and is only used for aggregation.
Now, this PR explores a way of moving in this direction:
* A new "telemetry" fx plugin is in charge of dealing with telemetry
* The FX plugin allows to plug and make decisions / take actions during the setup phase:
* We can inspect whether we are using Private Networks before the libp2p.Host has been initialized.
* We can send telemetry after the libp2p Host is initialized.
* Everything is self-contained. Custom builds can remove the plugin altogether without needing to surgically edit the code.
As for behaviour:
* The user can opt-in/out via EnvVar, file in the repo path or plugin configuration.
* Users on private networks or with custom bootstrappers are detected, offered a wall of text explaining why we need telemetry and invited to opt-in. Opt-out happens otherwise on a timeout (with no input). Their preferences are stored.
* Users on standard settings are opted-in by default. This is the status quo in Kubo already, except they don't get a chance to opt out.
The telemetry libp2p protocol is yet to be defined, but expect something similar to identify, with a protobuf being pushed to bootstrappers or to a specific telemetry node that we define. In the case of pnets, this will be done with a temporary peer.
* checkpoint
* telemetry plugin: second pass
* On first run it generates a UUID and shows a message to the user.
* UUID is persistend to "telemetry_uuid"
* Sends telemetry 1 minute after boot and every 24h
* LogEvent is the thing containing all the telemetry that is sent
* Opt-out possible via env-var or plugin configuration
* Telemetry: add changelog and environment variable documentation
* docs: improved daemon message
making it more obvious nothing was sent yet
and that user had 15m to out-out
plus some debug logs that confirm opt-out
* refactor: rename IPFS_TELEMETRY_MODE to IPFS_TELEMETRY
* fix: add User-Agent header to telemetry requests
---------
Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com>
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* feat(gateway): concurrency and timeout limits
Depends on https://github.com/ipfs/boxo/pull/994
* chore: boxo master with final boxo#994
this includes race-condition fixes from ipfs/boxo#994
and increased `DefaultMaxConcurrentRequests = 4096`
* docs: concise config.md and changelog
* feat(ipns): Add a parameter in name.publish to change the sequence number
* test: monotonic name publish --sequence
* docs: `name publish --sequence`
* chore: boxo main with PR 962
---------
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* fix(relay): feed connected peers to AutoRelay discovery
Feed all connected swarm peers to AutoRelay as potential relay
candidates. This allows peers from HTTP routing and manual connections
to serve as relays, not just DHT-discovered peers.
Fixes#10899
* docs: changelog
* feat: update log level command to show log levels
* test: add log level tests
* update TestCommands test
* docs: relation to GOLOG_LOG_LEVEL
* chore: update to latest go-log
* fix: do not output single subsystem name in CLI
* test: explicit subsystem request dont output subsystem
* LevelFromString renamed to Parse
* Modify `ipfs log level`
* Denote default level with sdubsystem name '(defult)'.
* make "*" an dalias for "all". Test to make sure both work the same.
* Fix command stream result handling
Do not close the stream prematurely in a deferred function. Also, avoid possible shadowing errors.
Closes#9007
* test(cli): test/cli/cid_test.go
includes regression tests for https://github.com/ipfs/kubo/issues/9007
and better coverage than the old test/sharness/t0290-cid.sh
(identified bug in basemoji)
* fix: base256emoji in cid bases --prefix
changed the character display logic from ASCII range check to
unicode.IsPrint() to properly handle Unicode characters including
emojis.
* docs: changelog
---------
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* Provide according to strategy
Updates boxo to a version with the changes from https://github.com/ipfs/boxo/pull/976, which decentralize the providing responsibilities (from a central providing.Exchange to blockstore, pinner, mfs).
The changes consist in initializing the Pinner, MFS and the blockstore with the provider.System, which is created first.
Since the provider.System is created first, the reproviding KeyChanFunc is set
later when we can create it once we have the Pinner, MFS and the blockstore.
Some additional work applies to the Add() workflow. Normally, blocks would get provided at the Blockstore or the Pinner, but when adding blocks AND a "pinned" strategy is used, the blockstore does not provide, and the
pinner does not traverse the DAG (and thus doesn't provide either), so we need to provide directly from the Adder. This is resolved by wrapping the DAGService in a "providingDAGService" which provides every added block, when using the "pinned" strategy.
`ipfs --offline add` when the ONLINE daemon is running will now announce blocks per the chosen strategy, where before it did not announce them. This is documented in the changelog. A couple of releases ago, adding with `ipfs --offline add` was faster, but this is no longer the case so we are not incurring in any penalties by sticking to the fact that the daemon is online and has a providing strategy that we follow.
Co-authored-by: gammazero <11790789+gammazero@users.noreply.github.com>
Co-authored-by: Marcin Rataj <lidel@lidel.org>
- Move `ipfs stat reprovide` to `ipfs provide stat`
- Mark `ipfs stat provide` as deprecated and replaces by `ipfs provide stat`
- Mark `ipfs stat reprovide` as deprecated and replaces by `ipfs provide stat`
- Remove redundant code from deprecated subcommands
Closes#10869
* feat(add): add support for naming pinned CID
Signed-off-by: kapil <kapilsareen584@gmail.com>
* fix(add): no double pinning and simplify pin-name
- modify PinRoot to accept name parameter, eliminating double pinning
- remove automatic filename fallback logic for cleaner behavior
- only create named pins when explicitly requested via --pin-name=value
- replace NoPinName constant with idiomatic empty string literals
- Update help text and tests to reflect explicit-only behavior
* docs: changelog
* chore: lint
* test: negative case for empty pin-name
* chore: gofmt
---------
Signed-off-by: kapil <kapilsareen584@gmail.com>
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* refactor: remove goprocess
The `goprocess` package is no longer needed. It can be replaces by modern `context` and `context.AfterFunc`.
* mod tidy
* log unmount errors on shutdown
* Do not log non-mounted errors on shutdown
* Use WaitGroup associated with IPFS node to wait for services to whutdown
* Prefer explicit Close to context.ArterFunc
* Do not use node-level WaitGroup
* Unmount for non-supported platforms
* fix return values
* test: daemon shuts down gracefully
make sure ongoing operations dont block shutdown
* test(cli): add TestFUSE
* test: smarter RequiresFUSE
opportunistically run FUSE tests if env has fusermount
and TEST_FUSE was not explicitly set
* docs: changelog
---------
Co-authored-by: gammazero <gammazero@users.noreply.github.com>
Co-authored-by: Marcin Rataj <lidel@lidel.org>
