* datastore: upgrade go-ds-flatfs to v0.6.0
See: https://github.com/ipfs/go-ds-flatfs/pull/142
* docs(changelog): add go-ds-flatfs atomic batch writes
*documents the new flatfs batch implementation that uses atomic
operations via temp directory, preventing orphan blocks on interrupted
imports and reducing memory usage.
* includes improved tests, batch cleanup fixes, and docs
* docs(changelog): reframe go-ds-flatfs entry for users
focus on user benefits instead of implementation details
Gracefully shutdown the ipfs node to ensure data is saved. Forceful shutdown is done if there is an error sending the interrupt signal to the process, such as on Windows.
* docs: mark custom routing as experimental
reorganize Routing.Type section for clarity, group production and
experimental options, consolidate DHT explanation, add limitations
section to delegated-routing.md documenting that HTTP-only routing
cannot provide content reliably
* chore(config): reorder Routing sections and improve callout formatting
move DelegatedRouters after Type, add config option names to CAUTION headers
* docs: address reviewer feedback on config.md
- clarify that `auto` can be combined with custom URLs in `Routing.DelegatedRouters`
- rename headers for consistency: `Routing.Routers.[name].Type`, `Routing.Routers.[name].Parameters`, `Routing.Methods`
- replace deprecated Strategic Providing reference with `Provide.*` config
- remove outdated caveat about 0.39 sweep limitation
- wording: "likely suffer" → "will be most affected"
* docs: remove redundant Summary section from delegated-routing.md
the IMPORTANT callout and Motivation section already cover what users
need to know. historical version info was noise for researchers trying
to configure custom routing.
addresses reviewer feedback from #11111.
---------
Co-authored-by: Daniel Norman <2color@users.noreply.github.com>
Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com>
Belt-and-suspenders defense against data races where routing
subsystem (DHT or delegated routing) may reuse backing array.
Clones AddrInfo before publishing to QueryEvent to ensure
isolated copy.
Closes https://github.com/ipfs/kubo/issues/11116
* Do not output keystore error on shutdown.
Closes#11127
* fix: add debug log for keystore sync interrupted by shutdown
log at DEBUG level when keystore sync is interrupted during shutdown,
preserving error details for debugging while keeping normal output clean
---------
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* feat(p2p): add --foreground flag to listen and forward commands
adds `-f/--foreground` option that keeps the command running until
interrupted (SIGTERM/Ctrl+C) or closed via `ipfs p2p close`. the
listener/forwarder is automatically removed when the command exits.
useful for systemd services and scripts that need cleanup on exit.
* docs: add p2p-tunnels.md with systemd examples
- add dedicated docs/p2p-tunnels.md covering:
- why p2p tunnels (NAT traversal, no public IP needed)
- quick start with netcat
- background and foreground modes
- systemd integration with path-based activation
- security considerations and troubleshooting
- document Experimental.Libp2pStreamMounting in docs/config.md
- simplify docs/experimental-features.md, link to new doc
- add "Learn more" links to ipfs p2p listen/forward --help
- update changelog entry with doc link
- add cross-reference in misc/README.md
* chore: reference kubo#5460 for p2p config
Ref. https://github.com/ipfs/kubo/issues/5460
* fix(daemon): write api/gateway files only after HTTP server is ready
fixes race condition where $IPFS_PATH/api and $IPFS_PATH/gateway files
were written before the HTTP servers were ready to accept connections.
this caused issues for tools like systemd path units that immediately
try to connect when these files appear.
changes:
- add corehttp.ServeWithReady() that signals when server is ready
- wait for ready signal before writing address files
- use sync.WaitGroup.Go() (Go 1.25) for cleaner goroutine management
- add TestAddressFileReady to verify both api and gateway files
* fix(daemon): buffer errc channel and wait for all listeners
- buffer error channel with len(listeners) to prevent deadlock when
multiple servers write errors simultaneously
- wait for ALL listeners to be ready before writing api/gateway file,
not just the first one
Feedback-from: https://github.com/ipfs/kubo/pull/11099#pullrequestreview-3593885839
* docs(changelog): improve p2p tunnel section clarity
reframe to lead with user benefit and add example output
* docs(p2p): remove obsolete race condition caveat
the "First launch fails but restarts work" troubleshooting section
described a race where the api file was written before the daemon was
ready. this was fixed in 80b703a which ensures api/gateway files are
only written after HTTP servers are ready to accept connections.
---------
Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com>
- add TTY auto-detection for progress display (matching `dag export`)
- use single-line progress with carriage return instead of flooding
- show human-readable sizes alongside raw bytes in summary
- update --progress flag to be auto-detected by default
progress format: `Fetched/Processed N blocks, M bytes (X MB)`
summary format: `Total Size: 99 (99 B)`
* fix(routing): use LegacyProvider for HTTP-only custom routing
when `Routing.Type=custom` with only HTTP routers and no DHT,
fall back to LegacyProvider instead of SweepingProvider.
SweepingProvider requires a DHT client which is unavailable in
HTTP-only configurations, causing it to return NoopProvider and
breaking provider record announcements to HTTP routers.
fixes#11089
* test(routing): verify provide stat works with HTTP-only routing
* docs(config): clarify SweepEnabled fallback for HTTP-only routing
---------
Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com>
* ci: parallelize gotest by separating test/cli into own job
split the Go Test workflow into two parallel jobs:
- `unit-tests`: runs unit tests (excluding test/cli)
- `cli-tests`: runs test/cli end-to-end tests
test/cli takes ~3 minutes (~50% of total gotest time), so running
it in parallel should reduce wall-clock CI time by ~1.5-2.5 minutes.
both jobs produce JUnit XML and HTML reports for consistent debugging.
* ci(gotest): reduce noise on test timeout panics
add GOTRACEBACK=single to show only one goroutine stack instead of all
when a test timeout panic occurs. this makes CI output much cleaner
when tests hang.
* fix(ci): prevent stderr from corrupting test JSON output
- remove 2>&1 which mixed "go: downloading" stderr messages into JSON
- add JSON validation before parsing
- print failed test names for easier debugging
* ci(gotest): use gotestsum for human-readable test output
- replace per-package coverage loop with single gotestsum invocation
- both unit-tests and cli-tests now show human-readable output
- simplified coverage collection (single -coverprofile, no gocovmerge)
- clarified step names to indicate they run tests
* ci: fix codecov uploads by adding token
- add CODECOV_TOKEN to gotest.yml and sharness.yml
- update codecov-action to v5.5.2
- add fail_ci_if_error: false for robustness
codecov stopped receiving coverage data ~1 year ago when they
started requiring tokens for public repos
* refactor(make): add test_unit and test_cli targets
- add `make test_unit` for unit tests with coverage (used by CI)
- add `make test_cli` for CLI integration tests (used by CI)
- only disable colors when CI env var is set (local dev gets colors)
- remove legacy targets: test_go_test, test_go_short, test_go_race, test_go_expensive
- update gotest.yml to use make targets instead of inline commands
- add test artifacts to .gitignore
* fix(ci): move client/rpc tests to cli-tests job
client/rpc tests use test/cli/harness which requires the ipfs binary.
Move them from test_unit to test_cli where the binary is built.
also:
- update gotestsum to v1.13.0
- simplify workflow step names
* fix(ci): use build tags when listing test packages
go list needs build tags to properly exclude packages like fuse/mfs
when running with TEST_FUSE=0 (nofuse tag).
* fix(ci): move test/integration to cli-tests job
test/integration tests need the ipfs binary, move them from test_unit
to test_cli.
* fix(test): fix flaky kubo-as-a-library and GetClosestPeers tests
kubo-as-a-library: use `Bootstrap()` instead of raw `Swarm().Connect()`
to fix race condition between swarm connection and bitswap peer
discovery. `Bootstrap()` properly integrates peers into the routing
system, ensuring bitswap learns about connected peers synchronously.
GetClosestPeers: simplify retry logic using `EventuallyWithT` with
10-minute timeout. tests all 4 routing types (`auto`, `autoclient`,
`dht`, `dhtclient`) against real bootstrap peers with patient polling.
* fix(example): use bidirectional Swarm().Connect() for reliable bitswap
- connect nodes bidirectionally (A→B and B→A) to simulate mutual peering
- mutual peering protects connection from resource manager culling
- use port 0 for random available ports (avoids CI conflicts)
- enable LoopbackAddressesOnLanDHT for local testing
- move retry logic to test file using require.Eventually
* fix(ci): add test_examples target and parallel example-tests job
- add `make test_examples` target to mk/golang.mk for consistency with test_unit/test_cli
- move example tests to separate parallel CI job (example-tests)
- example: use Bootstrap() with autoconf.FallbackBootstrapPeers for reliable bitswap
- example: increase context timeout to 10 minutes
- test: add 60s per-request timeout to GetClosestPeers (server has 30s routing timeout)
- test: reduce EventuallyWithT to 3 minutes (locally passes in under 1 minute)
* fix(ci): improve test targets, exclusion patterns, and artifact naming
- define COVERPKG_EXCLUDE and UNIT_EXCLUDE as documented variables
- use grep -vE with single regex instead of multiple grep -v calls
- add mkdir -p before rm to ensure directories exist
- add DEPS_GO dependency to test_cli target
- make CLI test timeout configurable via TEST_CLI_TIMEOUT (default 10m)
- fix test_examples cleanup on failure using subshell
- reduce GetClosestPeers test wait time from 3m to 2m
- rename artifacts to match job names: unit-tests-{junit,html}, cli-tests-{junit,html}
- update cli-tests upload-artifact from v5 to v6
* fix(ci): fix unit test exclusion and speed up example test
- fix UNIT_EXCLUDE regex to match client/rpc at end of path
- remove public bootstrap peers from example (only connect to nodeA)
- example test now runs in ~3s instead of timing out
* fix(test): fix flaky TestAddMultipleGCLive race condition
added time.Sleep after spawning GC goroutines to ensure they reach
GCLock() before the test proceeds. without this, the adder's
maybePauseForGC() might check GCRequested() before GC has even
requested the lock, causing the lock to not be released and GC to
block indefinitely.
this matches the existing pattern in TestAddGCLive which already
had this sleep.
also replaced context.Background() with t.Context() in both
TestAddMultipleGCLive and TestAddGCLive for proper test lifecycle
management.
* fix(example): use test harness settings for reliable CI
the kubo-as-a-library example was flaky on CI. applied test-harness-like
settings that match what transports_test.go uses:
- TCP-only on 127.0.0.1 with random port (no QUIC/UDP)
- explicitly disable non-TCP transports (QUIC, Relay, WebTransport, etc)
- use NilRouterOption (no routing) since we connect peers directly
- bitswap works with directly connected peers without DHT lookups
- 2-minute context timeout
- streaming output in test for debugging
* docs: improve README for first-time users
- add Quick Taste section with real CIDv1 example near top
- rewrite "What is Kubo?" with technical concepts (CIDs, DAGs, UnixFS, Bitswap)
- reorder features to follow user journey (CLI before advanced HTTP features)
- streamline install section with links to docs.ipfs.tech
- organize package managers in tables with Repology version badges
- add supply chain security warning for third-party packages
- surface important docs (metrics, debug guide, customizing)
- update maintainer info with Shipyard branding
Closes#11125Closes#7298Closes#5471Closes#5087
* docs(readme): add changelogs link and fix docs directory URL
* docs(readme): add mDNS/DHT links and clarify build steps
- link LAN discovery to mDNS spec and WAN to Amino DHT glossary
- show both make build and make install with output paths
addresses https://discuss.ipfs.tech/t/19933
- add docs/developer-guide.md with prerequisites, build, test, and troubleshooting
- link from README.md, docs/README.md, and CONTRIBUTING.md
- document test suite differences (unit vs e2e, test/cli vs test/sharness)
- include tips for running specific tests during development
* fix: update go-libp2p to v0.46.0
- reduced WebRTC log noise (go-libp2p#3426)
- fixed mDNS discovery on Windows/macOS (go-libp2p#3434)
- includes quic-go v0.57.1 (v0.56.0 + v0.57.0)
* fix(example): kubo-as-a-library test timeout
- use custom ports (4010/4011) to avoid conflicts with default 4001
- add 2-minute context timeout to fail fast
- get peer addresses dynamically instead of hardcoding wrong port
- wait for peer connection synchronously instead of fire-and-forget
- update comments to reference autoconf.FallbackBootstrapPeers
* chore: update p2p-forge to v0.7.0
* fix(test): wait for DHT readiness in GetClosestPeers test
the test was failing for `routing_type=auto` because it only waited for
swarm connections but not for the DHT routing table to be populated.
added a separate probe loop that waits for GetClosestPeers to succeed
before running the actual test assertions.
document known 0.39 limitation where sweep provider may fail to
estimate DHT size when accelerated client is still crawling the
network, resulting in single-region mode without efficiency gains.
also remove accelerated client recommendation from changelog
since it may mislead users into enabling both together.
stop publishing to ipfs/go-ipfs entirely - the deprecation stub
introduced in v0.39 is no longer needed. only ipfs/kubo is published.
- remove legacy-name job from docker-image.yml workflow
- remove .github/legacy/ (Dockerfile.goipfs-stub, goipfs_stub.sh)
- update bin scripts to use ipfs/kubo as default
- convert from zsh to bash for portability and shellcheck support
- resolve GitHub handles via multiple methods:
- noreply email pattern (user@users.noreply.github.com)
- merge commit messages (Merge pull request #N from user/branch)
- gh CLI API for PR authors (squash merge commits)
- gh CLI API for commit authors (fallback for non-PR commits)
- deduplicate contributors by GitHub handle instead of author name
- cache resolved mappings in ~/.cache/mkreleaselog/github-handles.json
- output clickable GitHub profile links in contributor table
- rewrite overview to lead with user value (self-hosting on consumer hardware)
- reorder highlights: provider features together, then UPnP, then housekeeping
- simplify titles (drop "Amino", "Fixed", verbose descriptions)
- link to Shipyard's sweep provider blogpost
This allows Kubo to respond to the GetClosestPeers() http routing v1 endpoint
as spec'ed here: https://github.com/ipfs/specs/pull/476
It is based on work from https://github.com/ipfs/boxo/pull/1021
We let IpfsNode implmement the contentRouter.Client interface with the new
method. We use our WAN-DHT to get the closest peers.
Additionally, Routing V1 HTTP API is exposed by default which enables light clients in browsers to use Kubo Gateway as delegated routing backend
Co-authored-by: Marcin Rataj <lidel@lidel.org>
