* ci: parallelize gotest by separating test/cli into own job
split the Go Test workflow into two parallel jobs:
- `unit-tests`: runs unit tests (excluding test/cli)
- `cli-tests`: runs test/cli end-to-end tests
test/cli takes ~3 minutes (~50% of total gotest time), so running
it in parallel should reduce wall-clock CI time by ~1.5-2.5 minutes.
both jobs produce JUnit XML and HTML reports for consistent debugging.
* ci(gotest): reduce noise on test timeout panics
add GOTRACEBACK=single to show only one goroutine stack instead of all
when a test timeout panic occurs. this makes CI output much cleaner
when tests hang.
* fix(ci): prevent stderr from corrupting test JSON output
- remove 2>&1 which mixed "go: downloading" stderr messages into JSON
- add JSON validation before parsing
- print failed test names for easier debugging
* ci(gotest): use gotestsum for human-readable test output
- replace per-package coverage loop with single gotestsum invocation
- both unit-tests and cli-tests now show human-readable output
- simplified coverage collection (single -coverprofile, no gocovmerge)
- clarified step names to indicate they run tests
* ci: fix codecov uploads by adding token
- add CODECOV_TOKEN to gotest.yml and sharness.yml
- update codecov-action to v5.5.2
- add fail_ci_if_error: false for robustness
codecov stopped receiving coverage data ~1 year ago when they
started requiring tokens for public repos
* refactor(make): add test_unit and test_cli targets
- add `make test_unit` for unit tests with coverage (used by CI)
- add `make test_cli` for CLI integration tests (used by CI)
- only disable colors when CI env var is set (local dev gets colors)
- remove legacy targets: test_go_test, test_go_short, test_go_race, test_go_expensive
- update gotest.yml to use make targets instead of inline commands
- add test artifacts to .gitignore
* fix(ci): move client/rpc tests to cli-tests job
client/rpc tests use test/cli/harness which requires the ipfs binary.
Move them from test_unit to test_cli where the binary is built.
also:
- update gotestsum to v1.13.0
- simplify workflow step names
* fix(ci): use build tags when listing test packages
go list needs build tags to properly exclude packages like fuse/mfs
when running with TEST_FUSE=0 (nofuse tag).
* fix(ci): move test/integration to cli-tests job
test/integration tests need the ipfs binary, move them from test_unit
to test_cli.
* fix(test): fix flaky kubo-as-a-library and GetClosestPeers tests
kubo-as-a-library: use `Bootstrap()` instead of raw `Swarm().Connect()`
to fix race condition between swarm connection and bitswap peer
discovery. `Bootstrap()` properly integrates peers into the routing
system, ensuring bitswap learns about connected peers synchronously.
GetClosestPeers: simplify retry logic using `EventuallyWithT` with
10-minute timeout. tests all 4 routing types (`auto`, `autoclient`,
`dht`, `dhtclient`) against real bootstrap peers with patient polling.
* fix(example): use bidirectional Swarm().Connect() for reliable bitswap
- connect nodes bidirectionally (A→B and B→A) to simulate mutual peering
- mutual peering protects connection from resource manager culling
- use port 0 for random available ports (avoids CI conflicts)
- enable LoopbackAddressesOnLanDHT for local testing
- move retry logic to test file using require.Eventually
* fix(ci): add test_examples target and parallel example-tests job
- add `make test_examples` target to mk/golang.mk for consistency with test_unit/test_cli
- move example tests to separate parallel CI job (example-tests)
- example: use Bootstrap() with autoconf.FallbackBootstrapPeers for reliable bitswap
- example: increase context timeout to 10 minutes
- test: add 60s per-request timeout to GetClosestPeers (server has 30s routing timeout)
- test: reduce EventuallyWithT to 3 minutes (locally passes in under 1 minute)
* fix(ci): improve test targets, exclusion patterns, and artifact naming
- define COVERPKG_EXCLUDE and UNIT_EXCLUDE as documented variables
- use grep -vE with single regex instead of multiple grep -v calls
- add mkdir -p before rm to ensure directories exist
- add DEPS_GO dependency to test_cli target
- make CLI test timeout configurable via TEST_CLI_TIMEOUT (default 10m)
- fix test_examples cleanup on failure using subshell
- reduce GetClosestPeers test wait time from 3m to 2m
- rename artifacts to match job names: unit-tests-{junit,html}, cli-tests-{junit,html}
- update cli-tests upload-artifact from v5 to v6
* fix(ci): fix unit test exclusion and speed up example test
- fix UNIT_EXCLUDE regex to match client/rpc at end of path
- remove public bootstrap peers from example (only connect to nodeA)
- example test now runs in ~3s instead of timing out
* fix(test): fix flaky TestAddMultipleGCLive race condition
added time.Sleep after spawning GC goroutines to ensure they reach
GCLock() before the test proceeds. without this, the adder's
maybePauseForGC() might check GCRequested() before GC has even
requested the lock, causing the lock to not be released and GC to
block indefinitely.
this matches the existing pattern in TestAddGCLive which already
had this sleep.
also replaced context.Background() with t.Context() in both
TestAddMultipleGCLive and TestAddGCLive for proper test lifecycle
management.
* fix(example): use test harness settings for reliable CI
the kubo-as-a-library example was flaky on CI. applied test-harness-like
settings that match what transports_test.go uses:
- TCP-only on 127.0.0.1 with random port (no QUIC/UDP)
- explicitly disable non-TCP transports (QUIC, Relay, WebTransport, etc)
- use NilRouterOption (no routing) since we connect peers directly
- bitswap works with directly connected peers without DHT lookups
- 2-minute context timeout
- streaming output in test for debugging
stop publishing to ipfs/go-ipfs entirely - the deprecation stub
introduced in v0.39 is no longer needed. only ipfs/kubo is published.
- remove legacy-name job from docker-image.yml workflow
- remove .github/legacy/ (Dockerfile.goipfs-stub, goipfs_stub.sh)
- update bin scripts to use ipfs/kubo as default
* test: add migration tests for Windows and macOS
- add dedicated CI workflow for migration tests on Windows/macOS
- workflow triggers on migration-related file changes only
* build: remove redundant go version checks
- remove GO_MIN_VERSION and check_go_version scripts
- go.mod already enforces minimum version (go 1.25)
- fixes make build on Windows
* fix: windows migration panic by reading config into memory
fixes migration panic on Windows when upgrading from v0.37 to v0.38
by reading the entire config file into memory before performing atomic
operations. this avoids file locking issues on Windows where open files
cannot be renamed.
also fixes:
- TestRepoDir to set USERPROFILE on Windows (not just HOME)
- CLI migration tests to sanitize directory names (remove colons)
minimal fix that solves the "panic: error can't be dealt with
transactionally: Access is denied" error without adding unnecessary
platform-specific complexity.
* fix: set PATH for CLI migration tests in CI
the CLI tests need the built ipfs binary to be in PATH
* fix: use ipfs shutdown for graceful daemon termination in tests
replaces platform-specific signal handling with ipfs shutdown command
which works consistently across all platforms including Windows
* fix: isolate PATH modifications in parallel migration tests
tests running in parallel with t.Parallel() were interfering with each
other through global PATH modifications via os.Setenv(). this caused
tests to download real migration binaries instead of using mocks,
leading to Windows failures due to path separator issues in external tools.
now each test builds its own custom PATH and passes it explicitly to
commands, preventing interference between parallel tests.
* chore: improve error messages in WithBackup
* fix: Windows CI migration test failures
- add .exe extension to mock migration binaries on Windows
- handle repo lock file properly in mock migration binary
- ensure lock is created and removed to prevent conflicts
* refactor: align atomicfile error handling with fs-repo-migrations
- check close error in Abort() before attempting removal
- leave temp file on rename failure for debugging (like fs-repo-15-to-16)
- improves consistency with external migration implementations
* fix: use req.Context in repo migrate to avoid double-lock
The repo migrate command was calling cctx.Context() which has a hidden
side effect: it lazily constructs the IPFS node by calling GetNode(),
which opens the repository and acquires repo.lock. When migrations then
tried to acquire the same lock, it failed with "lock is already held by us"
because go4.org/lock tracks locks per-process in a global map.
The fix uses req.Context instead, which is a plain context.Context with
no side effects. This provides what migrations need (cancellation handling)
without triggering node construction or repo opening.
Context types explained:
- req.Context: Standard Go context for request lifetime, cancellation,
and timeouts. No side effects.
- cctx.Context(): Kubo-specific method that lazily constructs the full
IPFS node (opens repo, acquires lock, initializes subsystems). Returns
the node's internal context.
Why req.Context is correct here:
- Migrations work on raw filesystem (only need ConfigRoot path)
- Command has SetDoesNotUseRepo(true) - doesn't need running node
- Migrations handle their own locking via lockfile.Lock()
- Need cancellation support but not node lifecycle
The bug only appeared with embedded migrations (v16+) because they run
in-process. External migrations (pre-v16) were separate processes, so
each had isolated state. Sequential migrations (forward then backward)
in the same process exposed this latent double-lock issue.
Also adds repo.lock acquisition to RunEmbeddedMigrations to prevent
concurrent migration access, and removes the now-unnecessary daemon
lock check from the migrate command handler.
* fix: use req.Context for migrations and autoconf in daemon startup
daemon.go was incorrectly using cctx.Context() in two critical places:
1. Line 337: migrations call - cctx.Context() triggers GetNode() which
opens the repo and acquires repo.lock BEFORE migrations run, causing
"lock is already held by us" errors when migrations try to lock
2. Line 390: autoconf client.Start() - uses context for HTTP timeouts
and background updater lifecycle, doesn't need node construction
Both now use req.Context (plain Go context) which provides:
- request lifetime and cancellation
- no side effects (doesn't construct node or open repo)
- correct lifecycle for HTTP requests and background goroutines
* feat: add docker stub for deprecated ipfs/go-ipfs name
implements docker part of #10941 by creating a stub image that redirects
users from ipfs/go-ipfs to ipfs/kubo
changes:
- add stub dockerfile and script in .github/legacy/
- modify docker-image.yml to push stub to ipfs/go-ipfs with same tags as ipfs/kubo
- remove ipfs/go-ipfs from get-docker-tags.sh to prevent docker-hub job from pushing to legacy name
- stub displays clear deprecation message directing users to ipfs/kubo:release
* docs: add v0.39 changelog highlight for go-ipfs deprecation
* ci: optimize build workflows
- use go version from go.mod instead of hardcoding
- group platforms by OS for parallel builds
- remove legacy try-build targets
* fix: checkout before setup-go in all workflows
setup-go needs go.mod to be present, so checkout must happen first
* chore: remove deprecated // +build syntax
go 1.17+ uses //go:build, the old syntax is no longer needed
* simplify: remove nofuse tag from CI workflows
- workflows now rely on platform build constraints
- keep make nofuse target for manual builds
- remove unused appveyor.yml
* ci: remove legacy travis variable and fix gateway-conformance
- remove TRAVIS env variable from 4 workflows
- fix gateway-conformance checkout path to match working-directory
- replace deprecated cache-go-action with built-in setup-go caching
* reprovide sweep draft
* update reprovider dep
* go mod tidy
* fix provider type
* change router type
* dual reprovider
* revert to provider.System
* back to start
* SweepingReprovider test
* fix nil pointer deref
* noop provider for nil dht
* disabled initial network estimation
* another iteration
* suppress missing self addrs err
* silence empty rt err on lan dht
* comments
* new attempt at integrating
* reverting changes in core/node/libp2p/routing.go
* removing SweepingProvider
* make reprovider optional
* add noop reprovider
* update KeyChanFunc type alias
* restore boxo KeyChanFunc
* fix missing KeyChanFunc
* test(sharness): PARALLEL=1 and timeout 30m
running sequentially to see where timeout occurs
* initialize MHStore
* revert workflow debug
* config
* config docs
* merged IpfsNode provider and reprovider
* move Provider interface to from kad-dht to node
* moved Provider interface from kad-dht to kubo/core/node
* mod_tidy
* Add Clear to Provider interface
* use latest kad-dht commit
* make linter happy
* updated boxo provide interface
* boxo PR fix
* using latest kad-dht commit
* use latest boxo release
* fix fx
* fx cyclic deps
* fix merge issues
* extended tests
* don't provide LAN DHT
* docs
* restore dual dht provider
* don't start provider before it is online
* address linter
* dual/provider fix
* add delay in provider tests for dht bootstrap
* add OfflineDelay parameter to config
* remove increase number of workers in test
* improved keystore gc process
* fix: replace incorrect logger import in coreapi
replaced github.com/labstack/gommon/log with the standard
github.com/ipfs/go-log/v2 logger used throughout kubo.
removed unused labstack dependency from go.mod files.
* fix: remove duplicate WithDefault call in provider config
* fix: use correct option method for burst workers
* fix: improve error messages for experimental sweeping provider
updated error messages to clearly indicate when commands are unavailable
due to experimental sweeping provider being enabled via Reprovider.Sweep.Enabled=true
* docs: remove obsolete KeyStoreGCInterval config
removed from config.md as option no longer exists (removed in b540fba1a)
updated keystore description to reflect gc happens at reprovide interval
* docs: add TODO placeholder changelog for experimental sweeping DHT provider
using v0.38-TODO.md name to avoid merge conflicts with master branch
and allow CI tests to run. will be renamed to v0.38.md once config
migration is added to the PR
* fix: provideKeysRec go routine
* clear keystore on close
* fix: datastore prefix
* fix: improve error handling in provideKeysRec
- close errCh channel to distinguish between nil and pending errors
- check for pending errors when provided.New closes
- handle context cancellation during error send
- prevent race condition where errors could be silently lost
this ensures DAG walk errors are always propagated correctly
* address gammazero's review
* rename BurstProvider to LegacyProvider
* use latest provider/keystore
* boxo: make mfs StartProviding async
* bump boxo
* chore: update boxo to f2b4e12fb9a8ac138ccb82aae3b51ec51d9f631c
- updated boxo dependency to specified commit
- updated go.mod and go.sum files across all modules
* use latest kad-dht/boxo
* Buffered SweepingProvider wrapper
* use latest kad-dht commit
* allow no DHT router
* use latest kad-dht & boxo
---------
Co-authored-by: Marcin Rataj <lidel@lidel.org>
Co-authored-by: gammazero <11790789+gammazero@users.noreply.github.com>
* feat(ci): docker linting
adds hadolint to validate dockerfile best practices
configures project-specific rules in .hadolint.yaml
* fix(ci): enable hadolint console output
adds verbose and tty format to see linting results in CI logs
* test: trigger hadolint warning
remove --no-install-recommends to test CI output
* fix(ci): fail hadolint on warnings
stricter linting to catch all best practice violations
* fix: add --no-install-recommends to apt-get
reduces image size by avoiding unnecessary packages
* refactor: use WORKDIR instead of cd in dockerfile
replaces cd commands with WORKDIR for cleaner dockerfile
removes unnecessary hadolint ignore rules DL3003 and DL3009
* chore: simplify hadolint config
removes unnecessary override rules for cleaner config
* feat(gateway): concurrency and timeout limits
Depends on https://github.com/ipfs/boxo/pull/994
* chore: boxo master with final boxo#994
this includes race-condition fixes from ipfs/boxo#994
and increased `DefaultMaxConcurrentRequests = 4096`
* docs: concise config.md and changelog
Dependabot couldn't find the original pull request head commit, 0292612bd5e16ed34f2974225fe170b15d4eb821.
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* feat(ipns): Add a parameter in name.publish to change the sequence number
* test: monotonic name publish --sequence
* docs: `name publish --sequence`
* chore: boxo main with PR 962
---------
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* fix/gateway: escape directory redirect URLs
When a director gets redirected to a URL with a treailing slash, special chars in the directory name must be escaped in the redirect URL. This upgrades to a version of box that has that fix.
Closes#10536
* Fix sharness test for new redirect URLs
* Update to latest boxo
* Use latest gateway-conformance
* feat(libp2p): enable shared TCP listeners
* docs: switch mentions of /ws to /tcp/4001
* feat: AutoTLS.AutoWSS
This adds AutoTLS.AutoWSS flag that is set to true by default.
It will check if Addresses.Swarm contain explicit /ws listener,
and if not found, it will append one per every /tcp listener
This way existing TCP ports are reused without any extra configuration,
but we don't break user's who have custom / explicit /ws listener
already.
I also moved logger around, to include Addresses.Swarm inspection
results in `autotls` logger.
* chore: go-libp2p v0.38.1
https://github.com/libp2p/go-libp2p/releases/tag/v0.38.0https://github.com/libp2p/go-libp2p/releases/tag/v0.38.1
* docs: AutoTLS.AutoWSS and go-libp2p v0.38.x
* chore: p2p-forge/client v0.2.0
https://github.com/ipshipyard/p2p-forge/releases/tag/v0.2.0
* fix: disable libp2p.ShareTCPListener() in PNET
* chore(ci): timeout sharness after 15m
average successful run is <9 minutes, no need to wait for 20
https://github.com/ipfs/kubo/actions/workflows/sharness.yml?query=is%3Asuccess
---------
Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com>
Co-authored-by: Marcin Rataj <lidel@lidel.org>
