fix(ci): parallelize gotest, cleanup output, flakiness (#11113)

* ci: parallelize gotest by separating test/cli into own job

split the Go Test workflow into two parallel jobs:
- `unit-tests`: runs unit tests (excluding test/cli)
- `cli-tests`: runs test/cli end-to-end tests

test/cli takes ~3 minutes (~50% of total gotest time), so running
it in parallel should reduce wall-clock CI time by ~1.5-2.5 minutes.

both jobs produce JUnit XML and HTML reports for consistent debugging.

* ci(gotest): reduce noise on test timeout panics

add GOTRACEBACK=single to show only one goroutine stack instead of all
when a test timeout panic occurs. this makes CI output much cleaner
when tests hang.

* fix(ci): prevent stderr from corrupting test JSON output

- remove 2>&1 which mixed "go: downloading" stderr messages into JSON
- add JSON validation before parsing
- print failed test names for easier debugging

* ci(gotest): use gotestsum for human-readable test output

- replace per-package coverage loop with single gotestsum invocation
- both unit-tests and cli-tests now show human-readable output
- simplified coverage collection (single -coverprofile, no gocovmerge)
- clarified step names to indicate they run tests

* ci: fix codecov uploads by adding token

- add CODECOV_TOKEN to gotest.yml and sharness.yml
- update codecov-action to v5.5.2
- add fail_ci_if_error: false for robustness

codecov stopped receiving coverage data ~1 year ago when they
started requiring tokens for public repos

* refactor(make): add test_unit and test_cli targets

- add `make test_unit` for unit tests with coverage (used by CI)
- add `make test_cli` for CLI integration tests (used by CI)
- only disable colors when CI env var is set (local dev gets colors)
- remove legacy targets: test_go_test, test_go_short, test_go_race, test_go_expensive
- update gotest.yml to use make targets instead of inline commands
- add test artifacts to .gitignore

* fix(ci): move client/rpc tests to cli-tests job

client/rpc tests use test/cli/harness which requires the ipfs binary.
Move them from test_unit to test_cli where the binary is built.

also:
- update gotestsum to v1.13.0
- simplify workflow step names

* fix(ci): use build tags when listing test packages

go list needs build tags to properly exclude packages like fuse/mfs
when running with TEST_FUSE=0 (nofuse tag).

* fix(ci): move test/integration to cli-tests job

test/integration tests need the ipfs binary, move them from test_unit
to test_cli.

* fix(test): fix flaky kubo-as-a-library and GetClosestPeers tests

kubo-as-a-library: use `Bootstrap()` instead of raw `Swarm().Connect()`
to fix race condition between swarm connection and bitswap peer
discovery. `Bootstrap()` properly integrates peers into the routing
system, ensuring bitswap learns about connected peers synchronously.

GetClosestPeers: simplify retry logic using `EventuallyWithT` with
10-minute timeout. tests all 4 routing types (`auto`, `autoclient`,
`dht`, `dhtclient`) against real bootstrap peers with patient polling.

* fix(example): use bidirectional Swarm().Connect() for reliable bitswap

- connect nodes bidirectionally (A→B and B→A) to simulate mutual peering
- mutual peering protects connection from resource manager culling
- use port 0 for random available ports (avoids CI conflicts)
- enable LoopbackAddressesOnLanDHT for local testing
- move retry logic to test file using require.Eventually

* fix(ci): add test_examples target and parallel example-tests job

- add `make test_examples` target to mk/golang.mk for consistency with test_unit/test_cli
- move example tests to separate parallel CI job (example-tests)
- example: use Bootstrap() with autoconf.FallbackBootstrapPeers for reliable bitswap
- example: increase context timeout to 10 minutes
- test: add 60s per-request timeout to GetClosestPeers (server has 30s routing timeout)
- test: reduce EventuallyWithT to 3 minutes (locally passes in under 1 minute)

* fix(ci): improve test targets, exclusion patterns, and artifact naming

- define COVERPKG_EXCLUDE and UNIT_EXCLUDE as documented variables
- use grep -vE with single regex instead of multiple grep -v calls
- add mkdir -p before rm to ensure directories exist
- add DEPS_GO dependency to test_cli target
- make CLI test timeout configurable via TEST_CLI_TIMEOUT (default 10m)
- fix test_examples cleanup on failure using subshell
- reduce GetClosestPeers test wait time from 3m to 2m
- rename artifacts to match job names: unit-tests-{junit,html}, cli-tests-{junit,html}
- update cli-tests upload-artifact from v5 to v6

* fix(ci): fix unit test exclusion and speed up example test

- fix UNIT_EXCLUDE regex to match client/rpc at end of path
- remove public bootstrap peers from example (only connect to nodeA)
- example test now runs in ~3s instead of timing out

* fix(test): fix flaky TestAddMultipleGCLive race condition

added time.Sleep after spawning GC goroutines to ensure they reach
GCLock() before the test proceeds. without this, the adder's
maybePauseForGC() might check GCRequested() before GC has even
requested the lock, causing the lock to not be released and GC to
block indefinitely.

this matches the existing pattern in TestAddGCLive which already
had this sleep.

also replaced context.Background() with t.Context() in both
TestAddMultipleGCLive and TestAddGCLive for proper test lifecycle
management.

* fix(example): use test harness settings for reliable CI

the kubo-as-a-library example was flaky on CI. applied test-harness-like
settings that match what transports_test.go uses:

- TCP-only on 127.0.0.1 with random port (no QUIC/UDP)
- explicitly disable non-TCP transports (QUIC, Relay, WebTransport, etc)
- use NilRouterOption (no routing) since we connect peers directly
- bitswap works with directly connected peers without DHT lookups
- 2-minute context timeout
- streaming output in test for debugging
This commit is contained in:
Marcin Rataj 2026-01-08 05:07:08 +01:00 committed by GitHub
parent 55b94751cc
commit 1301710a91
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
13 changed files with 262 additions and 182 deletions

View File

@ -14,11 +14,13 @@ concurrency:
cancel-in-progress: true
jobs:
go-test:
# Unit tests with coverage collection (uploaded to Codecov)
unit-tests:
if: github.repository == 'ipfs/kubo' || github.event_name == 'workflow_dispatch'
runs-on: ${{ fromJSON(github.repository == 'ipfs/kubo' && '["self-hosted", "linux", "x64", "2xlarge"]' || '"ubuntu-latest"') }}
timeout-minutes: 20
timeout-minutes: 15
env:
GOTRACEBACK: single # reduce noise on test timeout panics
TEST_DOCKER: 0
TEST_FUSE: 0
TEST_VERBOSE: 1
@ -36,12 +38,9 @@ jobs:
go-version-file: 'go.mod'
- name: Install missing tools
run: sudo apt update && sudo apt install -y zsh
- name: 👉️ If this step failed, go to «Summary» (top left) → inspect the «Failures/Errors» table
env:
# increasing parallelism beyond 2 doesn't speed up the tests much
PARALLEL: 2
- name: Run unit tests
run: |
make -j "$PARALLEL" test/unit/gotest.junit.xml &&
make test_unit &&
[[ ! $(jq -s -c 'map(select(.Action == "fail")) | .[]' test/unit/gotest.json) ]]
- name: Upload coverage to Codecov
uses: codecov/codecov-action@671740ac38dd9b0130fbe1cec585b89eea48d3de # v5.5.2
@ -49,28 +48,8 @@ jobs:
with:
name: unittests
files: coverage/unit_tests.coverprofile
- name: Test kubo-as-a-library example
run: |
# we want to first test with the kubo version in the go.mod file
go test -v ./...
# we also want to test the examples against the current version of kubo
# however, that version might be in a fork so we need to replace the dependency
# backup the go.mod and go.sum files to restore them after we run the tests
cp go.mod go.mod.bak
cp go.sum go.sum.bak
# make sure the examples run against the current version of kubo
go mod edit -replace github.com/ipfs/kubo=./../../..
go mod tidy
go test -v ./...
# restore the go.mod and go.sum files to their original state
mv go.mod.bak go.mod
mv go.sum.bak go.sum
working-directory: docs/examples/kubo-as-a-library
token: ${{ secrets.CODECOV_TOKEN }}
fail_ci_if_error: false
- name: Create a proper JUnit XML report
uses: ipdxco/gotest-json-to-junit-xml@v1
with:
@ -80,7 +59,7 @@ jobs:
- name: Archive the JUnit XML report
uses: actions/upload-artifact@v6
with:
name: unit
name: unit-tests-junit
path: test/unit/gotest.junit.xml
if: failure() || success()
- name: Create a HTML report
@ -93,7 +72,7 @@ jobs:
- name: Archive the HTML report
uses: actions/upload-artifact@v6
with:
name: html
name: unit-tests-html
path: test/unit/gotest.html
if: failure() || success()
- name: Create a Markdown report
@ -106,3 +85,86 @@ jobs:
- name: Set the summary
run: cat test/unit/gotest.md >> $GITHUB_STEP_SUMMARY
if: failure() || success()
# End-to-end integration/regression tests from test/cli
# (Go-based replacement for legacy test/sharness shell scripts)
cli-tests:
if: github.repository == 'ipfs/kubo' || github.event_name == 'workflow_dispatch'
runs-on: ${{ fromJSON(github.repository == 'ipfs/kubo' && '["self-hosted", "linux", "x64", "2xlarge"]' || '"ubuntu-latest"') }}
timeout-minutes: 15
env:
GOTRACEBACK: single # reduce noise on test timeout panics
TEST_VERBOSE: 1
GIT_PAGER: cat
IPFS_CHECK_RCMGR_DEFAULTS: 1
defaults:
run:
shell: bash
steps:
- name: Check out Kubo
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
- name: Install missing tools
run: sudo apt update && sudo apt install -y zsh
- name: Run CLI tests
env:
IPFS_PATH: ${{ runner.temp }}/ipfs-test
run: make test_cli
- name: Create JUnit XML report
uses: ipdxco/gotest-json-to-junit-xml@v1
with:
input: test/cli/cli-tests.json
output: test/cli/cli-tests.junit.xml
if: failure() || success()
- name: Archive JUnit XML report
uses: actions/upload-artifact@v6
with:
name: cli-tests-junit
path: test/cli/cli-tests.junit.xml
if: failure() || success()
- name: Create HTML report
uses: ipdxco/junit-xml-to-html@v1
with:
mode: no-frames
input: test/cli/cli-tests.junit.xml
output: test/cli/cli-tests.html
if: failure() || success()
- name: Archive HTML report
uses: actions/upload-artifact@v6
with:
name: cli-tests-html
path: test/cli/cli-tests.html
if: failure() || success()
- name: Create Markdown report
uses: ipdxco/junit-xml-to-html@v1
with:
mode: summary
input: test/cli/cli-tests.junit.xml
output: test/cli/cli-tests.md
if: failure() || success()
- name: Set summary
run: cat test/cli/cli-tests.md >> $GITHUB_STEP_SUMMARY
if: failure() || success()
# Example tests (kubo-as-a-library)
example-tests:
if: github.repository == 'ipfs/kubo' || github.event_name == 'workflow_dispatch'
runs-on: ${{ fromJSON(github.repository == 'ipfs/kubo' && '["self-hosted", "linux", "x64", "2xlarge"]' || '"ubuntu-latest"') }}
timeout-minutes: 5
env:
GOTRACEBACK: single
defaults:
run:
shell: bash
steps:
- name: Check out Kubo
uses: actions/checkout@v6
- name: Set up Go
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'
- name: Run example tests
run: make test_examples

View File

@ -60,6 +60,8 @@ jobs:
with:
name: sharness
files: kubo/coverage/sharness_tests.coverprofile
token: ${{ secrets.CODECOV_TOKEN }}
fail_ci_if_error: false
- name: Aggregate results
run: find kubo/test/sharness/test-results -name 't*-*.sh.*.counts' | kubo/test/sharness/lib/sharness/aggregate-results.sh > kubo/test/sharness/test-results/summary.txt
- name: 👉️ If this step failed, go to «Summary» (top left) → «HTML Report» → inspect the «Failures» column

5
.gitignore vendored
View File

@ -28,6 +28,11 @@ go-ipfs-source.tar.gz
docs/examples/go-ipfs-as-a-library/example-folder/Qm*
/test/sharness/t0054-dag-car-import-export-data/*.car
# test artifacts from make test_unit / test_cli
/test/unit/gotest.json
/test/unit/gotest.junit.xml
/test/cli/cli-tests.json
# ignore build output from snapcraft
/ipfs_*.snap
/parts

View File

@ -134,15 +134,14 @@ help:
@echo ''
@echo 'TESTING TARGETS:'
@echo ''
@echo ' test - Run all tests'
@echo ' test_short - Run short go tests and short sharness tests'
@echo ' test_go_short - Run short go tests'
@echo ' test_go_test - Run all go tests'
@echo ' test - Run all tests (test_go_fmt, test_unit, test_cli, test_sharness)'
@echo ' test_short - Run fast tests (test_go_fmt, test_unit)'
@echo ' test_unit - Run unit tests with coverage (excludes test/cli)'
@echo ' test_cli - Run CLI integration tests (requires built binary)'
@echo ' test_go_fmt - Check Go source formatting'
@echo ' test_go_build - Build kubo for all platforms from .github/build-platforms.yml'
@echo ' test_go_expensive - Run all go tests and build all platforms'
@echo ' test_go_race - Run go tests with the race detector enabled'
@echo ' test_go_lint - Run the `golangci-lint` vetting tool'
@echo ' test_go_lint - Run golangci-lint'
@echo ' test_sharness - Run sharness tests'
@echo ' coverage - Collects coverage info from unit tests and sharness'
@echo ' coverage - Collect coverage info from unit tests and sharness'
@echo
.PHONY: help

View File

@ -30,6 +30,7 @@ import (
const testPeerID = "QmTFauExutTsy4XP6JbMFcw2Wa9645HJt2bTqL6qYDCKfe"
func TestAddMultipleGCLive(t *testing.T) {
ctx := t.Context()
r := &repo.Mock{
C: config.Config{
Identity: config.Identity{
@ -38,13 +39,13 @@ func TestAddMultipleGCLive(t *testing.T) {
},
D: syncds.MutexWrap(datastore.NewMapDatastore()),
}
node, err := core.NewNode(context.Background(), &core.BuildCfg{Repo: r})
node, err := core.NewNode(ctx, &core.BuildCfg{Repo: r})
if err != nil {
t.Fatal(err)
}
out := make(chan interface{}, 10)
adder, err := NewAdder(context.Background(), node.Pinning, node.Blockstore, node.DAG)
adder, err := NewAdder(ctx, node.Pinning, node.Blockstore, node.DAG)
if err != nil {
t.Fatal(err)
}
@ -67,7 +68,7 @@ func TestAddMultipleGCLive(t *testing.T) {
go func() {
defer close(out)
_, _ = adder.AddAllAndPin(context.Background(), slf)
_, _ = adder.AddAllAndPin(ctx, slf)
// Ignore errors for clarity - the real bug would be gc'ing files while adding them, not this resultant error
}()
@ -80,9 +81,12 @@ func TestAddMultipleGCLive(t *testing.T) {
gc1started := make(chan struct{})
go func() {
defer close(gc1started)
gc1out = gc.GC(context.Background(), node.Blockstore, node.Repo.Datastore(), node.Pinning, nil)
gc1out = gc.GC(ctx, node.Blockstore, node.Repo.Datastore(), node.Pinning, nil)
}()
// Give GC goroutine time to reach GCLock (will block there waiting for adder)
time.Sleep(time.Millisecond * 100)
// GC shouldn't get the lock until after the file is completely added
select {
case <-gc1started:
@ -119,9 +123,12 @@ func TestAddMultipleGCLive(t *testing.T) {
gc2started := make(chan struct{})
go func() {
defer close(gc2started)
gc2out = gc.GC(context.Background(), node.Blockstore, node.Repo.Datastore(), node.Pinning, nil)
gc2out = gc.GC(ctx, node.Blockstore, node.Repo.Datastore(), node.Pinning, nil)
}()
// Give GC goroutine time to reach GCLock
time.Sleep(time.Millisecond * 100)
select {
case <-gc2started:
t.Fatal("gc shouldn't have started yet")
@ -155,6 +162,7 @@ func TestAddMultipleGCLive(t *testing.T) {
}
func TestAddGCLive(t *testing.T) {
ctx := t.Context()
r := &repo.Mock{
C: config.Config{
Identity: config.Identity{
@ -163,13 +171,13 @@ func TestAddGCLive(t *testing.T) {
},
D: syncds.MutexWrap(datastore.NewMapDatastore()),
}
node, err := core.NewNode(context.Background(), &core.BuildCfg{Repo: r})
node, err := core.NewNode(ctx, &core.BuildCfg{Repo: r})
if err != nil {
t.Fatal(err)
}
out := make(chan interface{})
adder, err := NewAdder(context.Background(), node.Pinning, node.Blockstore, node.DAG)
adder, err := NewAdder(ctx, node.Pinning, node.Blockstore, node.DAG)
if err != nil {
t.Fatal(err)
}
@ -193,7 +201,7 @@ func TestAddGCLive(t *testing.T) {
go func() {
defer close(addDone)
defer close(out)
_, err := adder.AddAllAndPin(context.Background(), slf)
_, err := adder.AddAllAndPin(ctx, slf)
if err != nil {
t.Error(err)
}
@ -211,7 +219,7 @@ func TestAddGCLive(t *testing.T) {
gcstarted := make(chan struct{})
go func() {
defer close(gcstarted)
gcout = gc.GC(context.Background(), node.Blockstore, node.Repo.Datastore(), node.Pinning, nil)
gcout = gc.GC(ctx, node.Blockstore, node.Repo.Datastore(), node.Pinning, nil)
}()
// gc shouldn't start until we let the add finish its current file.
@ -255,9 +263,6 @@ func TestAddGCLive(t *testing.T) {
last = c
}
ctx, cancel := context.WithTimeout(context.Background(), time.Second*5)
defer cancel()
set := cid.NewSet()
err = dag.Walk(ctx, dag.GetLinksWithDAG(node.DAG), last, set.Visit)
if err != nil {

View File

@ -3,33 +3,14 @@ include mk/header.mk
GOCC ?= go
$(d)/coverage_deps: $$(DEPS_GO) cmd/ipfs/ipfs
rm -rf $(@D)/unitcover && mkdir $(@D)/unitcover
rm -rf $(@D)/sharnesscover && mkdir $(@D)/sharnesscover
ifneq ($(IPFS_SKIP_COVER_BINS),1)
$(d)/coverage_deps: test/bin/gocovmerge
endif
.PHONY: $(d)/coverage_deps
# unit tests coverage
UTESTS_$(d) := $(shell $(GOCC) list -f '{{if (or (len .TestGoFiles) (len .XTestGoFiles))}}{{.ImportPath}}{{end}}' $(go-flags-with-tags) ./... | grep -v go-ipfs/vendor | grep -v go-ipfs/Godeps)
# unit tests coverage is now produced by test_unit target in mk/golang.mk
# (outputs coverage/unit_tests.coverprofile and test/unit/gotest.json)
UCOVER_$(d) := $(addsuffix .coverprofile,$(addprefix $(d)/unitcover/, $(subst /,_,$(UTESTS_$(d)))))
$(UCOVER_$(d)): $(d)/coverage_deps ALWAYS
$(eval TMP_PKG := $(subst _,/,$(basename $(@F))))
$(eval TMP_DEPS := $(shell $(GOCC) list -f '{{range .Deps}}{{.}} {{end}}' $(go-flags-with-tags) $(TMP_PKG) | sed 's/ /\n/g' | grep ipfs/go-ipfs) $(TMP_PKG))
$(eval TMP_DEPS_LIST := $(call join-with,$(comma),$(TMP_DEPS)))
$(GOCC) test $(go-flags-with-tags) $(GOTFLAGS) -v -covermode=atomic -json -coverpkg=$(TMP_DEPS_LIST) -coverprofile=$@ $(TMP_PKG) | tee -a test/unit/gotest.json
$(d)/unit_tests.coverprofile: $(UCOVER_$(d))
gocovmerge $^ > $@
TGTS_$(d) := $(d)/unit_tests.coverprofile
.PHONY: $(d)/unit_tests.coverprofile
TGTS_$(d) :=
# sharness tests coverage
$(d)/ipfs: GOTAGS += testrunmain
@ -46,7 +27,7 @@ endif
export IPFS_COVER_DIR:= $(realpath $(d))/sharnesscover/
$(d)/sharness_tests.coverprofile: export TEST_PLUGIN=0
$(d)/sharness_tests.coverprofile: $(d)/ipfs cmd/ipfs/ipfs-test-cover $(d)/coverage_deps test_sharness
$(d)/sharness_tests.coverprofile: $(d)/ipfs cmd/ipfs/ipfs-test-cover $(d)/coverage_deps test/bin/gocovmerge test_sharness
(cd $(@D)/sharnesscover && find . -type f | gocovmerge -list -) > $@

View File

@ -47,7 +47,7 @@ func setupPlugins(externalPluginsPath string) error {
return nil
}
func createTempRepo(swarmPort int) (string, error) {
func createTempRepo() (string, error) {
repoPath, err := os.MkdirTemp("", "ipfs-shell")
if err != nil {
return "", fmt.Errorf("failed to get temp dir: %s", err)
@ -59,15 +59,28 @@ func createTempRepo(swarmPort int) (string, error) {
return "", err
}
// Configure custom ports to avoid conflicts with other IPFS instances.
// This demonstrates how to customize the node's network addresses.
// Use TCP-only on loopback with random port for reliable local testing.
// This matches what kubo's test harness uses (test/cli/transports_test.go).
// QUIC/UDP transports are avoided because they may be throttled on CI.
cfg.Addresses.Swarm = []string{
fmt.Sprintf("/ip4/0.0.0.0/tcp/%d", swarmPort),
fmt.Sprintf("/ip4/0.0.0.0/udp/%d/quic-v1", swarmPort),
fmt.Sprintf("/ip4/0.0.0.0/udp/%d/quic-v1/webtransport", swarmPort),
fmt.Sprintf("/ip4/0.0.0.0/udp/%d/webrtc-direct", swarmPort),
"/ip4/127.0.0.1/tcp/0",
}
// Explicitly disable non-TCP transports for reliability.
cfg.Swarm.Transports.Network.QUIC = config.False
cfg.Swarm.Transports.Network.Relay = config.False
cfg.Swarm.Transports.Network.WebTransport = config.False
cfg.Swarm.Transports.Network.WebRTCDirect = config.False
cfg.Swarm.Transports.Network.Websocket = config.False
cfg.AutoTLS.Enabled = config.False
// Disable routing - we don't need DHT for direct peer connections.
// Bitswap works with directly connected peers without needing DHT lookups.
cfg.Routing.Type = config.NewOptionalString("none")
// Disable bootstrap for this example - we manually connect only the peers we need.
cfg.Bootstrap = []string{}
// When creating the repository, you can define custom settings on the repository, such as enabling experimental
// features (See experimental-features.md) or customizing the gateway endpoint.
// To do such things, you should modify the variable `cfg`. For example:
@ -107,8 +120,12 @@ func createNode(ctx context.Context, repoPath string) (*core.IpfsNode, error) {
nodeOptions := &core.BuildCfg{
Online: true,
Routing: libp2p.DHTOption, // This option sets the node to be a full DHT node (both fetching and storing DHT Records)
// Routing: libp2p.DHTClientOption, // This option sets the node to be a client DHT node (only fetching records)
// For this example, we use NilRouterOption (no routing) since we connect peers directly.
// Bitswap works with directly connected peers without needing DHT lookups.
// In production, you would typically use:
// Routing: libp2p.DHTOption, // Full DHT node (stores and fetches records)
// Routing: libp2p.DHTClientOption, // DHT client (only fetches records)
Routing: libp2p.NilRouterOption,
Repo: repo,
}
@ -118,8 +135,7 @@ func createNode(ctx context.Context, repoPath string) (*core.IpfsNode, error) {
var loadPluginsOnce sync.Once
// Spawns a node to be used just for this run (i.e. creates a tmp repo).
// The swarmPort parameter specifies the port for libp2p swarm listeners.
func spawnEphemeral(ctx context.Context, swarmPort int) (icore.CoreAPI, *core.IpfsNode, error) {
func spawnEphemeral(ctx context.Context) (icore.CoreAPI, *core.IpfsNode, error) {
var onceErr error
loadPluginsOnce.Do(func() {
onceErr = setupPlugins("")
@ -129,7 +145,7 @@ func spawnEphemeral(ctx context.Context, swarmPort int) (icore.CoreAPI, *core.Ip
}
// Create a Temporary Repo
repoPath, err := createTempRepo(swarmPort)
repoPath, err := createTempRepo()
if err != nil {
return nil, nil, fmt.Errorf("failed to create temp repo: %s", err)
}
@ -207,8 +223,7 @@ func main() {
defer cancel()
// Spawn a local peer using a temporary path, for testing purposes
// Using port 4010 to avoid conflict with default IPFS port 4001
ipfsA, nodeA, err := spawnEphemeral(ctx, 4010)
ipfsA, nodeA, err := spawnEphemeral(ctx)
if err != nil {
panic(fmt.Errorf("failed to spawn peer node: %s", err))
}
@ -222,9 +237,8 @@ func main() {
fmt.Printf("Added file to peer with CID %s\n", peerCidFile.String())
// Spawn a node using a temporary path, creating a temporary repo for the run
// Using port 4011 (different from nodeA's port 4010)
fmt.Println("Spawning Kubo node on a temporary repo")
ipfsB, _, err := spawnEphemeral(ctx, 4011)
ipfsB, _, err := spawnEphemeral(ctx)
if err != nil {
panic(fmt.Errorf("failed to spawn ephemeral node: %s", err))
}
@ -297,11 +311,12 @@ func main() {
fmt.Printf("Got directory back from IPFS (IPFS path: %s) and wrote it to %s\n", cidDirectory.String(), outputPathDirectory)
/// --- Part IV: Getting a file from the IPFS Network
/// --- Part IV: Getting a file from another IPFS node
fmt.Println("\n-- Going to connect to a few nodes in the Network as bootstrappers --")
fmt.Println("\n-- Connecting to nodeA and fetching content via bitswap --")
// Get nodeA's address so we can fetch the file we added to it
// Get nodeA's actual listening address dynamically.
// We configured TCP-only on 127.0.0.1 with random port, so this will be a TCP address.
peerAddrs, err := ipfsA.Swarm().LocalAddrs(ctx)
if err != nil {
panic(fmt.Errorf("could not get peer addresses: %s", err))
@ -309,26 +324,18 @@ func main() {
peerMa := peerAddrs[0].String() + "/p2p/" + nodeA.Identity.String()
bootstrapNodes := []string{
// In production, use autoconf.FallbackBootstrapPeers from boxo/autoconf
// which includes well-known IPFS bootstrap peers like:
// In production, use real bootstrap peers like:
// "/dnsaddr/bootstrap.libp2p.io/p2p/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN",
// "/dnsaddr/bootstrap.libp2p.io/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa",
// "/ip4/104.131.131.82/tcp/4001/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ",
// You can add custom peers here. For example, another IPFS node:
// "/ip4/192.0.2.1/tcp/4001/p2p/QmYourPeerID...",
// "/ip4/192.0.2.1/udp/4001/quic-v1/p2p/QmYourPeerID...",
// nodeA's address (the peer we created above that has our test file)
// For this example, we only connect to nodeA which has our test content.
peerMa,
}
fmt.Println("Connecting to peers...")
fmt.Println("Connecting to peer...")
err = connectToPeers(ctx, ipfsB, bootstrapNodes)
if err != nil {
panic(fmt.Errorf("failed to connect to peers: %s", err))
}
fmt.Println("Connected to peers")
fmt.Println("Connected to peer")
exampleCIDStr := peerCidFile.RootCid().String()

View File

@ -1,21 +1,39 @@
package main
import (
"bytes"
"io"
"os"
"os/exec"
"strings"
"testing"
"time"
)
func TestExample(t *testing.T) {
out, err := exec.Command("go", "run", "main.go").Output()
t.Log("Starting go run main.go...")
start := time.Now()
cmd := exec.Command("go", "run", "main.go")
cmd.Env = append(os.Environ(), "GOLOG_LOG_LEVEL=error") // reduce libp2p noise
// Stream output to both test log and capture buffer for verification
// This ensures we see progress even if the process is killed
var buf bytes.Buffer
cmd.Stdout = io.MultiWriter(os.Stdout, &buf)
cmd.Stderr = io.MultiWriter(os.Stderr, &buf)
err := cmd.Run()
elapsed := time.Since(start)
t.Logf("Command completed in %v", elapsed)
out := buf.String()
if err != nil {
var stderr string
if xe, ok := err.(*exec.ExitError); ok {
stderr = string(xe.Stderr)
t.Fatalf("running example (%v):\n%s", err, out)
}
t.Fatalf("running example (%v): %s\n%s", err, string(out), stderr)
}
if !strings.Contains(string(out), "All done!") {
t.Errorf("example did not run successfully")
if !strings.Contains(out, "All done!") {
t.Errorf("example did not complete successfully, output:\n%s", out)
}
}

View File

@ -41,40 +41,57 @@ define go-build
$(GOCC) build $(go-flags-with-tags) -o "$@" "$(1)"
endef
test_go_test: $$(DEPS_GO)
$(GOCC) test $(go-flags-with-tags) $(GOTFLAGS) ./...
.PHONY: test_go_test
# Only disable colors when running in CI (non-interactive terminal)
GOTESTSUM_NOCOLOR := $(if $(CI),--no-color,)
# Build all platforms from .github/build-platforms.yml
# Packages excluded from coverage (test code and examples are not production code)
COVERPKG_EXCLUDE := /(test|docs/examples)/
# Packages excluded from unit tests: coverage exclusions + client/rpc (tested by test_cli)
UNIT_EXCLUDE := /(test|docs/examples)/|/client/rpc$$
# Unit tests with coverage
# Produces JSON for CI reporting and coverage profile for Codecov
test_unit: test/bin/gotestsum $$(DEPS_GO)
mkdir -p test/unit coverage
rm -f test/unit/gotest.json coverage/unit_tests.coverprofile
gotestsum $(GOTESTSUM_NOCOLOR) --jsonfile test/unit/gotest.json -- $(go-flags-with-tags) $(GOTFLAGS) -covermode=atomic -coverprofile=coverage/unit_tests.coverprofile -coverpkg=$$($(GOCC) list $(go-tags) ./... | grep -vE '$(COVERPKG_EXCLUDE)' | tr '\n' ',' | sed 's/,$$//') $$($(GOCC) list $(go-tags) ./... | grep -vE '$(UNIT_EXCLUDE)')
.PHONY: test_unit
# CLI/integration tests (requires built binary in PATH)
# Includes test/cli, test/integration, and client/rpc
# Produces JSON for CI reporting
# Override TEST_CLI_TIMEOUT for local development: make test_cli TEST_CLI_TIMEOUT=5m
TEST_CLI_TIMEOUT ?= 10m
test_cli: cmd/ipfs/ipfs test/bin/gotestsum $$(DEPS_GO)
mkdir -p test/cli
rm -f test/cli/cli-tests.json
PATH="$(CURDIR)/cmd/ipfs:$(CURDIR)/test/bin:$$PATH" gotestsum $(GOTESTSUM_NOCOLOR) --jsonfile test/cli/cli-tests.json -- -v -timeout=$(TEST_CLI_TIMEOUT) ./test/cli/... ./test/integration/... ./client/rpc/...
.PHONY: test_cli
# Example tests (docs/examples/kubo-as-a-library)
# Tests against both published and current kubo versions
# Uses timeout to ensure CI gets output before job-level timeout kills everything
TEST_EXAMPLES_TIMEOUT ?= 2m
test_examples:
cd docs/examples/kubo-as-a-library && go test -v -timeout=$(TEST_EXAMPLES_TIMEOUT) ./... && cp go.mod go.mod.bak && cp go.sum go.sum.bak && (go mod edit -replace github.com/ipfs/kubo=./../../.. && go mod tidy && go test -v -timeout=$(TEST_EXAMPLES_TIMEOUT) ./...; ret=$$?; mv go.mod.bak go.mod; mv go.sum.bak go.sum; exit $$ret)
.PHONY: test_examples
# Build kubo for all platforms from .github/build-platforms.yml
test_go_build:
bin/test-go-build-platforms
.PHONY: test_go_build
test_go_short: GOTFLAGS += -test.short
test_go_short: test_go_test
.PHONY: test_go_short
test_go_race: GOTFLAGS += -race
test_go_race: test_go_test
.PHONY: test_go_race
test_go_expensive: test_go_test test_go_build
.PHONY: test_go_expensive
TEST_GO += test_go_expensive
# Check Go source formatting
test_go_fmt:
bin/test-go-fmt
.PHONY: test_go_fmt
TEST_GO += test_go_fmt
# Run golangci-lint (used by CI)
test_go_lint: test/bin/golangci-lint
golangci-lint run --timeout=3m ./...
.PHONY: test_go_lint
test_go: $(TEST_GO)
# Version check is no longer needed - go.mod enforces minimum version
.PHONY: check_go_version
TEST_GO := test_go_fmt test_unit test_cli test_examples
TEST += $(TEST_GO)
TEST_SHORT += test_go_fmt test_go_short
TEST_SHORT += test_go_fmt test_unit

View File

@ -2,7 +2,6 @@ package cli
import (
"context"
"encoding/json"
"strings"
"testing"
"time"
@ -21,11 +20,6 @@ import (
"github.com/stretchr/testify/require"
)
// swarmPeersOutput is used to parse the JSON output of 'ipfs swarm peers --enc=json'
type swarmPeersOutput struct {
Peers []struct{} `json:"Peers"`
}
func TestRoutingV1Server(t *testing.T) {
t.Parallel()
@ -206,11 +200,14 @@ func TestRoutingV1Server(t *testing.T) {
c, err := client.New(node.GatewayURL())
require.NoError(t, err)
// Try to get closest peers - should fail gracefully with an error
// Try to get closest peers - should fail gracefully with an error.
// Use 60-second timeout (server has 30s routing timeout).
testCid, err := cid.Decode("QmUNLLsPACCz1vLxQVkXqqLX5R1X345qqfHbsf67hvA3Nn")
require.NoError(t, err)
_, err = c.GetClosestPeers(context.Background(), testCid)
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
defer cancel()
_, err = c.GetClosestPeers(ctx, testCid)
require.Error(t, err)
// All these routing types should indicate DHT is not available
// The exact error message may vary based on implementation details
@ -224,7 +221,7 @@ func TestRoutingV1Server(t *testing.T) {
}
})
t.Run("GetClosestPeers returns peers for self", func(t *testing.T) {
t.Run("GetClosestPeers returns peers", func(t *testing.T) {
t.Parallel()
routingTypes := []string{"auto", "autoclient", "dht", "dhtclient"}
@ -242,47 +239,33 @@ func TestRoutingV1Server(t *testing.T) {
})
node.StartDaemon()
// Create client before waiting so we can probe DHT readiness
c, err := client.New(node.GatewayURL())
require.NoError(t, err)
// Query for closest peers to our own peer ID
key := peer.ToCid(node.PeerID())
// Wait for node to connect to bootstrap peers and populate WAN DHT routing table
minPeers := len(autoconf.FallbackBootstrapPeers)
require.EventuallyWithT(t, func(t *assert.CollectT) {
res := node.RunIPFS("swarm", "peers", "--enc=json")
var output swarmPeersOutput
err := json.Unmarshal(res.Stdout.Bytes(), &output)
assert.NoError(t, err)
peerCount := len(output.Peers)
// Wait until we have at least minPeers connected
assert.GreaterOrEqual(t, peerCount, minPeers,
"waiting for at least %d bootstrap peers, currently have %d", minPeers, peerCount)
}, 60*time.Second, time.Second)
// Wait for DHT to be ready by probing GetClosestPeers until it succeeds
require.EventuallyWithT(t, func(t *assert.CollectT) {
probeCtx, probeCancel := context.WithTimeout(context.Background(), 30*time.Second)
defer probeCancel()
probeIter, probeErr := c.GetClosestPeers(probeCtx, key)
if probeErr == nil {
probeIter.Close()
}
assert.NoError(t, probeErr, "DHT should be ready to handle GetClosestPeers")
}, 2*time.Minute, 5*time.Second)
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
// Wait for WAN DHT routing table to be populated.
// The server has a 30-second routing timeout, so we use 60 seconds
// per request to allow for network latency while preventing hangs.
// Total wait time is 2 minutes (locally passes in under 1 minute).
var records []*types.PeerRecord
require.EventuallyWithT(t, func(ct *assert.CollectT) {
ctx, cancel := context.WithTimeout(t.Context(), 60*time.Second)
defer cancel()
resultsIter, err := c.GetClosestPeers(ctx, key)
require.NoError(t, err)
records, err := iter.ReadAllResults(resultsIter)
require.NoError(t, err)
if !assert.NoError(ct, err) {
return
}
records, err = iter.ReadAllResults(resultsIter)
assert.NoError(ct, err)
}, 2*time.Minute, 5*time.Second)
// Verify we got some peers back from WAN DHT
assert.NotEmpty(t, records, "should return some peers close to own peerid")
require.NotEmpty(t, records, "should return peers close to own peerid")
// Per IPIP-0476, GetClosestPeers returns at most 20 peers
assert.LessOrEqual(t, len(records), 20, "IPIP-0476 limits GetClosestPeers to 20 peers")
// Verify structure of returned records
for _, record := range records {

View File

@ -15,7 +15,7 @@ require (
github.com/ipfs/iptb-plugins v0.5.1
github.com/multiformats/go-multiaddr v0.16.1
github.com/multiformats/go-multihash v0.2.3
gotest.tools/gotestsum v1.12.3
gotest.tools/gotestsum v1.13.0
)
require (

View File

@ -985,8 +985,8 @@ gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gotest.tools/gotestsum v1.12.3 h1:jFwenGJ0RnPkuKh2VzAYl1mDOJgbhobBDeL2W1iEycs=
gotest.tools/gotestsum v1.12.3/go.mod h1:Y1+e0Iig4xIRtdmYbEV7K7H6spnjc1fX4BOuUhWw2Wk=
gotest.tools/gotestsum v1.13.0 h1:+Lh454O9mu9AMG1APV4o0y7oDYKyik/3kBOiCqiEpRo=
gotest.tools/gotestsum v1.13.0/go.mod h1:7f0NS5hFb0dWr4NtcsAsF0y1kzjEFfAil0HiBQJE03Q=
gotest.tools/v3 v3.5.2 h1:7koQfIKdy+I8UTetycgUqXWSDwpgv193Ka+qRsmBY8Q=
gotest.tools/v3 v3.5.2/go.mod h1:LtdLGcnqToBH83WByAAi/wiwSFCArdFIUV/xxN4pcjA=
honnef.co/go/tools v0.6.1 h1:R094WgE8K4JirYjBaOpz/AvTyUu/3wbmAoskKN/pxTI=

View File

@ -2,7 +2,8 @@ include mk/header.mk
CLEAN += $(d)/gotest.json $(d)/gotest.junit.xml
$(d)/gotest.junit.xml: test/bin/gotestsum coverage/unit_tests.coverprofile
# Convert gotest.json (produced by test_unit) to JUnit XML format
$(d)/gotest.junit.xml: test/bin/gotestsum $(d)/gotest.json
gotestsum --no-color --junitfile $@ --raw-command cat $(@D)/gotest.json
include mk/footer.mk