Commit Graph

2322 Commits

Author SHA1 Message Date
Marcin Rataj
1f7bbcbf6a
fix(deps): bump boxo and kad-dht for resource leak fixes (#11192)
https://github.com/ipfs/boxo/pull/1103
https://github.com/libp2p/go-libp2p-kad-dht/pull/1226
2026-02-11 01:32:37 +01:00
Marcin Rataj
6a008fc74c
refactor: apply go fix modernizers from Go 1.26 (#11190)
* chore: apply go fix modernizers from Go 1.26

automated refactoring: interface{} to any, slices.Contains,
and other idiomatic updates.

* feat(ci): add `go fix` check to Go analysis workflow

ensures Go 1.26 modernizers are applied, fails CI if `go fix ./...`
produces any changes (similar to existing `go fmt` enforcement)
2026-02-11 01:01:32 +01:00
Marcin Rataj
36c29c55f0
feat: update to Go 1.26 (#11189)
* feat: update to Go 1.26

replace deprecated httputil.NewSingleHostReverseProxy (Director)
with ReverseProxy.Rewrite, switch math/rand to math/rand/v2 in
production code, update Dockerfile base image.

* fix test to accept response with HTTP status of 307 and 308 where 302 and 301 are expected

---------

Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com>
2026-02-11 00:08:28 +01:00
Marcin Rataj
c6702eaf88
fix: allow dag import of 1MiB chunks wrapped in dag-pb (#11185)
Some checks failed
CodeQL / codeql (push) Has been cancelled
Docker Check / lint (push) Has been cancelled
Docker Check / build (push) Has been cancelled
Gateway Conformance / gateway-conformance (push) Has been cancelled
Gateway Conformance / gateway-conformance-libp2p-experiment (push) Has been cancelled
Go Build / go-build (push) Has been cancelled
Go Check / go-check (push) Has been cancelled
Go Lint / go-lint (push) Has been cancelled
Go Test / unit-tests (push) Has been cancelled
Go Test / cli-tests (push) Has been cancelled
Go Test / example-tests (push) Has been cancelled
Interop / interop-prep (push) Has been cancelled
Sharness / sharness-test (push) Has been cancelled
Spell Check / spellcheck (push) Has been cancelled
Interop / helia-interop (push) Has been cancelled
Interop / ipfs-webui (push) Has been cancelled
IPIP-499's unixfs-v1-2025 profile uses 1MiB chunks. with
--raw-leaves=false, protobuf wrapping pushes blocks slightly over 1MiB.
the previous 1MiB SoftBlockLimit rejected these blocks on dag import.

raise SoftBlockLimit to 2MiB to match the bitswap spec, which requires
implementations to support blocks up to 2MiB.

- raise SoftBlockLimit to 2MiB per the bitswap spec
- update error messages and help text
- bump boxo to main with ipfs/boxo#1101 (raised ChunkSizeLimit/BlockSizeLimit,
  256-byte overhead budget)
- update sharness tests for 2MiB boundary
- add test/cli boundary tests for block put, dag put, dag import,
  ipfs add (raw and wrapped leaves), and bitswap exchange including
  regression tests for the libp2p message size hard limit
2026-02-06 23:55:40 +01:00
Daniel Norman
f57d13c2c2
feat: swarm addrs autonat command (#11184)
* feat: add swarm addrs autonat command

fixes #11171 by adding a self service way to debug public reachability
with autonat

* test: add test for ipfs swarm addr autonat command

* docs: add ipfs swarm addrs autonat to changelog

* test: update failing test

* fix: swarm addrs autonat bugfixes and cleanup

- fix help text to show capitalized reachability values (Public, Private,
  Unknown) matching actual output from network.Reachability.String()
- default Reachability to "Unknown" instead of empty string when the
  host interface assertion fails
- extract multiaddrsToStrings and writeAddrSection helpers to
  deduplicate repeated conversion loops and text formatting blocks

---------

Co-authored-by: Marcin Rataj <lidel@lidel.org>
2026-02-06 20:33:15 +01:00
Marcin Rataj
3a6b1ee122
feat(gateway): IPIP-0524 Gateway.AllowCodecConversion config option (#11090)
Some checks are pending
CodeQL / codeql (push) Waiting to run
Docker Check / lint (push) Waiting to run
Docker Check / build (push) Waiting to run
Gateway Conformance / gateway-conformance (push) Waiting to run
Gateway Conformance / gateway-conformance-libp2p-experiment (push) Waiting to run
Go Build / go-build (push) Waiting to run
Go Check / go-check (push) Waiting to run
Go Lint / go-lint (push) Waiting to run
Go Test / unit-tests (push) Waiting to run
Go Test / cli-tests (push) Waiting to run
Go Test / example-tests (push) Waiting to run
Interop / interop-prep (push) Waiting to run
Interop / helia-interop (push) Blocked by required conditions
Interop / ipfs-webui (push) Blocked by required conditions
Sharness / sharness-test (push) Waiting to run
Spell Check / spellcheck (push) Waiting to run
* feat(gateway): IPIP-0524 Gateway.AllowCodecConversion config option

Wire up boxo's AllowCodecConversion config to control codec conversion
behavior per IPIP-0524. When false (default), the gateway returns
406 Not Acceptable if the requested format doesn't match the block's
codec.

Clients should fetch raw blocks (`?format=raw`) and convert client-side.

Ref: https://github.com/ipfs/specs/pull/524
Ref: https://github.com/ipfs/boxo/pull/1077
Ref: https://github.com/ipfs/gateway-conformance/pull/254

* chore: update boxo for improved 406 codec conversion error

boxo now returns an actionable hint when codec conversion is rejected:
suggests fetching raw block with ?format=raw and converting client-side.

* chore: bump boxo and gateway-conformance to v0.10

* docs: add IPLD Logical Format note to AllowCodecConversion
2026-02-06 02:21:35 +01:00
Marcin Rataj
67c89bbd7e
feat(config): add Import.* for CID Profiles from IPIP-499 (#11148)
Some checks are pending
CodeQL / codeql (push) Waiting to run
Docker Check / lint (push) Waiting to run
Docker Check / build (push) Waiting to run
Gateway Conformance / gateway-conformance (push) Waiting to run
Gateway Conformance / gateway-conformance-libp2p-experiment (push) Waiting to run
Go Build / go-build (push) Waiting to run
Go Check / go-check (push) Waiting to run
Go Lint / go-lint (push) Waiting to run
Go Test / unit-tests (push) Waiting to run
Go Test / cli-tests (push) Waiting to run
Go Test / example-tests (push) Waiting to run
Interop / interop-prep (push) Waiting to run
Interop / helia-interop (push) Blocked by required conditions
Interop / ipfs-webui (push) Blocked by required conditions
Sharness / sharness-test (push) Waiting to run
Spell Check / spellcheck (push) Waiting to run
* feat(config): Import.* and unixfs-v1-2025 profile

implements IPIP-499: add config options for controlling UnixFS DAG
determinism and introduces `unixfs-v1-2025` and `unixfs-v0-2015`
profiles for cross-implementation CID reproducibility.

changes:
- add Import.* fields: HAMTDirectorySizeEstimation, SymlinkMode,
  DAGLayout, IncludeEmptyDirectories, IncludeHidden
- add validation for all Import.* config values
- add unixfs-v1-2025 profile (recommended for new data)
- add unixfs-v0-2015 profile (alias: legacy-cid-v0)
- remove deprecated test-cid-v1 and test-cid-v1-wide profiles
- wire Import.HAMTSizeEstimationMode() to boxo globals
- update go.mod to use boxo with SizeEstimationMode support

ref: https://specs.ipfs.tech/ipips/ipip-0499/

* feat(add): add --dereference-symlinks, --empty-dirs, --hidden CLI flags

add CLI flags for controlling file collection behavior during ipfs add:

- `--dereference-symlinks`: recursively resolve symlinks to their target
  content (replaces deprecated --dereference-args which only worked on
  CLI arguments). wired through go-ipfs-cmds to boxo's SerialFileOptions.
- `--empty-dirs` / `-E`: include empty directories (default: true)
- `--hidden` / `-H`: include hidden files (default: false)

these flags are CLI-only and not wired to Import.* config options because
go-ipfs-cmds library handles input file filtering before the directory
tree is passed to kubo. removed unused Import.UnixFSSymlinkMode config
option that was defined but never actually read by the CLI.

also:
- wire --trickle to Import.UnixFSDAGLayout config default
- update go-ipfs-cmds to v0.15.1-0.20260117043932-17687e216294
- add SYMLINK HANDLING section to ipfs add help text
- add CLI tests for all three flags

ref: https://github.com/ipfs/specs/pull/499

* test(add): add CID profile tests and wire SizeEstimationMode

add comprehensive test suite for UnixFS CID determinism per IPIP-499:
- verify exact HAMT threshold boundary for both estimation modes:
  - v0-2015 (links): sum(name_len + cid_len) == 262144
  - v1-2025 (block): serialized block size == 262144
- verify HAMT triggers at threshold + 1 byte for both profiles
- add all deterministic CIDs for cross-implementation testing

also wires SizeEstimationMode through CLI/API, allowing
Import.UnixFSHAMTSizeEstimation config to take effect.

bumps boxo to ipfs/boxo@6707376 which aligns HAMT threshold with
JS implementation (uses > instead of >=), fixing CID determinism
at the exact 256 KiB boundary.

* feat(add): --dereference-symlinks now resolves all symlinks

Previously, resolving symlinks required two flags:
- --dereference-args: resolved symlinks passed as CLI arguments
- --dereference-symlinks: resolved symlinks inside directories

Now --dereference-symlinks handles both cases. Users only need one flag
to fully dereference symlinks when adding files to IPFS.

The deprecated --dereference-args still works for backwards compatibility
but is no longer necessary.

* chore: update boxo and improve changelog

- update boxo to ebdaf07c (nil filter fix, thread-safety docs)
- simplify changelog for IPIP-499 section
- shorten test names, move context to comments

* chore: update boxo to 5cf22196

* chore: apply suggestions from code review

Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com>

* test(add): verify balanced DAG layout produces uniform leaf depth

add test that confirms kubo uses balanced layout (all leaves at same
depth) rather than balanced-packed (varying depths). creates 45MiB file
to trigger multi-level DAG and walks it to verify leaf depth uniformity.

includes trickle subtest to validate test logic can detect varying depths.

supports CAR export via DAG_LAYOUT_CAR_OUTPUT env var for test vectors.

* chore(deps): update boxo to 6141039ad8ef

switches to 6141039ad8

changes since 5cf22196ad0b:
- refactor(unixfs): use arithmetic for exact block size calculation
- refactor(unixfs): unify size tracking and make SizeEstimationMode immutable
- feat(unixfs): optimize SizeEstimationBlock and add mode/mtime tests

also clarifies that directory sharding globals affect both `ipfs add` and MFS.

* test(cli): improve HAMT threshold tests with exact +1 byte verification

- add UnixFSDataType() helper to directly check UnixFS type via protobuf
- refactor threshold tests to use exact +1 byte calculations instead of +1 file
- verify directory type directly (ft.TDirectory vs ft.THAMTShard) instead of
  inferring from link count
- clean up helper function signatures by removing unused cidLength parameter

* test(cli): consolidate profile tests into cid_profiles_test.go

remove duplicate profile threshold tests from add_test.go since they
are fully covered by the data-driven tests in cid_profiles_test.go.

changes:
- improve test names to describe what threshold is being tested
- add inline documentation explaining each test's purpose
- add byte-precise helper IPFSAddDeterministicBytes for threshold tests
- remove ~200 lines of duplicated test code from add_test.go
- keep non-profile tests (pinning, symlinks, hidden files) in add_test.go

* chore: update to rebased boxo and go-ipfs-cmds PRs

* docs: add HAMT threshold fix details to changelog

* feat(mfs): use Import config for CID version and hash function

make MFS commands (files cp, files write, files mkdir, files chcid)
respect Import.CidVersion and Import.HashFunction config settings
when CLI options are not explicitly provided.

also add tests for:
- files write respects Import.UnixFSRawLeaves=true
- single-block file: files write produces same CID as ipfs add
- updated comments clarifying CID parity with ipfs add

* feat(files): wire Import.UnixFSChunker and UnixFSDirectoryMaxLinks to MFS

`ipfs files` commands now respect these Import.* config options:
- UnixFSChunker: configures chunk size for `files write`
- UnixFSDirectoryMaxLinks: triggers HAMT sharding in `files mkdir`
- UnixFSHAMTDirectorySizeEstimation: controls size estimation mode

previously, MFS used hardcoded defaults ignoring user config.

changes:
- config/import.go: add UnixFSSplitterFunc() returning chunk.SplitterGen
- core/node/core.go: pass chunker, maxLinks, sizeEstimationMode to
  mfs.NewRoot() via new boxo RootOption API
- core/commands/files.go: pass maxLinks and sizeEstimationMode to
  mfs.Mkdir() and ensureContainingDirectoryExists(); document that
  UnixFSFileMaxLinks doesn't apply to files write (trickle DAG limitation)
- test/cli/files_test.go: add tests for UnixFSDirectoryMaxLinks and
  UnixFSChunker, including CID parity test with `ipfs add --trickle`

related: boxo@54e044f1b265

* feat(files): wire Import.UnixFSHAMTDirectoryMaxFanout and UnixFSHAMTDirectorySizeThreshold

wire remaining HAMT config options to MFS root:
- Import.UnixFSHAMTDirectoryMaxFanout via mfs.WithMaxHAMTFanout
- Import.UnixFSHAMTDirectorySizeThreshold via mfs.WithHAMTShardingSize

add CLI tests:
- files mkdir respects Import.UnixFSHAMTDirectoryMaxFanout
- files mkdir respects Import.UnixFSHAMTDirectorySizeThreshold
- config change takes effect after daemon restart

add UnixFSHAMTFanout() helper to test harness

update boxo to ac97424d99ab90e097fc7c36f285988b596b6f05

* fix(mfs): single-block files in CIDv1 dirs now produce raw CIDs

problem: `ipfs files write` in CIDv1 directories wrapped single-block
files in dag-pb even when raw-leaves was enabled, producing different
CIDs than `ipfs add --raw-leaves` for the same content.

fix: boxo now collapses single-block ProtoNode wrappers (with no
metadata) to RawNode in DagModifier.GetNode(). files with mtime/mode
stay as dag-pb since raw blocks cannot store UnixFS metadata.

also fixes sparse file writes where writing past EOF would lose data
because expandSparse didn't update the internal node pointer.

updates boxo to v0.36.1-0.20260203003133-7884ae23aaff
updates t0250-files-api.sh test hashes to match new behavior

* chore(test): use Go 1.22+ range-over-int syntax

* chore: update boxo to c6829fe26860

- fix typo in files write help text
- update boxo with CI fixes (gofumpt, race condition in test)

* chore: update go-ipfs-cmds to 192ec9d15c1f

includes binary content types fix: gzip, zip, vnd.ipld.car, vnd.ipld.raw,
vnd.ipfs.ipns-record

* chore: update boxo to 0a22cde9225c

includes refactor of maxLinks check in addLinkChild (review feedback).

* ci: fix helia-interop and improve caching

skip '@helia/mfs - should have the same CID after creating a file' test
until helia implements IPIP-499 (tracking: https://github.com/ipfs/helia/issues/941)

the test fails because kubo now collapses single-block files to raw CIDs
while helia explicitly uses reduceSingleLeafToSelf: false

changes:
- run aegir directly instead of helia-interop binary (binary ignores --grep flags)
- cache node_modules keyed by @helia/interop version from npm registry
- skip npm install on cache hit (matches ipfs-webui caching pattern)

* chore: update boxo to 1e30b954

includes latest upstream changes from boxo main

* chore: update go-ipfs-cmds to 1b2a641ed6f6

* chore: update boxo to f188f79fd412

switches to boxo@main after merging https://github.com/ipfs/boxo/pull/1088

* chore: update go-ipfs-cmds to af9bcbaf5709

switches to go-ipfs-cmds@master after merging https://github.com/ipfs/go-ipfs-cmds/pull/315

---------

Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com>
2026-02-04 22:03:15 +01:00
Marcin Rataj
df9574e090
test: IPIP-523 format query precedence over Accept header (#11086)
Some checks are pending
CodeQL / codeql (push) Waiting to run
Docker Check / lint (push) Waiting to run
Docker Check / build (push) Waiting to run
Gateway Conformance / gateway-conformance (push) Waiting to run
Gateway Conformance / gateway-conformance-libp2p-experiment (push) Waiting to run
Go Build / go-build (push) Waiting to run
Go Check / go-check (push) Waiting to run
Go Lint / go-lint (push) Waiting to run
Go Test / unit-tests (push) Waiting to run
Go Test / cli-tests (push) Waiting to run
Go Test / example-tests (push) Waiting to run
Interop / interop-prep (push) Waiting to run
Interop / helia-interop (push) Blocked by required conditions
Interop / ipfs-webui (push) Blocked by required conditions
Sharness / sharness-test (push) Waiting to run
Spell Check / spellcheck (push) Waiting to run
* test: IPIP-523 format query precedence over Accept header

update boxo to ipfs/boxo#1074 and gateway-conformance tests
to ipfs/gateway-conformance#252 for testing IPIP-523 changes
where ?format= URL query parameter takes precedence over
Accept HTTP header

* chore: update boxo and gateway-conformance for IPIP-523 testing

- boxo@9aeb0c835899fedb4f886040a4ccf3ba09fd97d4
- gateway-conformance@c82a2a9bc79303e0f07216a80cf454ef2a1e042c

* chore(ci): switch to gateway-conformance@v0.9

* chore: update boxo with IPIP-523 changes

* chore: update boxo to main after ipfs/boxo#1074 merge

* docs: add gateway-conformance v0.9 to changelog
2026-02-04 05:48:54 +01:00
Andrew Gillis
dd882c0bdb
fix: recognize content-type application/x-tar (#11180)
Some checks are pending
CodeQL / codeql (push) Waiting to run
Docker Check / lint (push) Waiting to run
Docker Check / build (push) Waiting to run
Gateway Conformance / gateway-conformance (push) Waiting to run
Gateway Conformance / gateway-conformance-libp2p-experiment (push) Waiting to run
Go Build / go-build (push) Waiting to run
Go Check / go-check (push) Waiting to run
Go Lint / go-lint (push) Waiting to run
Go Test / unit-tests (push) Waiting to run
Go Test / cli-tests (push) Waiting to run
Go Test / example-tests (push) Waiting to run
Interop / interop-prep (push) Waiting to run
Interop / helia-interop (push) Blocked by required conditions
Interop / ipfs-webui (push) Blocked by required conditions
Sharness / sharness-test (push) Waiting to run
Spell Check / spellcheck (push) Waiting to run
* fix: recognize content-type application/x-tar

For context, see: https://github.com/ipfs/go-ipfs-cmds/pull/320

Closes: #11179
2026-02-03 07:32:21 -08:00
dependabot[bot]
22ca729eb3
chore(deps): bump github.com/hashicorp/go-version from 1.7.0 to 1.8.0 (#11160)
Some checks are pending
CodeQL / codeql (push) Waiting to run
Docker Check / lint (push) Waiting to run
Docker Check / build (push) Waiting to run
Gateway Conformance / gateway-conformance (push) Waiting to run
Gateway Conformance / gateway-conformance-libp2p-experiment (push) Waiting to run
Go Build / go-build (push) Waiting to run
Go Check / go-check (push) Waiting to run
Go Lint / go-lint (push) Waiting to run
Go Test / unit-tests (push) Waiting to run
Go Test / cli-tests (push) Waiting to run
Go Test / example-tests (push) Waiting to run
Interop / interop-prep (push) Waiting to run
Interop / helia-interop (push) Blocked by required conditions
Interop / ipfs-webui (push) Blocked by required conditions
Sharness / sharness-test (push) Waiting to run
Spell Check / spellcheck (push) Waiting to run
* chore(deps): bump github.com/hashicorp/go-version from 1.7.0 to 1.8.0

Bumps [github.com/hashicorp/go-version](https://github.com/hashicorp/go-version) from 1.7.0 to 1.8.0.
- [Release notes](https://github.com/hashicorp/go-version/releases)
- [Changelog](https://github.com/hashicorp/go-version/blob/main/CHANGELOG.md)
- [Commits](https://github.com/hashicorp/go-version/compare/v1.7.0...v1.8.0)

---
updated-dependencies:
- dependency-name: github.com/hashicorp/go-version
  dependency-version: 1.8.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* chore: run make mod_tidy

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com>
2026-02-02 22:08:57 -08:00
dependabot[bot]
3522136c4f
chore(deps): bump github.com/multiformats/go-multiaddr-dns from 0.4.1 to 0.5.0 in the ipfs-ecosystem group (#11177)
Some checks failed
Docker Check / lint (push) Waiting to run
Docker Check / build (push) Waiting to run
Gateway Conformance / gateway-conformance (push) Waiting to run
Gateway Conformance / gateway-conformance-libp2p-experiment (push) Waiting to run
Go Build / go-build (push) Waiting to run
Go Check / go-check (push) Waiting to run
Go Lint / go-lint (push) Waiting to run
Go Test / unit-tests (push) Waiting to run
Go Test / cli-tests (push) Waiting to run
Go Test / example-tests (push) Waiting to run
Interop / interop-prep (push) Waiting to run
Interop / helia-interop (push) Blocked by required conditions
Interop / ipfs-webui (push) Blocked by required conditions
Sharness / sharness-test (push) Waiting to run
Spell Check / spellcheck (push) Waiting to run
CodeQL / codeql (push) Has been cancelled
* chore(deps): bump github.com/multiformats/go-multiaddr-dns

Bumps the ipfs-ecosystem group with 1 update: [github.com/multiformats/go-multiaddr-dns](https://github.com/multiformats/go-multiaddr-dns).


Updates `github.com/multiformats/go-multiaddr-dns` from 0.4.1 to 0.5.0
- [Release notes](https://github.com/multiformats/go-multiaddr-dns/releases)
- [Commits](https://github.com/multiformats/go-multiaddr-dns/compare/v0.4.1...v0.5.0)

---
updated-dependencies:
- dependency-name: github.com/multiformats/go-multiaddr-dns
  dependency-version: 0.5.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: ipfs-ecosystem
...

Signed-off-by: dependabot[bot] <support@github.com>

* chore: run make mod_tidy

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com>
2026-02-02 08:41:16 -08:00
Andrew Gillis
77ed3dd0ef
feat(rpc): Content-Type headers and IPNS record get/put (#11067)
Some checks failed
CodeQL / codeql (push) Has been cancelled
Docker Check / lint (push) Has been cancelled
Docker Check / build (push) Has been cancelled
Gateway Conformance / gateway-conformance (push) Has been cancelled
Gateway Conformance / gateway-conformance-libp2p-experiment (push) Has been cancelled
Go Build / go-build (push) Has been cancelled
Go Check / go-check (push) Has been cancelled
Go Lint / go-lint (push) Has been cancelled
Go Test / unit-tests (push) Has been cancelled
Go Test / cli-tests (push) Has been cancelled
Go Test / example-tests (push) Has been cancelled
Interop / interop-prep (push) Has been cancelled
Sharness / sharness-test (push) Has been cancelled
Spell Check / spellcheck (push) Has been cancelled
Interop / helia-interop (push) Has been cancelled
Interop / ipfs-webui (push) Has been cancelled
* fix http header when compress enabled for get command

Closes #2376

* fix(rpc): set Content-Type for ipfs get based on output format

- set application/x-tar when outputting tar (default and --archive)
- set application/gzip when compression is enabled (--compress)
- update go-ipfs-cmds with Tar encoding type and RFC 6713 compliant
  MIME types (application/gzip instead of application/x-gzip)

* test(rpc): add Content-Type header tests for ipfs get

* feat(rpc): add Content-Type headers for binary responses

set proper Content-Type headers for RPC endpoints that return binary data:

- `dag export`: application/vnd.ipld.car
- `block get`: application/vnd.ipld.raw
- `diag profile`: application/zip
- `get`: application/x-tar or application/gzip (already worked, migrated to new API)

uses the new OctetStream encoding type and SetContentType() method
from go-ipfs-cmds to specify custom MIME types for binary responses.

refs: https://github.com/ipfs/kubo/issues/2376

* feat(rpc): add `ipfs name get` command for IPNS record retrieval

add dedicated command to retrieve raw signed IPNS records from the
routing system. returns protobuf-encoded IPNS record with Content-Type
`application/vnd.ipfs.ipns-record`.

this provides a more convenient alternative to `ipfs routing get /ipns/<name>`
which returns JSON with base64-encoded data. the raw output can be piped
directly to `ipfs name inspect`:

    ipfs name get <name> | ipfs name inspect

spec: https://specs.ipfs.tech/ipns/ipns-record/

* feat(rpc): add `ipfs name put` command for IPNS record storage

adds `ipfs name put` to complement `ipfs name get`, allowing users to
store IPNS records obtained from external sources without needing the
private key. useful for backup, restore, and debugging workflows.

the command validates records by default (signature, sequence number).
use `--force` to bypass validation for testing how routing handles
malformed or outdated records.

also reorganizes test/cli files:
- rename http_rpc_* -> rpc_* to match existing convention
- merge name_get_put_test.go into name_test.go
- add file header comments documenting test purposes

* chore(deps): update go-ipfs-cmds to latest master

includes SetContentType() for dynamic Content-Type headers

---------

Co-authored-by: Marcin Rataj <lidel@lidel.org>
2026-01-30 23:41:55 +01:00
Marcin Rataj
7de7af0820
feat(dns): skip DNS lookups for AutoTLS hostnames (#11140)
* feat(dns): resolve libp2p.direct addresses locally without network I/O

p2p-forge hostnames encode IP addresses directly (e.g., 1-2-3-4.peerID.libp2p.direct -> 1.2.3.4),
so DNS queries are wasteful. kubo now parses these IPs in-memory.

- applies to both default libp2p.direct and custom AutoTLS.DomainSuffix
- TXT queries still delegate to network for ACME DNS-01 compatibility

- https://github.com/ipfs/kubo/pull/11140#discussion_r2683477754
  use fallback to network DNS instead of returning errors when local
  parsing fails, ensuring forward compatibility with future DNS records

- https://github.com/ipfs/kubo/pull/11140#discussion_r2683512408
  add peerID validation using peer.Decode(), matching libp2p.direct
  server behavior, with fallback on invalid peerID

- https://github.com/ipfs/kubo/pull/11140#discussion_r2683521930
  document interaction with DNS.Resolvers in config.md

- https://github.com/ipfs/kubo/pull/11140#discussion_r2683526647
  add AutoTLS.SkipDNSLookup config flag to disable local resolution
  (useful for debugging or custom DNS override scenarios)

- https://github.com/ipfs/kubo/pull/11140#discussion_r2683533462
  add E2E test verifying libp2p.direct resolves locally even when
  DNS.Resolvers points to a broken server

additional improvements:
- use madns.BasicResolver interface instead of custom basicResolver
- add compile-time interface checks for p2pForgeResolver and madns.Resolver
- refactor tests: merge IPv4/IPv6, add helpers, use config.DefaultDomainSuffix
- improve changelog to explain public good benefit (reducing DNS load)

Fixes #11136
2026-01-30 17:20:56 +01:00
Andrew Gillis
a137401272
Upgrade to Boxo v0.36.0 (#11175)
* Upgrade to Boxo v0.36.0
* sharness: add missing metrics
2026-01-27 15:51:01 -08:00
Andrew Gillis
f6a5c5f5c5
chore: upgrade go-ds-pebble to v0.5.9 (#11170)
Some checks are pending
CodeQL / codeql (push) Waiting to run
Docker Check / lint (push) Waiting to run
Docker Check / build (push) Waiting to run
Gateway Conformance / gateway-conformance (push) Waiting to run
Gateway Conformance / gateway-conformance-libp2p-experiment (push) Waiting to run
Go Build / go-build (push) Waiting to run
Go Check / go-check (push) Waiting to run
Go Lint / go-lint (push) Waiting to run
Go Test / unit-tests (push) Waiting to run
Go Test / cli-tests (push) Waiting to run
Go Test / example-tests (push) Waiting to run
Interop / interop-prep (push) Waiting to run
Interop / helia-interop (push) Blocked by required conditions
Interop / ipfs-webui (push) Blocked by required conditions
Sharness / sharness-test (push) Waiting to run
Spell Check / spellcheck (push) Waiting to run
- upgrade go-ds-pebble to [v0.5.9](https://github.com/ipfs/go-ds-pebble/releases/tag/v0.5.9)
- git ignore ipfswatch binary
2026-01-22 23:38:59 -08:00
Marcin Rataj
bc9b388610
fix(kad-dht): peerstore and exploration loop (#11139)
* fix(routing): update kad-dht with peerstore address clone fix

closes #11116

See https://github.com/ipfs/kubo/issues/11116 for context of this fix

* fix(routing): update kad-dht with CPL exploration fix

fixes an infinite loop when all peers share the same CPL during provider exploration

See https://github.com/libp2p/go-libp2p-kad-dht/pull/1216

* fix(routing): update kad-dht with shutdown loop check

https://github.com/libp2p/go-libp2p-kad-dht/pull/1217

* depend on latest kad-dht fix

* bump kad-dht to v0.37.0

---------

Co-authored-by: guillaumemichel <guillaume@michel.id>
Co-authored-by: Guillaume Michel <guillaumemichel@users.noreply.github.com>
2026-01-23 06:41:47 +01:00
segfault_bits
c1fd4d70f5
feat(cli): ls --long (#11103)
Some checks failed
CodeQL / codeql (push) Has been cancelled
Docker Check / lint (push) Has been cancelled
Docker Check / build (push) Has been cancelled
Gateway Conformance / gateway-conformance (push) Has been cancelled
Gateway Conformance / gateway-conformance-libp2p-experiment (push) Has been cancelled
Go Build / go-build (push) Has been cancelled
Go Check / go-check (push) Has been cancelled
Go Lint / go-lint (push) Has been cancelled
Go Test / unit-tests (push) Has been cancelled
Go Test / cli-tests (push) Has been cancelled
Go Test / example-tests (push) Has been cancelled
Interop / interop-prep (push) Has been cancelled
Sharness / sharness-test (push) Has been cancelled
Spell Check / spellcheck (push) Has been cancelled
Interop / helia-interop (push) Has been cancelled
Interop / ipfs-webui (push) Has been cancelled
* Implements the -l/--long flag for the ipfs ls command to display Unix-style
file permissions and modification times, similar to the traditional ls -l.

When the --long flag is used, the output includes:
- File mode/permissions in Unix format (e.g., -rw-r--r--, drwxr-xr-x)
- File hash (CID)
- File size (when --size is also specified)
- Modification time in human-readable format
- File name

The permission string implementation handles all file types and special bits:
- File types: regular (-), directory (d), symlink (l), named pipe (p),
  socket (s), character device (c), block device (b)
- Special permission bits: setuid (s/S), setgid (s/S), sticky (t/T)
  - Lowercase when execute bit is set, uppercase when not set

The timestamp format follows Unix ls conventions:
- Recent files (within 6 months): "Jan 02 15:04"
- Older files: "Jan 02  2006"

Signed-off-by: sneax <paladesh600@gmail.com>

* fix(ls): correct --long flag header order and help text

- fix header column order: was "Mode Hash Size Name ModTime" but data
  outputs "Mode Hash Size ModTime Name", now headers match data order
- remove redundant if/else branch in directory output that had
  identical code in both branches
- add example output to help text showing format with mode, hash,
  size, mtime, and name columns
- document that files without preserved metadata show '----------'
  for mode and '-' for mtime
- add changelog entry for v0.40

* test(ls): add format stability tests for --long flag

add tests to prevent formatting regressions in ipfs ls --long output:

unit tests (core/commands/ls_test.go):
- TestFormatMode: 20 cases covering all file types (regular, dir,
  symlink, pipe, socket, block/char devices) and special permission
  bits (setuid, setgid, sticky with/without execute)
- TestFormatModTime: zero time, old time (year format), future time,
  format length consistency

integration tests (test/cli/ls_test.go):
- explicit full output comparison with deterministic CIDs to catch
  any formatting changes
- header column order verification for --long with --size=true/false
- files without preserved metadata (---------- and - placeholders)
- directory output (trailing slash, d prefix in mode)

requested in: https://github.com/ipfs/kubo/pull/11103#issuecomment-3745043561

* fix(ls): improve --long flag docs and fix minor issues

- improved godocs for formatMode and formatModTime functions
- fixed permBit signature: char rune → char byte (avoids unnecessary cast)
- clarified help text: mode/mtime are optional UnixFS metadata
- documented that times are displayed in UTC
- fixed flaky time test by using 1 month ago instead of 1 hour
- removed hardcoded CID assertion that would break on DAG changes

* fix(ls): show "-" for missing mode in --long output

display "-" instead of "----------" when mode metadata is not preserved.
this avoids ambiguity with Unix mode 0000 and matches how missing mtime
is already displayed. follows common Unix tool conventions (ps, netstat)
where "-" indicates "not available".

---------

Signed-off-by: sneax <paladesh600@gmail.com>
Co-authored-by: Marcin Rataj <lidel@lidel.org>
2026-01-16 02:27:32 +01:00
Marcin Rataj
824a47ae11
feat(pubsub): persistent validation and diagnostic commands (#11110)
Some checks are pending
CodeQL / codeql (push) Waiting to run
Docker Check / lint (push) Waiting to run
Docker Check / build (push) Waiting to run
Gateway Conformance / gateway-conformance (push) Waiting to run
Gateway Conformance / gateway-conformance-libp2p-experiment (push) Waiting to run
Go Build / go-build (push) Waiting to run
Go Check / go-check (push) Waiting to run
Go Lint / go-lint (push) Waiting to run
Go Test / unit-tests (push) Waiting to run
Go Test / cli-tests (push) Waiting to run
Go Test / example-tests (push) Waiting to run
Interop / interop-prep (push) Waiting to run
Interop / helia-interop (push) Blocked by required conditions
Interop / ipfs-webui (push) Blocked by required conditions
Sharness / sharness-test (push) Waiting to run
Spell Check / spellcheck (push) Waiting to run
* feat(pubsub): persistent seqno validation and diagnostic commands

- upgrade go-libp2p-pubsub to v0.15.0
- add persistent seqno validator using BasicSeqnoValidator
  stores max seen seqno per peer at /pubsub/seqno/<peerid>
  survives daemon restarts, addresses message cycling in large networks (#9665)
- add `ipfs pubsub reset` command to clear validator state
- add `ipfs diag datastore get/count` commands for datastore inspection
  requires daemon to be stopped, useful for debugging
- change pubsub status from Deprecated to Experimental
- add CLI tests for pubsub and diag datastore commands
- remove flaky pubsub_msg_seen_cache_test.go (replaced by CLI tests)

* fix(pubsub): improve reset command and add deprecation warnings

- use batched delete for efficient bulk reset
- check key existence before reporting deleted count
- sync datastore after deletions to ensure persistence
- show "no validator state found" when resetting non-existent peer
- log deprecation warnings when using --enable-pubsub-experiment
  or --enable-namesys-pubsub CLI flags

* refactor(test): add datastore helpers to test harness

---------

Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com>
2026-01-16 00:27:09 +01:00
Marcin Rataj
edb7056747
feat(config): add Gateway.MaxRequestDuration option (#11138)
* feat(config): add Gateway.MaxRequestDuration option

exposes the previously hardcoded 1 hour gateway request deadline as a
configurable option, allowing operators to adjust it to fit deployment
needs. protects gateway from edge cases and slow client attacks.

boxo: https://github.com/ipfs/boxo/pull/1079

* test(gateway): add MaxRequestDuration integration test

verifies config is wired correctly and 504 is returned when exceeded

* docs: add MaxRequestDuration to gateway production guide

---------

Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com>
2026-01-15 19:31:16 +01:00
Andrew Gillis
6a595c27a9
upgrade go-libp2p-pubsub to v0.15.0 (#11144) 2026-01-13 20:27:58 -08:00
Lucas Molas
39c609b3db
feat(mfs): chroot command to change the root (#8648)
Some checks are pending
CodeQL / codeql (push) Waiting to run
Docker Check / lint (push) Waiting to run
Docker Check / build (push) Waiting to run
Gateway Conformance / gateway-conformance (push) Waiting to run
Gateway Conformance / gateway-conformance-libp2p-experiment (push) Waiting to run
Go Build / go-build (push) Waiting to run
Go Check / go-check (push) Waiting to run
Go Lint / go-lint (push) Waiting to run
Go Test / unit-tests (push) Waiting to run
Go Test / cli-tests (push) Waiting to run
Go Test / example-tests (push) Waiting to run
Interop / interop-prep (push) Waiting to run
Interop / helia-interop (push) Blocked by required conditions
Interop / ipfs-webui (push) Blocked by required conditions
Sharness / sharness-test (push) Waiting to run
Spell Check / spellcheck (push) Waiting to run
Co-authored-by: Marcin Rataj <lidel@lidel.org>
2026-01-13 21:47:36 +01:00
Andrew Gillis
56bf782cc6
test: fix flaky ipfswatch test (#11142)
Gracefully shutdown the ipfs node to ensure data is saved. Forceful shutdown is done if there is an error sending the interrupt signal to the process, such as on Windows.
2026-01-13 02:38:44 -08:00
Marcin Rataj
25ebab9dae
feat(p2p): add --foreground flag to listen and forward commands (#11099)
* feat(p2p): add --foreground flag to listen and forward commands

adds `-f/--foreground` option that keeps the command running until
interrupted (SIGTERM/Ctrl+C) or closed via `ipfs p2p close`. the
listener/forwarder is automatically removed when the command exits.

useful for systemd services and scripts that need cleanup on exit.

* docs: add p2p-tunnels.md with systemd examples

- add dedicated docs/p2p-tunnels.md covering:
  - why p2p tunnels (NAT traversal, no public IP needed)
  - quick start with netcat
  - background and foreground modes
  - systemd integration with path-based activation
  - security considerations and troubleshooting
- document Experimental.Libp2pStreamMounting in docs/config.md
- simplify docs/experimental-features.md, link to new doc
- add "Learn more" links to ipfs p2p listen/forward --help
- update changelog entry with doc link
- add cross-reference in misc/README.md

* chore: reference kubo#5460 for p2p config

Ref. https://github.com/ipfs/kubo/issues/5460

* fix(daemon): write api/gateway files only after HTTP server is ready

fixes race condition where $IPFS_PATH/api and $IPFS_PATH/gateway files
were written before the HTTP servers were ready to accept connections.
this caused issues for tools like systemd path units that immediately
try to connect when these files appear.

changes:
- add corehttp.ServeWithReady() that signals when server is ready
- wait for ready signal before writing address files
- use sync.WaitGroup.Go() (Go 1.25) for cleaner goroutine management
- add TestAddressFileReady to verify both api and gateway files

* fix(daemon): buffer errc channel and wait for all listeners

- buffer error channel with len(listeners) to prevent deadlock when
  multiple servers write errors simultaneously
- wait for ALL listeners to be ready before writing api/gateway file,
  not just the first one

Feedback-from: https://github.com/ipfs/kubo/pull/11099#pullrequestreview-3593885839

* docs(changelog): improve p2p tunnel section clarity

reframe to lead with user benefit and add example output

* docs(p2p): remove obsolete race condition caveat

the "First launch fails but restarts work" troubleshooting section
described a race where the api file was written before the daemon was
ready. this was fixed in 80b703a which ensures api/gateway files are
only written after HTTP servers are ready to accept connections.

---------

Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com>
2026-01-09 19:22:43 +01:00
Marcin Rataj
5288946fd1
feat(cli): improve ipfs dag stat output UX (#11097)
- add TTY auto-detection for progress display (matching `dag export`)
- use single-line progress with carriage return instead of flooding
- show human-readable sizes alongside raw bytes in summary
- update --progress flag to be auto-detected by default

progress format: `Fetched/Processed N blocks, M bytes (X MB)`
summary format: `Total Size: 99 (99 B)`
2026-01-09 18:41:10 +01:00
Marcin Rataj
23ba660ef0
fix(routing): use LegacyProvider for HTTP-only custom routing (#11112)
* fix(routing): use LegacyProvider for HTTP-only custom routing

when `Routing.Type=custom` with only HTTP routers and no DHT,
fall back to LegacyProvider instead of SweepingProvider.

SweepingProvider requires a DHT client which is unavailable in
HTTP-only configurations, causing it to return NoopProvider and
breaking provider record announcements to HTTP routers.

fixes #11089

* test(routing): verify provide stat works with HTTP-only routing

* docs(config): clarify SweepEnabled fallback for HTTP-only routing

---------

Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com>
2026-01-09 18:23:31 +01:00
Andrew Gillis
aa3c88dcdd
shutdown daemon after test (#11135)
Some checks failed
CodeQL / codeql (push) Has been cancelled
Docker Check / lint (push) Has been cancelled
Docker Check / build (push) Has been cancelled
Gateway Conformance / gateway-conformance (push) Has been cancelled
Gateway Conformance / gateway-conformance-libp2p-experiment (push) Has been cancelled
Go Build / go-build (push) Has been cancelled
Go Check / go-check (push) Has been cancelled
Go Lint / go-lint (push) Has been cancelled
Go Test / unit-tests (push) Has been cancelled
Go Test / cli-tests (push) Has been cancelled
Go Test / example-tests (push) Has been cancelled
Interop / interop-prep (push) Has been cancelled
Sharness / sharness-test (push) Has been cancelled
Spell Check / spellcheck (push) Has been cancelled
Interop / helia-interop (push) Has been cancelled
Interop / ipfs-webui (push) Has been cancelled
2026-01-07 20:51:19 -08:00
Marcin Rataj
1301710a91
fix(ci): parallelize gotest, cleanup output, flakiness (#11113)
* ci: parallelize gotest by separating test/cli into own job

split the Go Test workflow into two parallel jobs:
- `unit-tests`: runs unit tests (excluding test/cli)
- `cli-tests`: runs test/cli end-to-end tests

test/cli takes ~3 minutes (~50% of total gotest time), so running
it in parallel should reduce wall-clock CI time by ~1.5-2.5 minutes.

both jobs produce JUnit XML and HTML reports for consistent debugging.

* ci(gotest): reduce noise on test timeout panics

add GOTRACEBACK=single to show only one goroutine stack instead of all
when a test timeout panic occurs. this makes CI output much cleaner
when tests hang.

* fix(ci): prevent stderr from corrupting test JSON output

- remove 2>&1 which mixed "go: downloading" stderr messages into JSON
- add JSON validation before parsing
- print failed test names for easier debugging

* ci(gotest): use gotestsum for human-readable test output

- replace per-package coverage loop with single gotestsum invocation
- both unit-tests and cli-tests now show human-readable output
- simplified coverage collection (single -coverprofile, no gocovmerge)
- clarified step names to indicate they run tests

* ci: fix codecov uploads by adding token

- add CODECOV_TOKEN to gotest.yml and sharness.yml
- update codecov-action to v5.5.2
- add fail_ci_if_error: false for robustness

codecov stopped receiving coverage data ~1 year ago when they
started requiring tokens for public repos

* refactor(make): add test_unit and test_cli targets

- add `make test_unit` for unit tests with coverage (used by CI)
- add `make test_cli` for CLI integration tests (used by CI)
- only disable colors when CI env var is set (local dev gets colors)
- remove legacy targets: test_go_test, test_go_short, test_go_race, test_go_expensive
- update gotest.yml to use make targets instead of inline commands
- add test artifacts to .gitignore

* fix(ci): move client/rpc tests to cli-tests job

client/rpc tests use test/cli/harness which requires the ipfs binary.
Move them from test_unit to test_cli where the binary is built.

also:
- update gotestsum to v1.13.0
- simplify workflow step names

* fix(ci): use build tags when listing test packages

go list needs build tags to properly exclude packages like fuse/mfs
when running with TEST_FUSE=0 (nofuse tag).

* fix(ci): move test/integration to cli-tests job

test/integration tests need the ipfs binary, move them from test_unit
to test_cli.

* fix(test): fix flaky kubo-as-a-library and GetClosestPeers tests

kubo-as-a-library: use `Bootstrap()` instead of raw `Swarm().Connect()`
to fix race condition between swarm connection and bitswap peer
discovery. `Bootstrap()` properly integrates peers into the routing
system, ensuring bitswap learns about connected peers synchronously.

GetClosestPeers: simplify retry logic using `EventuallyWithT` with
10-minute timeout. tests all 4 routing types (`auto`, `autoclient`,
`dht`, `dhtclient`) against real bootstrap peers with patient polling.

* fix(example): use bidirectional Swarm().Connect() for reliable bitswap

- connect nodes bidirectionally (A→B and B→A) to simulate mutual peering
- mutual peering protects connection from resource manager culling
- use port 0 for random available ports (avoids CI conflicts)
- enable LoopbackAddressesOnLanDHT for local testing
- move retry logic to test file using require.Eventually

* fix(ci): add test_examples target and parallel example-tests job

- add `make test_examples` target to mk/golang.mk for consistency with test_unit/test_cli
- move example tests to separate parallel CI job (example-tests)
- example: use Bootstrap() with autoconf.FallbackBootstrapPeers for reliable bitswap
- example: increase context timeout to 10 minutes
- test: add 60s per-request timeout to GetClosestPeers (server has 30s routing timeout)
- test: reduce EventuallyWithT to 3 minutes (locally passes in under 1 minute)

* fix(ci): improve test targets, exclusion patterns, and artifact naming

- define COVERPKG_EXCLUDE and UNIT_EXCLUDE as documented variables
- use grep -vE with single regex instead of multiple grep -v calls
- add mkdir -p before rm to ensure directories exist
- add DEPS_GO dependency to test_cli target
- make CLI test timeout configurable via TEST_CLI_TIMEOUT (default 10m)
- fix test_examples cleanup on failure using subshell
- reduce GetClosestPeers test wait time from 3m to 2m
- rename artifacts to match job names: unit-tests-{junit,html}, cli-tests-{junit,html}
- update cli-tests upload-artifact from v5 to v6

* fix(ci): fix unit test exclusion and speed up example test

- fix UNIT_EXCLUDE regex to match client/rpc at end of path
- remove public bootstrap peers from example (only connect to nodeA)
- example test now runs in ~3s instead of timing out

* fix(test): fix flaky TestAddMultipleGCLive race condition

added time.Sleep after spawning GC goroutines to ensure they reach
GCLock() before the test proceeds. without this, the adder's
maybePauseForGC() might check GCRequested() before GC has even
requested the lock, causing the lock to not be released and GC to
block indefinitely.

this matches the existing pattern in TestAddGCLive which already
had this sleep.

also replaced context.Background() with t.Context() in both
TestAddMultipleGCLive and TestAddGCLive for proper test lifecycle
management.

* fix(example): use test harness settings for reliable CI

the kubo-as-a-library example was flaky on CI. applied test-harness-like
settings that match what transports_test.go uses:

- TCP-only on 127.0.0.1 with random port (no QUIC/UDP)
- explicitly disable non-TCP transports (QUIC, Relay, WebTransport, etc)
- use NilRouterOption (no routing) since we connect peers directly
- bitswap works with directly connected peers without DHT lookups
- 2-minute context timeout
- streaming output in test for debugging
2026-01-08 05:07:08 +01:00
Andrew Gillis
828526e515
datastore: upgrade go-ds-pebble to v0.5.8 (#11129)
Some checks are pending
CodeQL / codeql (push) Waiting to run
Docker Check / lint (push) Waiting to run
Docker Check / build (push) Waiting to run
Gateway Conformance / gateway-conformance (push) Waiting to run
Gateway Conformance / gateway-conformance-libp2p-experiment (push) Waiting to run
Go Build / go-build (push) Waiting to run
Go Check / go-check (push) Waiting to run
Go Lint / go-lint (push) Waiting to run
Go Test / go-test (push) Waiting to run
Interop / interop-prep (push) Waiting to run
Interop / helia-interop (push) Blocked by required conditions
Interop / ipfs-webui (push) Blocked by required conditions
Sharness / sharness-test (push) Waiting to run
Spell Check / spellcheck (push) Waiting to run
* datastore: upgrade go-ds-pebble to v0.5.8

- update 'go-ds-pebble` to [v0.5.8](https://github.com/ipfs/go-ds-pebble/releases/tag/v0.5.8)
  - updates github.com/cockroachdb/pebble to [v2.1.3](https://github.com/cockroachdb/pebble/releases/tag/v2.1.3)
    - enables Go 1.26 support
2026-01-06 01:38:03 -08:00
Marcin Rataj
91d392d9ae
chore: update go-libp2p 0.46 (#11105)
* fix: update go-libp2p to v0.46.0

- reduced WebRTC log noise (go-libp2p#3426)
- fixed mDNS discovery on Windows/macOS (go-libp2p#3434)
- includes quic-go v0.57.1 (v0.56.0 + v0.57.0)

* fix(example): kubo-as-a-library test timeout

- use custom ports (4010/4011) to avoid conflicts with default 4001
- add 2-minute context timeout to fail fast
- get peer addresses dynamically instead of hardcoding wrong port
- wait for peer connection synchronously instead of fire-and-forget
- update comments to reference autoconf.FallbackBootstrapPeers

* chore: update p2p-forge to v0.7.0

* fix(test): wait for DHT readiness in GetClosestPeers test

the test was failing for `routing_type=auto` because it only waited for
swarm connections but not for the DHT routing table to be populated.
added a separate probe loop that waits for GetClosestPeers to succeed
before running the actual test assertions.
2025-12-16 19:03:08 +01:00
Andrew Gillis
ab44726177
fix(ipfswatch): loading datastore plugins (#11078)
Some checks failed
CodeQL / codeql (push) Has been cancelled
Docker Check / lint (push) Has been cancelled
Docker Check / build (push) Has been cancelled
Gateway Conformance / gateway-conformance (push) Has been cancelled
Gateway Conformance / gateway-conformance-libp2p-experiment (push) Has been cancelled
Go Build / go-build (push) Has been cancelled
Go Check / go-check (push) Has been cancelled
Go Lint / go-lint (push) Has been cancelled
Go Test / go-test (push) Has been cancelled
Interop / interop-prep (push) Has been cancelled
Sharness / sharness-test (push) Has been cancelled
Spell Check / spellcheck (push) Has been cancelled
Interop / helia-interop (push) Has been cancelled
Interop / ipfs-webui (push) Has been cancelled
* ipfswatch: fix loading datastore plugins
* test: add CLI tests for ipfswatch

---------

Co-authored-by: Marcin Rataj <lidel@lidel.org>
2025-12-10 02:50:59 +01:00
vlerdman
d29c0b9c01
Add bytes progress tracker for ipfs pin add (#11074)
Some checks failed
CodeQL / codeql (push) Has been cancelled
Docker Check / lint (push) Has been cancelled
Docker Check / build (push) Has been cancelled
Gateway Conformance / gateway-conformance (push) Has been cancelled
Gateway Conformance / gateway-conformance-libp2p-experiment (push) Has been cancelled
Go Build / go-build (push) Has been cancelled
Go Check / go-check (push) Has been cancelled
Go Lint / go-lint (push) Has been cancelled
Go Test / go-test (push) Has been cancelled
Interop / interop-prep (push) Has been cancelled
Sharness / sharness-test (push) Has been cancelled
Spell Check / spellcheck (push) Has been cancelled
Interop / helia-interop (push) Has been cancelled
Interop / ipfs-webui (push) Has been cancelled
* Add bytes progress tracker for ipfs pin add
* upgrade to boxo that has ipfs/boxo#1071
2025-12-02 17:11:22 -08:00
rifeplight
3f2cc50eb8
refactor: replace context.WithCancel with t.Context (#11083)
Some checks failed
CodeQL / codeql (push) Waiting to run
Docker Check / lint (push) Waiting to run
Docker Check / build (push) Waiting to run
Gateway Conformance / gateway-conformance (push) Waiting to run
Gateway Conformance / gateway-conformance-libp2p-experiment (push) Waiting to run
Go Build / go-build (push) Waiting to run
Go Check / go-check (push) Waiting to run
Go Lint / go-lint (push) Waiting to run
Go Test / go-test (push) Waiting to run
Interop / interop-prep (push) Waiting to run
Interop / helia-interop (push) Blocked by required conditions
Interop / ipfs-webui (push) Blocked by required conditions
Sharness / sharness-test (push) Waiting to run
Spell Check / spellcheck (push) Waiting to run
Migrations / test (macos-latest) (push) Has been cancelled
Migrations / test (ubuntu-latest) (push) Has been cancelled
Migrations / test (windows-latest) (push) Has been cancelled
Signed-off-by: rifeplight <rifeplight@outlook.com>
2025-11-28 16:39:55 +01:00
Andrew Gillis
21f50ac931
upgrade go-libp2p-kad-dht to v0.36.0 (#11079) 2025-11-26 11:31:22 -08:00
Andrew Gillis
1141220674
upgrade pebble to 2.1.2 (#11075) 2025-11-25 15:25:55 +00:00
Hector Sanjuan
73ab037d1d
feat: support GetClosesPeers (IPIP-476) and ExposeRoutingAPI by default (#10954)
Some checks are pending
CodeQL / codeql (push) Waiting to run
Docker Check / lint (push) Waiting to run
Docker Check / build (push) Waiting to run
Gateway Conformance / gateway-conformance (push) Waiting to run
Gateway Conformance / gateway-conformance-libp2p-experiment (push) Waiting to run
Go Build / go-build (push) Waiting to run
Go Check / go-check (push) Waiting to run
Go Lint / go-lint (push) Waiting to run
Go Test / go-test (push) Waiting to run
Interop / interop-prep (push) Waiting to run
Interop / helia-interop (push) Blocked by required conditions
Interop / ipfs-webui (push) Blocked by required conditions
Sharness / sharness-test (push) Waiting to run
Spell Check / spellcheck (push) Waiting to run
This allows Kubo to respond to the GetClosestPeers() http routing v1 endpoint
as spec'ed here: https://github.com/ipfs/specs/pull/476

It is based on work from https://github.com/ipfs/boxo/pull/1021

We let IpfsNode implmement the contentRouter.Client interface with the new
method.  We use our WAN-DHT to get the closest peers. 

Additionally, Routing V1 HTTP API is exposed by default which enables light clients in browsers to use Kubo Gateway as delegated routing backend

Co-authored-by: Marcin Rataj <lidel@lidel.org>
2025-11-19 10:51:56 +00:00
Marcin Rataj
597f2b827d
test: add regression tests for config secrets protection (#11061) 2025-11-17 19:10:40 +01:00
Marcin Rataj
1404861086
test: add regression tests for API.Authorizations (#11060) 2025-11-17 18:52:05 +01:00
Marcin Rataj
c7eda21d68
test: verifyWorkerRun and helptext (#11063) 2025-11-17 18:51:33 +01:00
Marcin Rataj
cec7432043
feat: fast provide support in dag import (#11058)
Some checks failed
CodeQL / codeql (push) Has been cancelled
Docker Check / lint (push) Has been cancelled
Docker Check / build (push) Has been cancelled
Gateway Conformance / gateway-conformance (push) Has been cancelled
Gateway Conformance / gateway-conformance-libp2p-experiment (push) Has been cancelled
Go Build / go-build (push) Has been cancelled
Go Check / go-check (push) Has been cancelled
Go Lint / go-lint (push) Has been cancelled
Go Test / go-test (push) Has been cancelled
Interop / interop-prep (push) Has been cancelled
Sharness / sharness-test (push) Has been cancelled
Spell Check / spellcheck (push) Has been cancelled
Interop / helia-interop (push) Has been cancelled
Interop / ipfs-webui (push) Has been cancelled
* fix(add): respect Provide config in fast-provide-root

fast-provide-root should honor the same config settings as the regular
provide system:
- skip when Provide.Enabled is false
- skip when Provide.DHT.Interval is 0
- respect Provide.Strategy (all/pinned/roots/mfs/combinations)

This ensures fast-provide only runs when appropriate based on user
configuration and the nature of the content being added (pinned vs
unpinned, added to MFS or not).

* feat(config): options to adjust global defaults

Add Import.FastProvideRoot and Import.FastProvideWait configuration options
to control default behavior of fast-provide-root and fast-provide-wait flags
in ipfs add command. Users can now set global defaults in config while
maintaining per-command flag overrides.

- Add Import.FastProvideRoot (default: true)
- Add Import.FastProvideWait (default: false)
- Add ResolveBoolFromConfig helper for config resolution
- Update docs with configuration details
- Add log-based tests verifying actual behavior

* refactor: extract fast-provide logic into reusable functions

Extract fast-provide logic from add command into reusable components:
- Add config.ShouldProvideForStrategy helper for strategy matching
- Add ExecuteFastProvide function reusable across add and dag import commands
- Move DefaultFastProvideTimeout constant to config/provide.go
- Simplify add.go from 72 lines to 6 lines for fast-provide
- Move fast-provide tests to dedicated TestAddFastProvide function

Benefits:
- cleaner API: callers only pass content characteristics
- all strategy logic centralized in one place
- better separation of concerns
- easier to add fast-provide to other commands in future

* feat(dag): add fast-provide support for dag import

Adds --fast-provide-root and --fast-provide-wait flags to `ipfs dag import`,
mirroring the fast-provide functionality available in `ipfs add`.

Changes:
- Add --fast-provide-root and --fast-provide-wait flags to dag import command
- Implement fast-provide logic for all root CIDs in imported CAR files
- Works even when --pin-roots=false (strategy checked internally)
- Share ExecuteFastProvide implementation between add and dag import
- Move ExecuteFastProvide to cmdenv package to avoid import cycles
- Add logging when fast-provide is disabled
- Conditional error handling: return error when wait=true, warn when wait=false
- Update config docs to mention both ipfs add and ipfs dag import
- Update changelog to use "provide" terminology and include dag import examples
- Add comprehensive test coverage (TestDagImportFastProvide with 6 test cases)

The fast-provide feature allows immediate DHT announcement of root CIDs
for faster content discovery, bypassing the regular background queue.

* docs: improve fast-provide documentation

Refine documentation to better explain fast-provide and sweep provider working
together, and highlight the performance improvement.

Changelog:
- add fast-provide to sweep provider features list
- explain performance improvement: root CIDs discoverable in <1s vs 30+ seconds
- note this uses optimistic DHT operations (faster with sweep provider)
- simplify examples, point to --help for details

Config docs:
- fix: --fast-provide-roots should be --fast-provide-root (singular)
- clarify Import.FastProvideRoot focuses on root CIDs while sweep handles all blocks
- simplify Import.FastProvideWait description

Command help:
- ipfs add: explain sweep provider context upfront
- ipfs dag import: add fast-provide explanation section
- both explain the split: fast-provide for roots, sweep for all blocks

* test: add tests for ShouldProvideForStrategy

add tests covering all provide strategy combinations with focus on
bitflag OR logic (the else-if bug fix). organized by behavior:
- all strategy always provides
- single strategies match only their flag
- combined strategies use OR logic
- zero strategy never provides

* refactor: error cmd on error and wait=true

change ExecuteFastProvide() to return error, enabling proper error
propagation when --fast-provide-wait=true. in sync mode, provide
failures now error the command as expected. in async mode (default),
always returns nil with errors logged in background goroutine.

also remove duplicate ExecuteFastProvide() from provide.go (75 lines),
keeping single implementation in cmdenv/env.go for reuse across add
and dag import commands.

call sites simplified:
- add.go: check and propagate error from ExecuteFastProvide
- dag/import.go: return error from ForEach callback, remove confusing
  conditional error handling

semantics:
- precondition skips (DHT unavailable, etc): return nil (not failure)
- async mode (wait=false): return nil, log errors in goroutine
- sync mode (wait=true): return wrapped error on provide failure
2025-11-14 21:06:25 -08:00
Hector Sanjuan
d45c615e73
feat(telemetry): collect high level provide DHT sweep settings (#11056)
Some checks failed
CodeQL / codeql (push) Has been cancelled
Docker Check / lint (push) Has been cancelled
Docker Check / build (push) Has been cancelled
Gateway Conformance / gateway-conformance (push) Has been cancelled
Gateway Conformance / gateway-conformance-libp2p-experiment (push) Has been cancelled
Go Build / go-build (push) Has been cancelled
Go Check / go-check (push) Has been cancelled
Go Lint / go-lint (push) Has been cancelled
Go Test / go-test (push) Has been cancelled
Interop / interop-prep (push) Has been cancelled
Sharness / sharness-test (push) Has been cancelled
Spell Check / spellcheck (push) Has been cancelled
Interop / helia-interop (push) Has been cancelled
Interop / ipfs-webui (push) Has been cancelled
* telemetry: collect provideDHTSweepEnabled

Fixes #11055.

* telemetry: track custom Provide.DHT.Interval and MaxWorkers

collects whether users customize Interval and MaxWorkers from defaults
to help identify if defaults need adjustment

* docs: improve telemetry documentation structure and clarity

restructure docs/telemetry.md into meaningful sections (routing & discovery,
content providing, network configuration), add exact config field paths for all
tracked settings, and establish code as source of truth by linking from LogEvent
struct while removing redundant field comments

---------

Co-authored-by: Marcin Rataj <lidel@lidel.org>
2025-11-13 01:28:15 +01:00
Guillaume Michel
702c63b6db
feat: enable DHT Provide Sweep by default (#10955)
Co-authored-by: Marcin Rataj <lidel@lidel.org>
Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com>
2025-11-12 23:55:17 +01:00
Andrew Gillis
149ca2fd3b
Upgrade to Boxo v0.35.2 (#11050)
Some checks failed
CodeQL / codeql (push) Has been cancelled
Docker Check / lint (push) Has been cancelled
Docker Check / build (push) Has been cancelled
Gateway Conformance / gateway-conformance (push) Has been cancelled
Gateway Conformance / gateway-conformance-libp2p-experiment (push) Has been cancelled
Go Build / go-build (push) Has been cancelled
Go Check / go-check (push) Has been cancelled
Go Lint / go-lint (push) Has been cancelled
Go Test / go-test (push) Has been cancelled
Interop / interop-prep (push) Has been cancelled
Sharness / sharness-test (push) Has been cancelled
Spell Check / spellcheck (push) Has been cancelled
Interop / helia-interop (push) Has been cancelled
Interop / ipfs-webui (push) Has been cancelled
* Upgrade to Boxo v0.35.2
2025-11-06 10:30:03 -08:00
Marcin Rataj
f067a86170
fix(go-log@2.9/go-libp2p@0.45): dynamic log level control and tail (#11039)
This fix restores dynamic log level control and tail for go-libp2p loggers

Updated to:
https://github.com/libp2p/go-libp2p/releases/tag/v0.45.0
https://github.com/ipfs/go-log/releases/tag/v2.9.0

these changes restore dynamic log level control and tail for go-libp2p
subsystems after the migration to slog, fixing the regression introduced
in https://github.com/libp2p/go-libp2p/pull/3364

Fixes https://github.com/ipfs/kubo/issues/11035

For details why and how, see explainer in
https://github.com/ipfs/go-log/releases/tag/v2.9.0
2025-11-06 16:00:56 +01:00
Andrew Gillis
a4323abc10
Upgrade to Boxo v0.35.1 (#11043)
Some checks are pending
CodeQL / codeql (push) Waiting to run
Docker Check / lint (push) Waiting to run
Docker Check / build (push) Waiting to run
Gateway Conformance / gateway-conformance (push) Waiting to run
Gateway Conformance / gateway-conformance-libp2p-experiment (push) Waiting to run
Go Build / go-build (push) Waiting to run
Go Check / go-check (push) Waiting to run
Go Lint / go-lint (push) Waiting to run
Go Test / go-test (push) Waiting to run
Interop / interop-prep (push) Waiting to run
Interop / helia-interop (push) Blocked by required conditions
Interop / ipfs-webui (push) Blocked by required conditions
Sharness / sharness-test (push) Waiting to run
Spell Check / spellcheck (push) Waiting to run
* Upgrade to Boxo v0.35.1
* use tagged boxo release
* fix lint error
2025-10-29 17:18:34 -07:00
Guillaume Michel
c2bf0f9515
feat(provider): resume cycle (#11031)
Some checks are pending
CodeQL / codeql (push) Waiting to run
Docker Check / lint (push) Waiting to run
Docker Check / build (push) Waiting to run
Gateway Conformance / gateway-conformance (push) Waiting to run
Gateway Conformance / gateway-conformance-libp2p-experiment (push) Waiting to run
Go Build / go-build (push) Waiting to run
Go Check / go-check (push) Waiting to run
Go Lint / go-lint (push) Waiting to run
Go Test / go-test (push) Waiting to run
Interop / interop-prep (push) Waiting to run
Interop / helia-interop (push) Blocked by required conditions
Interop / ipfs-webui (push) Blocked by required conditions
Sharness / sharness-test (push) Waiting to run
Spell Check / spellcheck (push) Waiting to run
* bump kad-dht: resume reprovide cycle

* daemon: --provide-fresh-start flag

* changelog

* docs

* go-fmt

* chore: latest go-libp2p-kad-dht#1170

after conflict resolution, to confirm CI is still green

* kad-dht: depend on latest master

* move daemon flag to Provider.DHT.ResumeEnabled config

* refactor: sweep provider datastore

* bump kad-dht

* bump kad-dht

* bump kad-dht

* make datastore keys constant

* use kad-dht master

* add emoji to changelog entry

* go-fmt

* bump kad-dht

* test(provider): add tests for resume cycle feature

validates Provide.DHT.ResumeEnabled behavior:
- preserves cycle state when enabled (default)
- resets cycle when disabled

tests verify current_time_offset across restarts using JSON output

---------

Co-authored-by: Marcin Rataj <lidel@lidel.org>
2025-10-29 11:07:46 +01:00
Andrew Gillis
2e9c4ec500
chore: upgrade pebble to v2.1.1 (#11040)
Some checks failed
CodeQL / codeql (push) Waiting to run
Docker Check / lint (push) Waiting to run
Docker Check / build (push) Waiting to run
Gateway Conformance / gateway-conformance (push) Waiting to run
Gateway Conformance / gateway-conformance-libp2p-experiment (push) Waiting to run
Go Build / go-build (push) Waiting to run
Go Check / go-check (push) Waiting to run
Go Lint / go-lint (push) Waiting to run
Go Test / go-test (push) Waiting to run
Interop / interop-prep (push) Waiting to run
Interop / helia-interop (push) Blocked by required conditions
Interop / ipfs-webui (push) Blocked by required conditions
Sharness / sharness-test (push) Waiting to run
Spell Check / spellcheck (push) Waiting to run
Migrations / test (macos-latest) (push) Has been cancelled
Migrations / test (ubuntu-latest) (push) Has been cancelled
Migrations / test (windows-latest) (push) Has been cancelled
2025-10-28 23:15:32 +01:00
Guillaume Michel
8c5f302d25
fix(cli): provide stat cosmetics (#11034) 2025-10-28 22:38:37 +01:00
Marcin Rataj
ae78c7821c
fix: go-libp2p v0.44 with self-healing UPnP port mappings (#11032)
Some checks failed
CodeQL / codeql (push) Has been cancelled
Docker Check / lint (push) Has been cancelled
Docker Check / build (push) Has been cancelled
Gateway Conformance / gateway-conformance (push) Has been cancelled
Gateway Conformance / gateway-conformance-libp2p-experiment (push) Has been cancelled
Go Build / go-build (push) Has been cancelled
Go Check / go-check (push) Has been cancelled
Go Lint / go-lint (push) Has been cancelled
Go Test / go-test (push) Has been cancelled
Interop / interop-prep (push) Has been cancelled
Sharness / sharness-test (push) Has been cancelled
Spell Check / spellcheck (push) Has been cancelled
Interop / helia-interop (push) Has been cancelled
Interop / ipfs-webui (push) Has been cancelled
* chore(deps): update go-libp2p to v0.44.0

- includes self-healing UPnP port mappings after router restarts
- update go-netroute to v0.3.0
- update quic-go to v0.55.0
- add changelog entry for UPnP fix

* docs: improve provide and UPnP clarity in changelog and docs

- add alert polling rationale to changelog
- add UPnP config note with default clarification
- clarify sweep timing and prefix length explanations
- add concrete examples for time offset and record holders
- improve workers stats formatting
- add See Also section to provide-stats.md

* docs: add RISC-V prebuilt binaries to changelog and README

- highlight linux-riscv64 availability with open hardware context
- update README with arm64 builds, remove 32-bit examples
2025-10-25 01:13:34 +02:00
Guillaume Michel
16479ec692
feat(provide): detailed ipfs provide stat (#11019)
Some checks are pending
CodeQL / codeql (push) Waiting to run
Docker Check / lint (push) Waiting to run
Docker Check / build (push) Waiting to run
Gateway Conformance / gateway-conformance (push) Waiting to run
Gateway Conformance / gateway-conformance-libp2p-experiment (push) Waiting to run
Go Build / go-build (push) Waiting to run
Go Check / go-check (push) Waiting to run
Go Lint / go-lint (push) Waiting to run
Go Test / go-test (push) Waiting to run
Interop / interop-prep (push) Waiting to run
Interop / helia-interop (push) Blocked by required conditions
Interop / ipfs-webui (push) Blocked by required conditions
Sharness / sharness-test (push) Waiting to run
Spell Check / spellcheck (push) Waiting to run
* feat: provide stats

* added N/A

* format

* workers stats alignment

* ipfs provide stat --all --compact

* consolidating compact stat

* update column alignment

* flags combinations errors

* command description

* change schedule AvgPrefixLen to float

* changelog

* alignments

* provide stat description draft

* rephrased provide-stats.md

* linking provide-stats.md from command description

* documentation test

* fix: refactor provide stat command type handling

- add extractSweepingProvider() helper to reduce nested type switching
- extract lowWorkerThreshold constant for worker availability check
- fix --lan error handling to work with buffered providers

* docs: add clarifying comments

* fix(commands): improve provide stat compact mode

- prevent panic when both columns are empty
- fix column alignment with UTF-8 characters
- only track col0MaxWidth for first column (as intended)

* test: add tests for ipfs provide stat command

- test basic functionality, flags, JSON output
- test legacy provider behavior
- test integration with content scheduling
- test disabled provider configurations
- add parseSweepStats helper with t.Helper()

* docs: improve provide command help text

- update tagline to "Control and monitor content providing"
- simplify help descriptions
- make error messages more consistent
- update tests to match new error messages

* metrics rename

```
Next reprovide at:
Next prefix:
```
updated to:
```
Next region prefix:
Next region reprovide:
```

* docs: improve Provide system documentation clarity

Enhance documentation for the Provide system to better explain how provider
records work and the differences between sweep and legacy modes.

Changes to docs/config.md:
- Provide section: add clear explanation of provider records and their role
- Provide.DHT: add provider record lifecycle and two provider systems overview
- Provide.DHT.Interval: explain relationship to expiration, contrast sweep vs legacy behavior
- Provide.DHT.SweepEnabled: rewrite to explain legacy problem, sweep solution, and efficiency gains
- Monitoring section: prioritize command-line tools (ipfs provide stat) before Prometheus

Changes to core/commands/provide.go:
- ipfs provide stat help: add explanation of provider records, TTL expiration, and how sweep batching works

Changes to docs/changelogs/v0.39.md:
- Add context about why stats matter for monitoring provider health
- Emphasize real-time monitoring workflow with watch command
- Explain what users can observe (rates, queues, worker availability)

* depend on latest kad-dht master

* docs: nits

---------

Co-authored-by: Marcin Rataj <lidel@lidel.org>
2025-10-23 20:29:36 +02:00
Andrew Gillis
f9dc739933
Upgrade to boxo master (#11026)
Some checks failed
CodeQL / codeql (push) Has been cancelled
Docker Check / lint (push) Has been cancelled
Docker Check / build (push) Has been cancelled
Gateway Conformance / gateway-conformance (push) Has been cancelled
Gateway Conformance / gateway-conformance-libp2p-experiment (push) Has been cancelled
Go Build / go-build (push) Has been cancelled
Go Check / go-check (push) Has been cancelled
Go Lint / go-lint (push) Has been cancelled
Go Test / go-test (push) Has been cancelled
Interop / interop-prep (push) Has been cancelled
Sharness / sharness-test (push) Has been cancelled
Spell Check / spellcheck (push) Has been cancelled
Interop / helia-interop (push) Has been cancelled
Interop / ipfs-webui (push) Has been cancelled
Upgrade to latest go-dsqueue and go-ds-pebble
2025-10-16 17:06:01 -07:00