Commit Graph

15598 Commits

Author SHA1 Message Date
guillaumemichel
0dbf157919
restore dual dht provider 2025-08-23 00:02:18 +02:00
Guillaume Michel
aaba9b4338
Merge branch 'master' into reprovide-sweep 2025-08-22 23:00:22 +02:00
Andrew Gillis
ae068a8061
fix: harness tests random panic (#10933)
Some checks failed
CodeQL / codeql (push) Has been cancelled
Docker Build / docker-build (push) Has been cancelled
Gateway Conformance / gateway-conformance (push) Has been cancelled
Gateway Conformance / gateway-conformance-libp2p-experiment (push) Has been cancelled
Go Build / go-build (push) Has been cancelled
Go Check / go-check (push) Has been cancelled
Go Lint / go-lint (push) Has been cancelled
Go Test / go-test (push) Has been cancelled
Interop / interop-prep (push) Has been cancelled
Sharness / sharness-test (push) Has been cancelled
Spell Check / spellcheck (push) Has been cancelled
Interop / helia-interop (push) Has been cancelled
Interop / ipfs-webui (push) Has been cancelled
* fix: harness tests random panic

Connecting nodes in parallel can cause TLS handshake failures. For each node, connect to the other nodes serially. It is not necessary to connect in parallel as it does not save any significant time.

Closes #10932
2025-08-22 13:57:35 -07:00
guillaumemichel
812f7fcf6d
docs 2025-08-22 22:40:26 +02:00
guillaumemichel
79038738aa
don't provide LAN DHT 2025-08-22 13:34:36 +02:00
Guillaume Michel
814085f76c
Merge branch 'master' into reprovide-sweep 2025-08-22 12:00:34 +02:00
Marcin Rataj
dd3f59db5a
chore: 0.38.0-dev
Some checks are pending
CodeQL / codeql (push) Waiting to run
Docker Build / docker-build (push) Waiting to run
Gateway Conformance / gateway-conformance (push) Waiting to run
Gateway Conformance / gateway-conformance-libp2p-experiment (push) Waiting to run
Go Build / go-build (push) Waiting to run
Go Check / go-check (push) Waiting to run
Go Lint / go-lint (push) Waiting to run
Go Test / go-test (push) Waiting to run
Interop / interop-prep (push) Waiting to run
Interop / helia-interop (push) Blocked by required conditions
Interop / ipfs-webui (push) Blocked by required conditions
Sharness / sharness-test (push) Waiting to run
Spell Check / spellcheck (push) Waiting to run
2025-08-21 19:51:25 +02:00
Hector Sanjuan
846e219ead
feat: Reprovider.Strategy: rename "flat" to "all" (#10928)
* Reprovider strategy: rename "flat" to "all".

Value "flat" now parses to "all". Behaviour from "all" removed.

Fixes #10864 which has detailed explanation.

* core/node/provider.go: remove unused function mfsRootProvider

It was used in the "all" strategy.

* docs: improve reprovider.strategy=all changelog framing

- highlight memory efficiency improvements
- clarify this removes v0.28 workaround
- update config.md memory requirements
- fix announce-on profile typo

* feat: deprecate Reprovider.Strategy=flat

- add deprecation warning in daemon.go when flat strategy is detected
- document that flat is deprecated in ParseReproviderStrategy comment
- add explicit test case for flat -> all mapping
- flat continues to work but users are warned to migrate to all

---------

Co-authored-by: Marcin Rataj <lidel@lidel.org>
2025-08-21 19:02:29 +02:00
guillaumemichel
dc2bb67d22
extended tests 2025-08-21 16:30:45 +02:00
Marcin Rataj
426477ef78
docs: improve ipfs add --help (#10926)
Some checks are pending
CodeQL / codeql (push) Waiting to run
Docker Build / docker-build (push) Waiting to run
Gateway Conformance / gateway-conformance (push) Waiting to run
Gateway Conformance / gateway-conformance-libp2p-experiment (push) Waiting to run
Go Build / go-build (push) Waiting to run
Go Check / go-check (push) Waiting to run
Go Lint / go-lint (push) Waiting to run
Go Test / go-test (push) Waiting to run
Interop / interop-prep (push) Waiting to run
Interop / helia-interop (push) Blocked by required conditions
Interop / ipfs-webui (push) Blocked by required conditions
Sharness / sharness-test (push) Waiting to run
Spell Check / spellcheck (push) Waiting to run
* fix(cmds): improve ipfs add --help clarity and organization

- clarify --raw-leaves behavior with CIDv0/v1 defaults
- add Import.* config references to all relevant options
- replace deprecated 'ipfs object links' with 'ipfs ls' in examples
- add section headers for better navigation (BASIC EXAMPLES, MFS INTEGRATION, etc.)
- group related options logically (output control, CID & hashing, experimental features)
- standardize experimental warnings format
- add MerkleDAG docs link and clarify chunking behavior
- fix incorrect Import.UnixFSHAMTThreshold reference

addresses confusion from #10918 where --raw-leaves appeared to have no effect
because CIDv1 automatically enables it by default

* docs: fix typos in Import.* configuration documentation

- fix 'chilcren' → 'children'
- fix 'HAMT directory have' → 'HAMT directories have'
- fix 'A HAMT is an structure' → 'A HAMT is a structure'

* Update core/commands/add.go

Co-authored-by: Daniel Norman <1992255+2color@users.noreply.github.com>

---------

Co-authored-by: Daniel Norman <1992255+2color@users.noreply.github.com>
2025-08-21 14:57:30 +02:00
Marcin Rataj
c12d24949c
feat: optimize docker builds (#10925)
* feat(docker): optimize Dockerfile for faster builds

- add BuildKit syntax directive for advanced caching features
- implement cache mounts for Go modules and build cache
- reduce layers by combining RUN commands (5→2 in final stage)
- optimize apt-get with --no-install-recommends flag
- use COPY --chmod to avoid separate permission fixing

Performance improvements:
- incremental builds after code changes: ~8.6x faster (1m51s → 13s)
- go module/build cache persists between builds
- reduced layer count improves cache efficiency

* ci: optimize Docker builds with BuildKit caching

- enable BuildKit with GitHub Actions cache backend
- add Docker Hub registry cache for cross-workflow sharing
- move Docker login earlier to enable registry cache writes
- use dual cache strategy (gha + registry) for faster builds

expected improvements:
- PR builds can reuse main branch cache from Docker Hub
- rebuild after code changes ~5-10x faster with persistent cache
- cross-PR cache sharing reduces redundant builds
2025-08-21 14:44:19 +02:00
guillaumemichel
c9fbe62069
fix merge issues 2025-08-20 15:15:08 +02:00
guillaumemichel
07189277e2
fx cyclic deps 2025-08-20 14:08:31 +02:00
guillaumemichel
a44d3f57e5
fix fx 2025-08-20 10:42:57 +02:00
guillaumemichel
55df2329e4
Merge branch 'master' into reprovide-sweep 2025-08-20 10:26:25 +02:00
guillaumemichel
1a389ffc5d
use latest boxo release 2025-08-20 10:18:12 +02:00
guillaumemichel
c571318428
using latest kad-dht commit 2025-08-20 10:07:59 +02:00
Marcin Rataj
ccb49de852
feat(config): AutoConf with "auto" placeholders (#10883)
Some checks failed
CodeQL / codeql (push) Has been cancelled
Docker Build / docker-build (push) Has been cancelled
Gateway Conformance / gateway-conformance (push) Has been cancelled
Gateway Conformance / gateway-conformance-libp2p-experiment (push) Has been cancelled
Go Build / go-build (push) Has been cancelled
Go Check / go-check (push) Has been cancelled
Go Lint / go-lint (push) Has been cancelled
Go Test / go-test (push) Has been cancelled
Interop / interop-prep (push) Has been cancelled
Sharness / sharness-test (push) Has been cancelled
Spell Check / spellcheck (push) Has been cancelled
Interop / helia-interop (push) Has been cancelled
Interop / ipfs-webui (push) Has been cancelled
https://github.com/ipfs/kubo/pull/10883
https://github.com/ipshipyard/config.ipfs-mainnet.org/issues/3

---------

Co-authored-by: gammazero <gammazero@users.noreply.github.com>
2025-08-20 05:59:11 +02:00
Andrew Gillis
c468f44fc1
upgrade to go-test v0.2.3 (#10923)
Some checks are pending
CodeQL / codeql (push) Waiting to run
Docker Build / docker-build (push) Waiting to run
Gateway Conformance / gateway-conformance (push) Waiting to run
Gateway Conformance / gateway-conformance-libp2p-experiment (push) Waiting to run
Go Build / go-build (push) Waiting to run
Go Check / go-check (push) Waiting to run
Go Lint / go-lint (push) Waiting to run
Go Test / go-test (push) Waiting to run
Interop / interop-prep (push) Waiting to run
Interop / helia-interop (push) Blocked by required conditions
Interop / ipfs-webui (push) Blocked by required conditions
Sharness / sharness-test (push) Waiting to run
Spell Check / spellcheck (push) Waiting to run
2025-08-19 11:13:03 -07:00
Marcin Rataj
4bafb22b76
fix(ci): make NewRandPort thread-safe (#10921)
Some checks failed
Docker Build / docker-build (push) Waiting to run
Gateway Conformance / gateway-conformance (push) Waiting to run
Gateway Conformance / gateway-conformance-libp2p-experiment (push) Waiting to run
Go Build / go-build (push) Waiting to run
Go Check / go-check (push) Waiting to run
Go Lint / go-lint (push) Waiting to run
Go Test / go-test (push) Waiting to run
Interop / interop-prep (push) Waiting to run
Interop / helia-interop (push) Blocked by required conditions
Interop / ipfs-webui (push) Blocked by required conditions
Sharness / sharness-test (push) Waiting to run
Spell Check / spellcheck (push) Waiting to run
CodeQL / codeql (push) Has been cancelled
* chore: disable AutoTLS in TCP-only transport tests

Tests were failing intermittently. Disabling AutoTLS when WebSocket
transport is disabled appears to resolve the issue.

* fix: make NewRandPort thread-safe

Track allocated ports globally to prevent conflicts
when tests run in parallel.
2025-08-18 22:12:22 +02:00
Marcin Rataj
a81cc29282
fix: resolve TestAddMultipleGCLive race condition (#10916)
test was expecting immediate GC lock acquisition after pipe close,
but timing wasn't guaranteed. replaced blocking wait with 5-second
timeout to handle timing variations while still detecting deadlocks.
2025-08-18 20:49:26 +02:00
Hector Sanjuan
c5b13b14e8
docs: fix broadcast control release docs (#10919)
s/want/have blocks
2025-08-18 20:48:53 +02:00
Hector Sanjuan
4255cc3889
feat: telemetry plugin (#10866)
* Initial pass at Telemetry plugin

Currently, IP Shipyard, with the help of Probelab, monitor and extract
Amino/IPFS public network metrics with the use of DHT crawlers and
bootstrappers (via peerlog plugin). For example, we log all peer IDs seen and
their AgentVersion/Addresses obtained from the `identify` protocol, which
provides insights into protocol usage, total number of peers etc.

We would like to increase the ability to obtain more insights from the network
by collecting some more information in the future, but also to give users more
control over this collection (i.e. opt-out). The information collected will
not allow unique identification of anyone and is only used for aggregation.

Now, this PR explores a way of moving in this direction:

* A new "telemetry" fx plugin is in charge of dealing with telemetry
* The FX plugin allows to plug and make decisions / take actions during the setup phase:
  * We can inspect whether we are using Private Networks before the libp2p.Host has been initialized.
  * We can send telemetry after the libp2p Host is initialized.
  * Everything is self-contained. Custom builds can remove the plugin altogether without needing to surgically edit the code.

As for behaviour:

* The user can opt-in/out via EnvVar, file in the repo path or plugin configuration.
* Users on private networks or with custom bootstrappers are detected, offered a wall of text explaining why we need telemetry and invited to opt-in. Opt-out happens otherwise on a timeout (with no input). Their preferences are stored.
* Users on standard settings are opted-in by default. This is the status quo in Kubo already, except they don't get a chance to opt out.

The telemetry libp2p protocol is yet to be defined, but expect something similar to identify, with a protobuf being pushed to bootstrappers or to a specific telemetry node that we define. In the case of pnets, this will be done with a temporary peer.

* checkpoint

* telemetry plugin: second pass

* On first run it generates a UUID and shows a message to the user.
* UUID is persistend to "telemetry_uuid"
* Sends telemetry 1 minute after boot and every 24h
* LogEvent is the thing containing all the telemetry that is sent
* Opt-out possible via env-var or plugin configuration

* Telemetry: add changelog and environment variable documentation

* docs: improved daemon message

making it more obvious nothing was sent yet
and that user had 15m to out-out

plus some debug logs that confirm opt-out

* refactor: rename IPFS_TELEMETRY_MODE to IPFS_TELEMETRY

* fix: add User-Agent header to telemetry requests

---------

Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com>
Co-authored-by: Marcin Rataj <lidel@lidel.org>
2025-08-18 20:46:05 +02:00
Andrew Gillis
187fce7a6d
fix typos in docs and comments (#10920) 2025-08-18 08:43:25 -07:00
Andrew Gillis
7109534464
Upgrade to Boxo v0.34.0 (#10917)
Some checks failed
CodeQL / codeql (push) Has been cancelled
Docker Build / docker-build (push) Has been cancelled
Gateway Conformance / gateway-conformance (push) Has been cancelled
Gateway Conformance / gateway-conformance-libp2p-experiment (push) Has been cancelled
Go Build / go-build (push) Has been cancelled
Go Check / go-check (push) Has been cancelled
Go Lint / go-lint (push) Has been cancelled
Go Test / go-test (push) Has been cancelled
Interop / interop-prep (push) Has been cancelled
Sharness / sharness-test (push) Has been cancelled
Spell Check / spellcheck (push) Has been cancelled
Interop / helia-interop (push) Has been cancelled
Interop / ipfs-webui (push) Has been cancelled
* Use ValidatingBlockstore when HashOnRead is configured
* fix ValidatingBlockstore creation
* upgrade to boxo v0.34.0
* Use tagged boxo release
2025-08-15 14:43:26 -07:00
Marcin Rataj
1ca5c9dd84
test: fix flaky repo verify (#10743)
Some checks are pending
CodeQL / codeql (push) Waiting to run
Docker Build / docker-build (push) Waiting to run
Gateway Conformance / gateway-conformance (push) Waiting to run
Gateway Conformance / gateway-conformance-libp2p-experiment (push) Waiting to run
Go Build / go-build (push) Waiting to run
Go Check / go-check (push) Waiting to run
Go Lint / go-lint (push) Waiting to run
Go Test / go-test (push) Waiting to run
Interop / interop-prep (push) Waiting to run
Interop / helia-interop (push) Blocked by required conditions
Interop / ipfs-webui (push) Blocked by required conditions
Sharness / sharness-test (push) Waiting to run
Spell Check / spellcheck (push) Waiting to run
The test was flaky because `ipfs repo verify` doesn't detect corruption
of the empty directory block (served from memory even when corrupted on
disk). Exclude both empty file and empty directory blocks from random
corruption to make the test reliable.

---------

Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com>
2025-08-15 05:21:56 +02:00
Marcin Rataj
64c47c374a
feat(config): Gateway.RetrievalTimeout|MaxConcurrentRequests (#10905)
* feat(gateway): concurrency and timeout limits

Depends on https://github.com/ipfs/boxo/pull/994

* chore: boxo master with final boxo#994

this includes race-condition fixes from ipfs/boxo#994
and increased `DefaultMaxConcurrentRequests = 4096`

* docs: concise config.md and changelog
2025-08-15 02:36:33 +02:00
dependabot[bot]
5cb0943936
chore(deps): bump actions/checkout from 4 to 5 (#10907)
Dependabot couldn't find the original pull request head commit, 0292612bd5e16ed34f2974225fe170b15d4eb821.

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-08-15 00:32:05 +02:00
Andrew Gillis
b8a1aa471c
chore: replace random test utils with equivalents in go-test/random (#10915)
Some checks are pending
CodeQL / codeql (push) Waiting to run
Docker Build / docker-build (push) Waiting to run
Gateway Conformance / gateway-conformance (push) Waiting to run
Gateway Conformance / gateway-conformance-libp2p-experiment (push) Waiting to run
Go Build / go-build (push) Waiting to run
Go Check / go-check (push) Waiting to run
Go Lint / go-lint (push) Waiting to run
Go Test / go-test (push) Waiting to run
Interop / interop-prep (push) Waiting to run
Interop / helia-interop (push) Blocked by required conditions
Interop / ipfs-webui (push) Blocked by required conditions
Sharness / sharness-test (push) Waiting to run
Spell Check / spellcheck (push) Waiting to run
Replace test functionality that is dublicated in go-test/random
2025-08-14 10:37:55 -07:00
Andrew Gillis
501b1f8bc3
feat: require go1.25 for building kubo (#10913)
Some checks are pending
CodeQL / codeql (push) Waiting to run
Docker Build / docker-build (push) Waiting to run
Gateway Conformance / gateway-conformance (push) Waiting to run
Gateway Conformance / gateway-conformance-libp2p-experiment (push) Waiting to run
Go Build / go-build (push) Waiting to run
Go Check / go-check (push) Waiting to run
Go Lint / go-lint (push) Waiting to run
Go Test / go-test (push) Waiting to run
Interop / interop-prep (push) Waiting to run
Interop / helia-interop (push) Blocked by required conditions
Interop / ipfs-webui (push) Blocked by required conditions
Sharness / sharness-test (push) Waiting to run
Spell Check / spellcheck (push) Waiting to run
* Require go1.25 for building kubo
* update Dockerfile to fix warnings
2025-08-13 18:01:50 -07:00
Piotr Galar
2b858bb3e6
feat(ci): reusable spellcheck from unified CI (#10873)
* ci: use spellcheck from unified CI
* chore: fix spelling

---------

Co-authored-by: Marcin Rataj <lidel@lidel.org>
2025-08-14 00:35:35 +00:00
Marcin Rataj
8e5ef5503d
fix(ci): docker build (#10914)
* fix(ci): pass -buildvcs=false to go list in Makefile

Docker builds were failing because go list commands in mk/golang.mk
didn't inherit GOFLAGS from the environment, causing empty package
names when .git directory was incomplete.

This issue always existed but was masked until recent changes in
GitHub Actions runners or Docker golang image exposed it.

* fix(ci): use modern ENV key=value format in Dockerfile

Fixes Docker build deprecation warnings.
2025-08-14 01:49:35 +02:00
dependabot[bot]
abe6d9da13
chore(deps): bump actions/download-artifact from 4 to 5 (#10908)
Some checks are pending
CodeQL / codeql (push) Waiting to run
Docker Build / docker-build (push) Waiting to run
Gateway Conformance / gateway-conformance (push) Waiting to run
Gateway Conformance / gateway-conformance-libp2p-experiment (push) Waiting to run
Go Build / go-build (push) Waiting to run
Go Check / go-check (push) Waiting to run
Go Lint / go-lint (push) Waiting to run
Go Test / go-test (push) Waiting to run
Interop / interop-prep (push) Waiting to run
Interop / helia-interop (push) Blocked by required conditions
Interop / ipfs-webui (push) Blocked by required conditions
Sharness / sharness-test (push) Waiting to run
Spell Check / spellcheck (push) Waiting to run
Bumps [actions/download-artifact](https://github.com/actions/download-artifact) from 4 to 5.
- [Release notes](https://github.com/actions/download-artifact/releases)
- [Commits](https://github.com/actions/download-artifact/compare/v4...v5)

---
updated-dependencies:
- dependency-name: actions/download-artifact
  dependency-version: '5'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com>
2025-08-13 02:24:18 -07:00
Andrew Gillis
a3b01cd05f
Replace uber-go/multierr with errors.Join (#10912) 2025-08-13 02:03:42 -07:00
Sergey Gorbunov
d81f524cce
feat(ipns): support passing custom sequence number during publishing (#10851)
Some checks are pending
CodeQL / codeql (push) Waiting to run
Docker Build / docker-build (push) Waiting to run
Gateway Conformance / gateway-conformance (push) Waiting to run
Gateway Conformance / gateway-conformance-libp2p-experiment (push) Waiting to run
Go Build / go-build (push) Waiting to run
Go Check / go-check (push) Waiting to run
Go Lint / go-lint (push) Waiting to run
Go Test / go-test (push) Waiting to run
Interop / interop-prep (push) Waiting to run
Interop / helia-interop (push) Blocked by required conditions
Interop / ipfs-webui (push) Blocked by required conditions
Sharness / sharness-test (push) Waiting to run
Spell Check / spellcheck (push) Waiting to run
* feat(ipns): Add a parameter in name.publish to change the sequence number
* test: monotonic name publish --sequence
* docs: `name publish --sequence`
* chore: boxo main with PR 962

---------

Co-authored-by: Marcin Rataj <lidel@lidel.org>
2025-08-13 04:15:45 +02:00
Guillaume Michel
7250eb8786
bump go-libp2p-kad-dht (#10911) 2025-08-12 17:09:56 -07:00
guillaumemichel
9764140cc9
boxo PR fix
Some checks failed
Spell Check / spellcheck (push) Has been cancelled
2025-08-12 15:53:02 +02:00
guillaumemichel
6788e6e351
updated boxo provide interface 2025-08-12 15:31:57 +02:00
guillaumemichel
b650b9f4f4
make linter happy 2025-08-12 14:36:36 +02:00
guillaumemichel
49f74e8cd5
use latest kad-dht commit 2025-08-12 14:10:38 +02:00
gammazero
5ad8e3a944 Add Clear to Provider interface 2025-08-11 22:54:52 -10:00
gammazero
9531315909 mod_tidy 2025-08-11 22:18:00 -10:00
guillaumemichel
107a0bb25a moved Provider interface from kad-dht to kubo/core/node 2025-08-11 22:17:10 -10:00
guillaumemichel
688944ae33 move Provider interface to from kad-dht to node 2025-08-11 22:10:36 -10:00
guillaumemichel
d67173ae27 merged IpfsNode provider and reprovider 2025-08-11 21:58:36 -10:00
guillaumemichel
d140885902 config docs 2025-08-11 21:43:12 -10:00
guillaumemichel
82d6feeb49 config 2025-08-11 21:43:12 -10:00
guillaumemichel
bf5125a197 revert workflow debug 2025-08-11 21:43:12 -10:00
guillaumemichel
5a57c4f051 initialize MHStore 2025-08-11 21:43:12 -10:00
Marcin Rataj
48292612a0 test(sharness): PARALLEL=1 and timeout 30m
running sequentially to see where timeout occurs
2025-08-11 21:43:12 -10:00