* feat(ci): docker linting
adds hadolint to validate dockerfile best practices
configures project-specific rules in .hadolint.yaml
* fix(ci): enable hadolint console output
adds verbose and tty format to see linting results in CI logs
* test: trigger hadolint warning
remove --no-install-recommends to test CI output
* fix(ci): fail hadolint on warnings
stricter linting to catch all best practice violations
* fix: add --no-install-recommends to apt-get
reduces image size by avoiding unnecessary packages
* refactor: use WORKDIR instead of cd in dockerfile
replaces cd commands with WORKDIR for cleaner dockerfile
removes unnecessary hadolint ignore rules DL3003 and DL3009
* chore: simplify hadolint config
removes unnecessary override rules for cleaner config
(cherry picked from commit 82fef0c045)
* Tests: disable telemetry in tests by default
Disable the plugin in cli tests and sharness by default. Enable only in
telemetry tests.
There are cases when tests get stuck or get killed and leave daemons hanging around. We don't want to be getting telemetry from those.
* sharness: attempt to fix
* sharness: add missing --bool flag
* fix(ci): add omitempty to Plugin.Config field
The sharness problem is that when the telemetry plugin is configured
initially with 'ipfs config --bool', it creates a structure without
the 'Config: null' field, but when the config is copied and replaced,
it expects the structure to be preserved.
Adding omitempty ensures the Config field is omitted from JSON when
nil, making the config structure consistent between initial creation
and replacement operations.
---------
Co-authored-by: Marcin Rataj <lidel@lidel.org>
(cherry picked from commit 15f723a15e)
* fix: harness tests random panic
Connecting nodes in parallel can cause TLS handshake failures. For each node, connect to the other nodes serially. It is not necessary to connect in parallel as it does not save any significant time.
Closes#10932
(cherry picked from commit ae068a8061)
* Reprovider strategy: rename "flat" to "all".
Value "flat" now parses to "all". Behaviour from "all" removed.
Fixes#10864 which has detailed explanation.
* core/node/provider.go: remove unused function mfsRootProvider
It was used in the "all" strategy.
* docs: improve reprovider.strategy=all changelog framing
- highlight memory efficiency improvements
- clarify this removes v0.28 workaround
- update config.md memory requirements
- fix announce-on profile typo
* feat: deprecate Reprovider.Strategy=flat
- add deprecation warning in daemon.go when flat strategy is detected
- document that flat is deprecated in ParseReproviderStrategy comment
- add explicit test case for flat -> all mapping
- flat continues to work but users are warned to migrate to all
---------
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* fix(cmds): improve ipfs add --help clarity and organization
- clarify --raw-leaves behavior with CIDv0/v1 defaults
- add Import.* config references to all relevant options
- replace deprecated 'ipfs object links' with 'ipfs ls' in examples
- add section headers for better navigation (BASIC EXAMPLES, MFS INTEGRATION, etc.)
- group related options logically (output control, CID & hashing, experimental features)
- standardize experimental warnings format
- add MerkleDAG docs link and clarify chunking behavior
- fix incorrect Import.UnixFSHAMTThreshold reference
addresses confusion from #10918 where --raw-leaves appeared to have no effect
because CIDv1 automatically enables it by default
* docs: fix typos in Import.* configuration documentation
- fix 'chilcren' → 'children'
- fix 'HAMT directory have' → 'HAMT directories have'
- fix 'A HAMT is an structure' → 'A HAMT is a structure'
* Update core/commands/add.go
Co-authored-by: Daniel Norman <1992255+2color@users.noreply.github.com>
---------
Co-authored-by: Daniel Norman <1992255+2color@users.noreply.github.com>
* chore: disable AutoTLS in TCP-only transport tests
Tests were failing intermittently. Disabling AutoTLS when WebSocket
transport is disabled appears to resolve the issue.
* fix: make NewRandPort thread-safe
Track allocated ports globally to prevent conflicts
when tests run in parallel.
test was expecting immediate GC lock acquisition after pipe close,
but timing wasn't guaranteed. replaced blocking wait with 5-second
timeout to handle timing variations while still detecting deadlocks.
* Initial pass at Telemetry plugin
Currently, IP Shipyard, with the help of Probelab, monitor and extract
Amino/IPFS public network metrics with the use of DHT crawlers and
bootstrappers (via peerlog plugin). For example, we log all peer IDs seen and
their AgentVersion/Addresses obtained from the `identify` protocol, which
provides insights into protocol usage, total number of peers etc.
We would like to increase the ability to obtain more insights from the network
by collecting some more information in the future, but also to give users more
control over this collection (i.e. opt-out). The information collected will
not allow unique identification of anyone and is only used for aggregation.
Now, this PR explores a way of moving in this direction:
* A new "telemetry" fx plugin is in charge of dealing with telemetry
* The FX plugin allows to plug and make decisions / take actions during the setup phase:
* We can inspect whether we are using Private Networks before the libp2p.Host has been initialized.
* We can send telemetry after the libp2p Host is initialized.
* Everything is self-contained. Custom builds can remove the plugin altogether without needing to surgically edit the code.
As for behaviour:
* The user can opt-in/out via EnvVar, file in the repo path or plugin configuration.
* Users on private networks or with custom bootstrappers are detected, offered a wall of text explaining why we need telemetry and invited to opt-in. Opt-out happens otherwise on a timeout (with no input). Their preferences are stored.
* Users on standard settings are opted-in by default. This is the status quo in Kubo already, except they don't get a chance to opt out.
The telemetry libp2p protocol is yet to be defined, but expect something similar to identify, with a protobuf being pushed to bootstrappers or to a specific telemetry node that we define. In the case of pnets, this will be done with a temporary peer.
* checkpoint
* telemetry plugin: second pass
* On first run it generates a UUID and shows a message to the user.
* UUID is persistend to "telemetry_uuid"
* Sends telemetry 1 minute after boot and every 24h
* LogEvent is the thing containing all the telemetry that is sent
* Opt-out possible via env-var or plugin configuration
* Telemetry: add changelog and environment variable documentation
* docs: improved daemon message
making it more obvious nothing was sent yet
and that user had 15m to out-out
plus some debug logs that confirm opt-out
* refactor: rename IPFS_TELEMETRY_MODE to IPFS_TELEMETRY
* fix: add User-Agent header to telemetry requests
---------
Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com>
Co-authored-by: Marcin Rataj <lidel@lidel.org>
The test was flaky because `ipfs repo verify` doesn't detect corruption
of the empty directory block (served from memory even when corrupted on
disk). Exclude both empty file and empty directory blocks from random
corruption to make the test reliable.
---------
Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com>
* feat(gateway): concurrency and timeout limits
Depends on https://github.com/ipfs/boxo/pull/994
* chore: boxo master with final boxo#994
this includes race-condition fixes from ipfs/boxo#994
and increased `DefaultMaxConcurrentRequests = 4096`
* docs: concise config.md and changelog
Dependabot couldn't find the original pull request head commit, 0292612bd5e16ed34f2974225fe170b15d4eb821.
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* fix(ci): pass -buildvcs=false to go list in Makefile
Docker builds were failing because go list commands in mk/golang.mk
didn't inherit GOFLAGS from the environment, causing empty package
names when .git directory was incomplete.
This issue always existed but was masked until recent changes in
GitHub Actions runners or Docker golang image exposed it.
* fix(ci): use modern ENV key=value format in Dockerfile
Fixes Docker build deprecation warnings.
* feat(ipns): Add a parameter in name.publish to change the sequence number
* test: monotonic name publish --sequence
* docs: `name publish --sequence`
* chore: boxo main with PR 962
---------
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* fix(relay): feed connected peers to AutoRelay discovery
Feed all connected swarm peers to AutoRelay as potential relay
candidates. This allows peers from HTTP routing and manual connections
to serve as relays, not just DHT-discovered peers.
Fixes#10899
* docs: changelog
this is rare, but if someone runs a lot of tests and had some unrelated
FUSE failures, sharness can get stuck on lsof step.
this ensures we never get stuck + there is more visibility so people
who care about FUSE have better time debugging
* feat: update log level command to show log levels
* test: add log level tests
* update TestCommands test
* docs: relation to GOLOG_LOG_LEVEL
* chore: update to latest go-log
* fix: do not output single subsystem name in CLI
* test: explicit subsystem request dont output subsystem
* LevelFromString renamed to Parse
* Modify `ipfs log level`
* Denote default level with sdubsystem name '(defult)'.
* make "*" an dalias for "all". Test to make sure both work the same.
Test was failing becuase path passed to `path.NewPath` (from boxo) did not include a required namespace. Prepending the namespace to the path with "/ipfs/" fixes this.
* Fix command stream result handling
Do not close the stream prematurely in a deferred function. Also, avoid possible shadowing errors.
Closes#9007
* test(cli): test/cli/cid_test.go
includes regression tests for https://github.com/ipfs/kubo/issues/9007
and better coverage than the old test/sharness/t0290-cid.sh
(identified bug in basemoji)
* fix: base256emoji in cid bases --prefix
changed the character display logic from ASCII range check to
unicode.IsPrint() to properly handle Unicode characters including
emojis.
* docs: changelog
---------
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* fix: `ipfs cid format` without repo
these commands should work without daemon or repo
but were missing SetDoesNotUseRepo(true)
* test: test/cli/commands_without_repo_test.go
* Provide according to strategy
Updates boxo to a version with the changes from https://github.com/ipfs/boxo/pull/976, which decentralize the providing responsibilities (from a central providing.Exchange to blockstore, pinner, mfs).
The changes consist in initializing the Pinner, MFS and the blockstore with the provider.System, which is created first.
Since the provider.System is created first, the reproviding KeyChanFunc is set
later when we can create it once we have the Pinner, MFS and the blockstore.
Some additional work applies to the Add() workflow. Normally, blocks would get provided at the Blockstore or the Pinner, but when adding blocks AND a "pinned" strategy is used, the blockstore does not provide, and the
pinner does not traverse the DAG (and thus doesn't provide either), so we need to provide directly from the Adder. This is resolved by wrapping the DAGService in a "providingDAGService" which provides every added block, when using the "pinned" strategy.
`ipfs --offline add` when the ONLINE daemon is running will now announce blocks per the chosen strategy, where before it did not announce them. This is documented in the changelog. A couple of releases ago, adding with `ipfs --offline add` was faster, but this is no longer the case so we are not incurring in any penalties by sticking to the fact that the daemon is online and has a providing strategy that we follow.
Co-authored-by: gammazero <11790789+gammazero@users.noreply.github.com>
Co-authored-by: Marcin Rataj <lidel@lidel.org>
- Move `ipfs stat reprovide` to `ipfs provide stat`
- Mark `ipfs stat provide` as deprecated and replaces by `ipfs provide stat`
- Mark `ipfs stat reprovide` as deprecated and replaces by `ipfs provide stat`
- Remove redundant code from deprecated subcommands
Closes#10869
* feat(add): add support for naming pinned CID
Signed-off-by: kapil <kapilsareen584@gmail.com>
* fix(add): no double pinning and simplify pin-name
- modify PinRoot to accept name parameter, eliminating double pinning
- remove automatic filename fallback logic for cleaner behavior
- only create named pins when explicitly requested via --pin-name=value
- replace NoPinName constant with idiomatic empty string literals
- Update help text and tests to reflect explicit-only behavior
* docs: changelog
* chore: lint
* test: negative case for empty pin-name
* chore: gofmt
---------
Signed-off-by: kapil <kapilsareen584@gmail.com>
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* refactor: remove goprocess
The `goprocess` package is no longer needed. It can be replaces by modern `context` and `context.AfterFunc`.
* mod tidy
* log unmount errors on shutdown
* Do not log non-mounted errors on shutdown
* Use WaitGroup associated with IPFS node to wait for services to whutdown
* Prefer explicit Close to context.ArterFunc
* Do not use node-level WaitGroup
* Unmount for non-supported platforms
* fix return values
* test: daemon shuts down gracefully
make sure ongoing operations dont block shutdown
* test(cli): add TestFUSE
* test: smarter RequiresFUSE
opportunistically run FUSE tests if env has fusermount
and TEST_FUSE was not explicitly set
* docs: changelog
---------
Co-authored-by: gammazero <gammazero@users.noreply.github.com>
Co-authored-by: Marcin Rataj <lidel@lidel.org>
No behaviour changes.
Currently we are using ProvideManyRouter for Bitswap, which is only meant to
use ContentDiscovery. This makes things more clear in that there is a
designated ContentDiscovery instance.
After boxo v0.33.1, this is a recommended step to fix http retrieval bugs.
Having a single ConnectEventManager prevents misdirected operations in the
network.Router to change the Connectedness state in a way that the counterpart
(httpnet or bsnet) can later correct.
