* Configure bitswap braodcast reduction
Add new config items to `Internal.Bitswap` to allow configuration of bitswap broadcast reduction behavior. Broadcast reduction behavior is enabled by default, and uses settings that should be suitable for most installations of kubo.
* update sharness metrics test
* Explicit defaults for broadcast reduction configuration
* Update docs/config.md
* ipfs filestore fix [<obj>]... - Verify objects in filestore and remove bad links
* Option --remove-bad-blocks for 'ipfs filestore verify'
* fix call to DeleteBlock
* action of --remove-bad-blocks in output only when option specified
* Add sharness test for removing bad blocks
---------
Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com>
* chore: update to go-log/v2
go-log v2 has been out for quite a while now and it is time to deprecate v1.
Replace all use of go-log with go-log/v2
Makes /api/v0/log/tail useful over HTTP
Updates dependencies that have moved to go-lov/v2
Removes support for ContextWithLoggable as this is not needed for tracing-like functionality
- Replaces: PR #8765
- Closes issue #8753
- Closes issue #9245
- Closes issue #10809
Other fixes:
* update go-ipfs-cmds
* update http logs test
* fix test
* Read/send one line of log data at a time
* Update -log-level docs
* fix(config): explicit Provider.Enabled flag
Adds missing config option described in
https://github.com/ipfs/kubo/issues/10803
* refactor: remove Experimental.StrategicProviding
removing experiment, replaced with Provider.Enabled
* test(cli): routing [re]provide
updated and added tests for manually triggering provide and reprovide
and making them respect global configuration flag to avoid
inconsistent behaviors
* docs: improve DelegatedRouters
* refactor: default DefaultProviderWorkerCount=16
- simplified default for both
- 16 is safer for non-accelerated DHT client
- acceletated DHT performs better without limit anyway - updated docs
* repo: optionally wait to acquire repo lock
Retry acquiring repo lock for a specified amount of time. Retry once per second until time is expired ro lock is acquired.
Lock wait timeout is specified using the environment variable `IPFS_WAIT_REPO_LOCK` and assigning it a value that is parsable as a golang `time.Duration`. For example:
```
IPFS_WAIT_REPO_LOCK="5s"
```
Closes#10482
* document IPFS_WAIT_REPO_LOCK
---------
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* Add MFS command line options, extend existing mount functions for MFS, set defaults.
* Directory listing and file stat.
* Add a read-only MFS view.
* Add mkdir and interface checks.
* Add remove and rename functionality.
* Implement all required write interfaces.
* Adjust mount functions for other architechtures.
* Merge branch 'master' into feat/10710-mfs-fuse-mount
* Write a basic read/write test.
* Write more basic tests, add a mutex to the file object, fix modtime.
* Add a concurrency test, remove mutexes from file and directory structures.
* Refactor naming(mfdir -> mfsdir) and add documentation.
* Add CID retrieval through ipfs_cid xattr.
* Add docs, add xattr listing, fix bugs for mv and stat, refactor.
* Add MFS command line options, extend existing mount functions for MFS, set defaults.
* docs phrasing
* docs: Mounts.MFS
* docs: warn about lazy-loaded DAGs
* test: TEST_FUSE=1 ./t0030-mount.sh -v
---------
Co-authored-by: Guillaume Michel <guillaumemichel@users.noreply.github.com>
Co-authored-by: guillaumemichel <guillaume@michel.id>
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* Feat: http retrieval as experimental feature
This introduces the http-retrieval capability as an experimental feature.
It can be enabled in the configuration `Experimental.HTTPRetrieval.Enabled = true`.
Documentation and changelog to be added later.
* refactor: HTTPRetrieval.Enabled as Flag
* docs(config): HTTPRetrieval section
* refactor: reusable MockHTTPContentRouter
* feat: HTTPRetrieval.TLSInsecureSkipVerify
allows self-signed certificates in tests
* feat(config): HTTPRetrieval.MaxBlockSize
* test: end-to-end HTTPRetrieval.Enabled
this spawns two http services on localhost:
1. HTTP router that returns HTTP provider when /routing/v1/providers/cid i queried
2. HTTP provider that returns a block when /ipfs/cid is queried
3. Configures Kubo to use (1) instead of cid.contact
this seems to work (running test with DEBUG=true shows (1) was queried
for the test CID and returned multiaddr of (2), but Kubo never requested
test CID block from (2) – needs investigation
* fix: enable /routing/v1/peers for non-cid.contact
we artificially limited every delegated routing endpoint because of
cid.contact being limited to one endpoint
* feat: Routing.DelegatedRouters
make it easy to override the hardcoded implicit HTTP routeur URL
without having to set the entire custom Router.Routers and
Router.Methods
(http_retrieval_client_test.go still needs to be fixed in future commit)
* test: flag remaining work
* docs: review feedback
* refactor: providerQueryMgr with bitswapNetworks
this fixes two regressions:
(1) introduced in https://github.com/ipfs/kubo/issues/10717
where we only used bitswapLib2p query manager
(this is why E2E did not act on http provider)
(2) introduced in https://github.com/ipfs/kubo/pull/10765
where it was not possible to set binary peerID in IgnoreProviders
(we changed to []string)
* refactor: Bitswap.Libp2pEnabled
replaces Bitswap.Enabled with Bitswap.Libp2pEnabled
adds tests that confirm it is possible to disable libp2p bitswap fully
and only keep http in client mode
also, removes the need for passing empty blockstore in client-only mode
* docs: changelog
---------
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* feat: add Bitswap configuration and related tests
* fix: update Bitswap function to use 'provide' parameter for server enablement
* docs: update changelog for Bitswap functionality changes
* fix: update Bitswap server enablement logic and improve related tests
* fix: rename BitswapConfig to Bitswap and update references
* docs: config and changelog
* fix: `ipfs cat` panic when `Bitswap.Enabled=false`
Fixes panic described in:
https://github.com/ipfs/kubo/pull/10782#discussion_r2069116219
---------
Co-authored-by: gystemd <gystemd@gmail.com>
Co-authored-by: gammazero <11790789+gammazero@users.noreply.github.com>
Co-authored-by: Giulio Piva <giulio.piva@dedicated.world>
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* datastore: metrics optional and off by default
When ipfs is initialized, the datastore metrics wrapper is not configured by default as it previously was. To enable datastore metrics during initialization, specifying the appropriate `--profile` option.
To enable datastore metrics tracking wrapper, initialize with datastore profile name + "-measure" suffix. For example:
```
ipfs init --profile flatfs-measure
```
Closes#10767
* fix sharness tests for new datastore dafaults
* Add sharness test to check metrics added by flatfs-measure profile
* Document updated metrics in changelog
* update config doc with new profiles
* docs(changelog): separate section
* initialize non-measure pebbleds with FormatMajorVersion config
* docs: fix typos, add docs link
---------
Co-authored-by: gammazero <gammazero@users.noreply.github.com>
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* Upgrade to pebble v2.0.3
- Configure latest pebble database format at init
- Do not automatically ratchet database format if set in config
- Daemon messge about new available pebble format
- Document pebble config with formatMajorVersion
- Add warning to users running badger, nudging them to switch to flatfs or pebble
- docs: explain Pebble's `FormatMajorVersion`
- Use pebbleds instead of badgerds in t0060-daemon.sh
- Print badgerds warning message to stderr
* adjust ipfs stats provide
* update boxo dep
* bump boxo
* fixing tests
* docs/chore: mark stat reprovide as experimental
* docs: Provider.Strategy
explicitly document it is not used - without this legacy users will have
it in their config and be very confused
---------
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* use go-datastore without go-process
* update go-ds-xxx dependencies
* update go-libp2p-kad-dht
* bitswap api changes
* Do not use multiple multi-error packages, pick one
* update boxo
* update expected metrics
