* provider: clear reprovide queue when reprovide strategy changes
When the currently configured reprovide strategy does not match the previous strategy read from the datastore, then clear the reprovide queue and update the reprovide strategy that is stored in the datastore.
Depends on https://github.com/ipfs/boxo/pull/978Closes#10829
* Update docs/changelogs/v0.36.md
Co-authored-by: Guillaume Michel <guillaumemichel@users.noreply.github.com>
* update log message
* update boxo
* Move change log to v0.37.md
* Add `provide clear` command to clear provide queue
The `provide clear` command clears all items from the provide queue and prints out the number of items removed from the queue. The `quiet` option tells the command not to print output.
* refactor(cmds): ipfs provide clear
moving to new namespace to avoid conflicts, and also document other
commands
* docs: clarify Reprovider.Strategy
* chore: remove undesired md link
this wires up https://github.com/ipfs/boxo/pull/971
to make sure explicitly allowlisted hosts have
their own metric label
if we ever need more flexibility here, this can be exposed as
a separate config
* Configure bitswap braodcast reduction
Add new config items to `Internal.Bitswap` to allow configuration of bitswap broadcast reduction behavior. Broadcast reduction behavior is enabled by default, and uses settings that should be suitable for most installations of kubo.
* update sharness metrics test
* Explicit defaults for broadcast reduction configuration
* Update docs/config.md
* chore: update to go-log/v2
go-log v2 has been out for quite a while now and it is time to deprecate v1.
Replace all use of go-log with go-log/v2
Makes /api/v0/log/tail useful over HTTP
Updates dependencies that have moved to go-lov/v2
Removes support for ContextWithLoggable as this is not needed for tracing-like functionality
- Replaces: PR #8765
- Closes issue #8753
- Closes issue #9245
- Closes issue #10809
Other fixes:
* update go-ipfs-cmds
* update http logs test
* fix test
* Read/send one line of log data at a time
* Update -log-level docs
* fix(config): explicit Provider.Enabled flag
Adds missing config option described in
https://github.com/ipfs/kubo/issues/10803
* refactor: remove Experimental.StrategicProviding
removing experiment, replaced with Provider.Enabled
* test(cli): routing [re]provide
updated and added tests for manually triggering provide and reprovide
and making them respect global configuration flag to avoid
inconsistent behaviors
* docs: improve DelegatedRouters
* refactor: default DefaultProviderWorkerCount=16
- simplified default for both
- 16 is safer for non-accelerated DHT client
- acceletated DHT performs better without limit anyway - updated docs
* Feat: http retrieval as experimental feature
This introduces the http-retrieval capability as an experimental feature.
It can be enabled in the configuration `Experimental.HTTPRetrieval.Enabled = true`.
Documentation and changelog to be added later.
* refactor: HTTPRetrieval.Enabled as Flag
* docs(config): HTTPRetrieval section
* refactor: reusable MockHTTPContentRouter
* feat: HTTPRetrieval.TLSInsecureSkipVerify
allows self-signed certificates in tests
* feat(config): HTTPRetrieval.MaxBlockSize
* test: end-to-end HTTPRetrieval.Enabled
this spawns two http services on localhost:
1. HTTP router that returns HTTP provider when /routing/v1/providers/cid i queried
2. HTTP provider that returns a block when /ipfs/cid is queried
3. Configures Kubo to use (1) instead of cid.contact
this seems to work (running test with DEBUG=true shows (1) was queried
for the test CID and returned multiaddr of (2), but Kubo never requested
test CID block from (2) – needs investigation
* fix: enable /routing/v1/peers for non-cid.contact
we artificially limited every delegated routing endpoint because of
cid.contact being limited to one endpoint
* feat: Routing.DelegatedRouters
make it easy to override the hardcoded implicit HTTP routeur URL
without having to set the entire custom Router.Routers and
Router.Methods
(http_retrieval_client_test.go still needs to be fixed in future commit)
* test: flag remaining work
* docs: review feedback
* refactor: providerQueryMgr with bitswapNetworks
this fixes two regressions:
(1) introduced in https://github.com/ipfs/kubo/issues/10717
where we only used bitswapLib2p query manager
(this is why E2E did not act on http provider)
(2) introduced in https://github.com/ipfs/kubo/pull/10765
where it was not possible to set binary peerID in IgnoreProviders
(we changed to []string)
* refactor: Bitswap.Libp2pEnabled
replaces Bitswap.Enabled with Bitswap.Libp2pEnabled
adds tests that confirm it is possible to disable libp2p bitswap fully
and only keep http in client mode
also, removes the need for passing empty blockstore in client-only mode
* docs: changelog
---------
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* feat: add Bitswap configuration and related tests
* fix: update Bitswap function to use 'provide' parameter for server enablement
* docs: update changelog for Bitswap functionality changes
* fix: update Bitswap server enablement logic and improve related tests
* fix: rename BitswapConfig to Bitswap and update references
* docs: config and changelog
* fix: `ipfs cat` panic when `Bitswap.Enabled=false`
Fixes panic described in:
https://github.com/ipfs/kubo/pull/10782#discussion_r2069116219
---------
Co-authored-by: gystemd <gystemd@gmail.com>
Co-authored-by: gammazero <11790789+gammazero@users.noreply.github.com>
Co-authored-by: Giulio Piva <giulio.piva@dedicated.world>
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* adjust ipfs stats provide
* update boxo dep
* bump boxo
* fixing tests
* docs/chore: mark stat reprovide as experimental
* docs: Provider.Strategy
explicitly document it is not used - without this legacy users will have
it in their config and be very confused
---------
Co-authored-by: Marcin Rataj <lidel@lidel.org>
Adds `Routing.IgnoreProviders`.
This requires initializing a custom providerQueryManager and using it instead
of the default created internally in Bitswap. Since the default is created
with some internal default configuration options (MaxProviders), this hardcodes it.
Fixes#10596.
The reproviding process can take long. Currently, each CID to be provided is
obtained by making a query to the pinner and reading one by one as the CIDs
get provided.
While this query is ongoing, the pinner holds a Read mutex on the pinset.
If a pin-add-request arrives, a goroutine will start waiting for a Write mutex
on the pinset. From that point, no new Read mutexes can be taken until the writer
can proceed and finishes.
However, no one can proceed because the read mutex is still held while the
reproviding is ongoing.
The fix is mostly in Boxo, where we add a "buffered" provider which reads the
cids onto memory so that they can be provided at its own pace without making
everyone wait.
The consequence is we will need more RAM memory. Rule of thumb is 1GiB extra per 20M cids to be reprovided.
* use go-datastore without go-process
* update go-ds-xxx dependencies
* update go-libp2p-kad-dht
* bitswap api changes
* Do not use multiple multi-error packages, pick one
* update boxo
* update expected metrics
* feat(libp2p): enable shared TCP listeners
* docs: switch mentions of /ws to /tcp/4001
* feat: AutoTLS.AutoWSS
This adds AutoTLS.AutoWSS flag that is set to true by default.
It will check if Addresses.Swarm contain explicit /ws listener,
and if not found, it will append one per every /tcp listener
This way existing TCP ports are reused without any extra configuration,
but we don't break user's who have custom / explicit /ws listener
already.
I also moved logger around, to include Addresses.Swarm inspection
results in `autotls` logger.
* chore: go-libp2p v0.38.1
https://github.com/libp2p/go-libp2p/releases/tag/v0.38.0https://github.com/libp2p/go-libp2p/releases/tag/v0.38.1
* docs: AutoTLS.AutoWSS and go-libp2p v0.38.x
* chore: p2p-forge/client v0.2.0
https://github.com/ipshipyard/p2p-forge/releases/tag/v0.2.0
* fix: disable libp2p.ShareTCPListener() in PNET
* chore(ci): timeout sharness after 15m
average successful run is <9 minutes, no need to wait for 20
https://github.com/ipfs/kubo/actions/workflows/sharness.yml?query=is%3Asuccess
---------
Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com>
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* feat: expose BlockKeyCacheSize and enable WriteThrough when bloom filter disabled
* import/config: add BatchMaxSize and BatchMaxNodes
* config: make BlockKeyCacheSize an OptionalInteger
* config: add and wire datastore.WriteThrough option
* config: omitempty on BlockKeyCacheSize
* changelog: rewrite entry about new options for the datastore
* config: add docs for BatchMaxNodes and BatchMaxSize
* config: make WriteThrough an optional Flag
* changelog: improve description of new datastore/import options
* refactor: DefaultWriteThrough as bool
* chore: boxo v0.26.0
* docs: config and changelog fixes
* chore: update to boxo without goprocess
* Use boxo fix for registering metrics
* chore: switch to boxo main with PR 723
---------
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* fix(autotls): store certificates at the location from the repo path
* docs(autotls): cert storale and other caveats
---------
Co-authored-by: Marcin Rataj <lidel@lidel.org>
(cherry picked from commit 1ca0ae0af6)
Allow configuration of the bitswap server's replace WantHave with WantBlock maximum block size using the Internal.Bitswap.WantHaveReplaceSize config item. This sets the maximum size of a block in bytes up to which we will replace a want-have with a want-block. Setting a size of 0 disables this replacement and means that block sizes are not read for WantHave requests.
See ipfs/boxo#672 for more details
Updated boxo to version with PR 672
---------
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* feat: libp2p.EnableAutoNATv2
Part of https://github.com/ipfs/kubo/issues/10091
We include a flag that allows shutting down V2 in case there are issues
with it.
* docs: EnableAutoNATv2
