* Delegated Routing.
Implementation of Reframe specs (https://github.com/ipfs/specs/blob/master/REFRAME.md) using go-delegated-routing library.
* Requested changes.
* Init using op string
* Separate possible ContentRouters for TopicDiscovery.
If we don't do this, we have a ciclic dependency creating TieredRouter.
Now we can create first all possible content routers, and after that,
create Routers.
* Set dht default routing type
* Add tests and remove uneeded code
* Add documentation.
* docs: Routing.Routers
* Requested changes.
Signed-off-by: Antonio Navarro Perez <antnavper@gmail.com>
* Add some documentation on new fx functions.
* Add changelog entry and integration tests
* test: sharness for 'dht' in 'routing' commands
Since 'routing' is currently the same as 'dht' (minus query command)
we need to test both, that way we won't have unnoticed divergence
in the default behavior.
* test(sharness): delegated routing via reframe URL
* Add more tests for delegated routing.
* If any put operation fails, the tiered router will fail.
* refactor: Routing.Routers: Parameters.Endpoint
As agreed in https://github.com/ipfs/kubo/pull/8997#issuecomment-1175684716
* Try to improve CHANGELOG entry.
* chore: update reframe spec link
* Update go-delegated-routing dependency
* Fix config error test
* use new changelog format
* Remove port conflict
* go mod tidy
* ProviderManyWrapper to ProviderMany
* Update docs/changelogs/v0.14.md
Co-authored-by: Adin Schmahmann <adin.schmahmann@gmail.com>
Co-authored-by: Marcin Rataj <lidel@lidel.org>
Co-authored-by: Adin Schmahmann <adin.schmahmann@gmail.com>
* fix: remove mdns_legacy
We've been running both implementations for a long, long time.
It is time to remove legacy version and lower the number of LAN packets
IPFS node produces.
See https://github.com/ipfs/go-ipfs/pull/9048#discussion_r906814717
for the Interval removal rational.
This remove github.com/minio/blake2b-simd and replace it with golang.org/x/crypto/blake2b which slightly faster and reduce the code surface.
See https://github.com/multiformats/go-multihash/pull/157 for more info.
* feat: upgrade to go-libp2p-kad-dht@v0.16.0
* rename "cid format --codec" to "cid format --mc"
* refactor(test): from --codec to --mc
Co-authored-by: Marcin Rataj <lidel@lidel.org>
This periodically logs how many times Resource Manager limits were
exceeded. If they aren't exceeded, then nothing is logged. The log
levels are at ERROR log level so that they are shown by default.
The motivation is so that users know when they have exceeded resource
manager limits. To find what is exceeding the limits, they'll need to
turn on debug logging and inspect the errors being logged. This could
collect the specific limits being reached, but that's more complicated
to implement and could result in much longer log messages.
* update go-libp2p to v0.19.0
* chore: go-namesys v0.5.0
* refactor(config): cleanup relay handling
* docs(config): document updated defaults
* fix(tests): panic during sharness
* fix: t0160-resolve.sh
See https://github.com/ipfs/go-namesys/pull/32
* fix: t0182-circuit-relay.sh
* test: transport encryption
Old tests were no longer working because go-libp2p 0.19 removed
the undocumented 'ls' pseudoprotocol.
This replaces these tests with handshake attempt (name is echoed back on
OK or 'na' is returned when protocol is not available) for tls and noise
variants + adds explicit test that safeguards us against enabling
plaintext by default by a mistake.
* fix: ./t0182-circuit-relay.sh
test is flaky, for now we just restart the testbed when we get
NO_RESERVATION error
* refactor: AutoRelayFeeder with exp. backoff
It starts at feeding peers ever 15s, then backs off each time
until it is done once an hour
Should be acceptable until we have smarter mechanism in go-lib2p 0.20
* feat(AutoRelay): prioritize Peering.Peers
This ensures we feed trusted Peering.Peers in addition to any peers
discovered over DHT.
* docs(CHANGELOG): document breaking changes
Co-authored-by: Marcin Rataj <lidel@lidel.org>
Co-authored-by: Gus Eggert <gus@gus.dev>
* feat: persist limit changes to config
This changes the "ipfs swarm limit" command so that when limit changes
are applied via the command line, they are persisted to the repo
config, so that they remain in effect when the daemon restarts.
Any existing limit.json can be dropped into the IPFS config easily
using something like:
cat ~/.ipfs/config | jq ".Swarm.ResourceMgr.Limits = $(cat limit.json)" | sponge ~/.ipfs/config
This also upgrades to Resource Manager v0.3.0, which exports the config
schema so that we don't have to maintain our own copy of it.
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* fix: go mod tidy
* fix: update expected prometheus metrics
Go 1.17 includes some new runtime metrics that are automatically
published by the Prometheus client, so this adds them to the expected
metric list so that the tests pass and we don't accidentally drop them
in the future.
Since go1.16, there are built in tools that allow for embeding
filesystem inside the binary. We now make use of the `embed` package to
have all files put into the binary, removing the need to generate the
files and removes dependencies
Co-authored-by: Jorropo <jorropo.pgm@gmail.com>
This adds simple check that will scream loud and clear every time
go-libp2p libraries change any of the implicit defaults
related to go-libp2p-resource-manager
* update go-libp2p to v0.18.0
* initialize the resource manager
* add resource manager stats/limit commands
* load limit file when building resource manager
* log absent limit file
* write rcmgr to file when IPFS_DEBUG_RCMGR is set
* fix: mark swarm limit|stats as experimental
* feat(cfg): opt-in Swarm.ResourceMgr
This ensures we can safely test the resource manager without impacting
default behavior.
- Resource manager is disabled by default
- Default for Swarm.ResourceMgr.Enabled is false for now
- Swarm.ResourceMgr.Limits allows user to tweak limits per specific
scope in a way that is persisted across restarts
- 'ipfs swarm limit system' outputs human-readable json
- 'ipfs swarm limit system new-limits.json' sets new runtime limits
(but does not change Swarm.ResourceMgr.Limits in the config)
Conventions to make libp2p devs life easier:
- 'IPFS_RCMGR=1 ipfs daemon' overrides the config and enables resource manager
- 'limit.json' overrides implicit defaults from libp2p (if present)
* docs(config): small tweaks
* fix: skip libp2p.ResourceManager if disabled
This ensures 'ipfs swarm limit|stats' work only when enabled.
* fix: use NullResourceManager when disabled
This reverts commit b19f7c9eca.
after clarification feedback from
https://github.com/ipfs/go-ipfs/pull/8680#discussion_r841680182
* style: rename IPFS_RCMGR to LIBP2P_RCMGR
preexisting libp2p toggles use LIBP2P_ prefix
* test: Swarm.ResourceMgr
* fix: location of opt-in limit.json and rcmgr.json.gz
Places these files inside of IPFS_PATH
* Update docs/config.md
* feat: expose rcmgr metrics when enabled (#8785)
* add metrics for the resource manager
* export protocol and service name in Prometheus metrics
* fix: expose rcmgr metrics only when enabled
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* refactor: rcmgr_metrics.go
* refactor: rcmgr_defaults.go
This file defines implicit limit defaults used when Swarm.ResourceMgr.Enabled
We keep vendored copy to ensure go-ipfs is not impacted when go-libp2p
decides to change defaults in any of the future releases.
* refactor: adjustedDefaultLimits
Cleans up the way we initialize defaults and adds a fix for case
when connection manager runs with high limits.
It also hides `Swarm.ResourceMgr.Limits` until we have a better
understanding what syntax makes sense.
* chore: cleanup after a review
* fix: restore go-ipld-prime v0.14.2
* fix: restore go-ds-flatfs v0.5.1
Co-authored-by: Lucas Molas <schomatis@gmail.com>
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* feat: add support for carv2 import
* update to multicodec 0.4.0
* add sharness test for carv2
Co-authored-by: Keenan Nemetz <keenan.nemetz@gmail.com>
* add deprecation warning when tracer plugins are loaded
* add response format attribute to span in gateway handler
* add note about tracing's experimental status in godoc
* add nil check for TTL when adding name span attrs
* add basic sharness test for integration with otel collector
* add nil check in UnixFSAPI.processLink
* test: sharness check all json objs for swarm span
* add env var docs to docs/environment-variables.md
* chore: pin the otel collector version
* add tracing spans per response type (#8841)
* docs: tracing with jaeger-ui
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* feat(cmds): add deprecated and experimental status
Added programmatic state annotation introduced in
https://github.com/ipfs/go-ipfs-cmds/pull/225
for already deprecated / experimental commands.
* chore: go-ipfs-cmds v0.7.0
Co-authored-by: Lucas Molas <schomatis@gmail.com>
Bubble ErrNotFound improvements.
The is a really nice UX improvement because it know tell you which ipld block is missing instead of a generic "not found".
