This is intended as a replacement for sharness. These are vanilla Go
tests which can be run in your IDE for quick iteration on end-to-end
CLI tests.
This also removes IPTB by duplicating its functionality in the test
harness. This isn't a big deal...IPTB's complexity is mostly around
the fact that its state needs to be saved to disk in between `iptb`
command invocations, and that it uses Go plugins to inject
functionality, neither of which are relevant here.
If we merge this, we'll have to live with bifurcated tests for a while
until they are all migrated. I'd recommend we self-enforce a rule
that, if we need to touch a sharness test, we migrate it and one more
test over to Go tests first. Then eventually we will have migrated
everything.
This PR adds several new functionalities to make easier the usage of ResourceManager:
- Now resource manager logs when resources are exceeded are on ERROR instead of warning.
- The resources exceeded error now shows what kind of limit was reached and the scope.
- When there was no limit exceeded, we print a message for the user saying that limits are not exceeded anymore.
- Added `swarm limit all` command to show all set limits with the same format as `swarm stats all`
- Added `min-used-limit-perc` option to `swarm stats all` to only show stats that are above a specific percentage
- Simplify a lot default values.
- **Enable ResourceManager by default.**
Output example:
```
2022-11-09T10:51:40.565+0100 ERROR resourcemanager libp2p/rcmgr_logging.go:59 Consider inspecting logs and raising the resource manager limits. Documentation: https://github.com/ipfs/kubo/blob/master/docs/config.md#swarmresourcemgr
2022-11-09T10:51:50.565+0100 ERROR resourcemanager libp2p/rcmgr_logging.go:55 Resource limits were exceeded 483095 times with error "transient: cannot reserve inbound stream: resource limit exceeded".
2022-11-09T10:51:50.565+0100 ERROR resourcemanager libp2p/rcmgr_logging.go:59 Consider inspecting logs and raising the resource manager limits. Documentation: https://github.com/ipfs/kubo/blob/master/docs/config.md#swarmresourcemgr
2022-11-09T10:52:00.565+0100 ERROR resourcemanager libp2p/rcmgr_logging.go:55 Resource limits were exceeded 455294 times with error "transient: cannot reserve inbound stream: resource limit exceeded".
2022-11-09T10:52:00.565+0100 ERROR resourcemanager libp2p/rcmgr_logging.go:59 Consider inspecting logs and raising the resource manager limits. Documentation: https://github.com/ipfs/kubo/blob/master/docs/config.md#swarmresourcemgr
2022-11-09T10:52:10.565+0100 ERROR resourcemanager libp2p/rcmgr_logging.go:55 Resource limits were exceeded 471384 times with error "transient: cannot reserve inbound stream: resource limit exceeded".
2022-11-09T10:52:10.565+0100 ERROR resourcemanager libp2p/rcmgr_logging.go:59 Consider inspecting logs and raising the resource manager limits. Documentation: https://github.com/ipfs/kubo/blob/master/docs/config.md#swarmresourcemgr
2022-11-09T10:52:20.565+0100 ERROR resourcemanager libp2p/rcmgr_logging.go:55 Resource limits were exceeded 8 times with error "peer:12D3KooWKqcaBtcmZKLKCCoDPBuA6AXGJMNrLQUPPMsA5Q6D1eG6: cannot reserve inbound stream: resource limit exceeded".
2022-11-09T10:52:20.565+0100 ERROR resourcemanager libp2p/rcmgr_logging.go:55 Resource limits were exceeded 192 times with error "peer:12D3KooWPjetWPGQUih9LZTGHdyAM9fKaXtUxDyBhA93E3JAWCXj: cannot reserve inbound stream: resource limit exceeded".
2022-11-09T10:52:20.565+0100 ERROR resourcemanager libp2p/rcmgr_logging.go:55 Resource limits were exceeded 469746 times with error "transient: cannot reserve inbound stream: resource limit exceeded".
2022-11-09T10:52:20.565+0100 ERROR resourcemanager libp2p/rcmgr_logging.go:59 Consider inspecting logs and raising the resource manager limits. Documentation: https://github.com/ipfs/kubo/blob/master/docs/config.md#swarmresourcemgr
2022-11-09T10:52:30.565+0100 ERROR resourcemanager libp2p/rcmgr_logging.go:55 Resource limits were exceeded 484137 times with error "transient: cannot reserve inbound stream: resource limit exceeded".
2022-11-09T10:52:30.565+0100 ERROR resourcemanager libp2p/rcmgr_logging.go:55 Resource limits were exceeded 29 times with error "peer:12D3KooWPjetWPGQUih9LZTGHdyAM9fKaXtUxDyBhA93E3JAWCXj: cannot reserve inbound stream: resource limit exceeded".
2022-11-09T10:52:30.565+0100 ERROR resourcemanager libp2p/rcmgr_logging.go:59 Consider inspecting logs and raising the resource manager limits. Documentation: https://github.com/ipfs/kubo/blob/master/docs/config.md#swarmresourcemgr
2022-11-09T10:52:40.565+0100 ERROR resourcemanager libp2p/rcmgr_logging.go:55 Resource limits were exceeded 468843 times with error "transient: cannot reserve inbound stream: resource limit exceeded".
2022-11-09T10:52:40.566+0100 ERROR resourcemanager libp2p/rcmgr_logging.go:59 Consider inspecting logs and raising the resource manager limits. Documentation: https://github.com/ipfs/kubo/blob/master/docs/config.md#swarmresourcemgr
2022-11-09T10:52:50.566+0100 ERROR resourcemanager libp2p/rcmgr_logging.go:55 Resource limits were exceeded 366638 times with error "transient: cannot reserve inbound stream: resource limit exceeded".
2022-11-09T10:52:50.566+0100 ERROR resourcemanager libp2p/rcmgr_logging.go:59 Consider inspecting logs and raising the resource manager limits. Documentation: https://github.com/ipfs/kubo/blob/master/docs/config.md#swarmresourcemgr
2022-11-09T10:53:00.566+0100 ERROR resourcemanager libp2p/rcmgr_logging.go:55 Resource limits were exceeded 405526 times with error "transient: cannot reserve inbound stream: resource limit exceeded".
2022-11-09T10:53:00.566+0100 ERROR resourcemanager libp2p/rcmgr_logging.go:55 Resource limits were exceeded 107 times with error "peer:12D3KooWQZQCwevTDGhkE9iGYk5sBzWRDUSX68oyrcfM9tXyrs2Q: cannot reserve inbound stream: resource limit exceeded".
2022-11-09T10:53:00.566+0100 ERROR resourcemanager libp2p/rcmgr_logging.go:59 Consider inspecting logs and raising the resource manager limits. Documentation: https://github.com/ipfs/kubo/blob/master/docs/config.md#swarmresourcemgr
2022-11-09T10:53:10.566+0100 ERROR resourcemanager libp2p/rcmgr_logging.go:55 Resource limits were exceeded 336923 times with error "transient: cannot reserve inbound stream: resource limit exceeded".
2022-11-09T10:53:10.566+0100 ERROR resourcemanager libp2p/rcmgr_logging.go:59 Consider inspecting logs and raising the resource manager limits. Documentation: https://github.com/ipfs/kubo/blob/master/docs/config.md#swarmresourcemgr
2022-11-09T10:53:20.565+0100 ERROR resourcemanager libp2p/rcmgr_logging.go:55 Resource limits were exceeded 71 times with error "transient: cannot reserve inbound stream: resource limit exceeded".
2022-11-09T10:53:20.565+0100 ERROR resourcemanager libp2p/rcmgr_logging.go:59 Consider inspecting logs and raising the resource manager limits. Documentation: https://github.com/ipfs/kubo/blob/master/docs/config.md#swarmresourcemgr
2022-11-09T10:53:30.565+0100 ERROR resourcemanager libp2p/rcmgr_logging.go:64 Resrouce limits are no longer being exceeded.
```
## Validation tests
- Accelerated DHT client runs with no errors when ResourceManager is active. No problems were observed.
- Running an attack with 200 connections and 1M streams using yamux protocol. Node was usable during the attack. With ResourceManager deactivated, the node was killed by the OS because of the amount of memory consumed.
- Actions done when the attack was active:
- Add files
- Force a reprovide
- Use the gateway to resolve an IPNS address.
It closes#9001
It closes#9351
It closes#9322
* feat: --reset flag on swarm limit command
This flag allows to the user to reset limits to default values.
Signed-off-by: Antonio Navarro Perez <antnavper@gmail.com>
* Use adjusted default limits and remove already fixed FIXME
Signed-off-by: Antonio Navarro Perez <antnavper@gmail.com>
* Apply suggestions from code review
Co-authored-by: Gus Eggert <gus@gus.dev>
* Return correct defaults
* Remove resetting all values from a map.
Signed-off-by: Antonio Navarro Perez <antnavper@gmail.com>
Signed-off-by: Antonio Navarro Perez <antnavper@gmail.com>
Co-authored-by: Gus Eggert <gus@gus.dev>
`swarm stats all` requires that the ResourceManager instance
implements `rcmgr.ResourceManagerState`, and `loggingResourceManager`
was not implementing it, so the command was failing.
Also added a sharness test to check that the command is executing
correctly, because `jq -e` doesn't return an error if the json is nil.
Signed-off-by: Antonio Navarro Perez <antnavper@gmail.com>
New multi-router configuration system based on https://hackmd.io/G1KRDEX5T3qyfoBMkIrBew#Methods
- Added a new routing type: "custom"
- Added specific struct types for different Routers (instead of map[string]interface{})
- Added `Duration` config type, to make easier time string parsing
- Added config documentation.
- Use the latest go-delegated-routing library version with GET support.
- Added changelog notes for this feature.
It:
- closes#9157
- closes#9079
- closes#9186
* feat(cmds/add): --to-files option as files cp
* tests(to-files): ensure error handling is covered
this adds bunch of tests that guard UX around importing multiple files
into MFS, and the logic around trailing slash to indicate if the MFS
destination if a directory.
If the destination has a trailing slash, we ensure that the directory
exists and is a dir and not a file. this allows us to support
adding multipl files into MFS dir:
ipfs add file1.txt file2.txt --to-files /some/mfs/dir/
* docs: to-files helptext
Co-authored-by: Antonio Navarro Perez <antnavper@gmail.com>
Co-authored-by: Marcin Rataj <lidel@lidel.org>
This cleans up old CORS tests and adds more resolution
(proper Origin test, testing custom header behavior)
It also adds basic regression tests for /api/v0
subset exposed on Gateway port.
This adds a plugin interface that lets the plugin modify the fx
options that are passed to fx when the app is initialized. This means
plugins can inject their own implementations of IPFS interfaces. This
enables granular customization of go-ipfs behavior by plugins, such
as:
- Bitswap with custom filters (e.g. for CID blocking) Custom interface
- implementations such as Pinner or DAGService
- Dynamic configuration of libp2p ...
One downside of this is that we're exposing the entire dependency
graph, init hooks, initialization, etc. to users, so this comes with a
caveat that we reserve the right to make breaking changes to the graph
structure and initialization logic (although this historically happens
rarely). If these things are changed, we should mention them in
release notes and changelogs though, since they could impact users of
this plugin interface.
I'm not particularly fond of DI frameworks (and neither are some of
the folks work on/near go-ipfs), but it seems unlikely that somebody
will rewrite the dependency wiring and lifecycle hooks of go-ipfs, and
add dynamic extension points, so this seems like a palatable
compromise.
There are also problems that we should clean up in how model the
go-ipfs app in fx, such as:
- We make extensive use of nested fx.Options, which fx itself
discourages because it "limits the user's ability to customize their
application". It should be easy to flatten these out into a single
[]fx.Option slice.
- We pass around a list of opaque libp2p opts, which makes it hard to
customize after-the-fact...we should consider naming each of these
opts and providing them to fx as proper dependencies, so that they can
be explicitly overridden.
- We call fx.Invoke() in some places with anonymous functions. We
should instead only pass exported functions to fx.Invoke(), so that
they have exported names, which would make it easier to remove/augment
the invocations that happen when the app is initialized.
These aren't blocking issues, they just make it harder and more
brittle to customize go-ipfs with this plugin.
Support hashes from 20 <= x <= 128 bytes, but does not yet have any option to generate anything else than 32 bytes hashes.
This will be forward compatible when we add such option.
The file contains the gateway your node is hosting in the http://<host>:<port> RFC 3986 format.
Structurally it works exactly the same as the API file.
