* feat(cmds/add): --to-files option as files cp
* tests(to-files): ensure error handling is covered
this adds bunch of tests that guard UX around importing multiple files
into MFS, and the logic around trailing slash to indicate if the MFS
destination if a directory.
If the destination has a trailing slash, we ensure that the directory
exists and is a dir and not a file. this allows us to support
adding multipl files into MFS dir:
ipfs add file1.txt file2.txt --to-files /some/mfs/dir/
* docs: to-files helptext
Co-authored-by: Antonio Navarro Perez <antnavper@gmail.com>
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* Delegated Routing.
Implementation of Reframe specs (https://github.com/ipfs/specs/blob/master/REFRAME.md) using go-delegated-routing library.
* Requested changes.
* Init using op string
* Separate possible ContentRouters for TopicDiscovery.
If we don't do this, we have a ciclic dependency creating TieredRouter.
Now we can create first all possible content routers, and after that,
create Routers.
* Set dht default routing type
* Add tests and remove uneeded code
* Add documentation.
* docs: Routing.Routers
* Requested changes.
Signed-off-by: Antonio Navarro Perez <antnavper@gmail.com>
* Add some documentation on new fx functions.
* Add changelog entry and integration tests
* test: sharness for 'dht' in 'routing' commands
Since 'routing' is currently the same as 'dht' (minus query command)
we need to test both, that way we won't have unnoticed divergence
in the default behavior.
* test(sharness): delegated routing via reframe URL
* Add more tests for delegated routing.
* If any put operation fails, the tiered router will fail.
* refactor: Routing.Routers: Parameters.Endpoint
As agreed in https://github.com/ipfs/kubo/pull/8997#issuecomment-1175684716
* Try to improve CHANGELOG entry.
* chore: update reframe spec link
* Update go-delegated-routing dependency
* Fix config error test
* use new changelog format
* Remove port conflict
* go mod tidy
* ProviderManyWrapper to ProviderMany
* Update docs/changelogs/v0.14.md
Co-authored-by: Adin Schmahmann <adin.schmahmann@gmail.com>
Co-authored-by: Marcin Rataj <lidel@lidel.org>
Co-authored-by: Adin Schmahmann <adin.schmahmann@gmail.com>
* feat: upgrade to go-libp2p-kad-dht@v0.16.0
* rename "cid format --codec" to "cid format --mc"
* refactor(test): from --codec to --mc
Co-authored-by: Marcin Rataj <lidel@lidel.org>
This periodically logs how many times Resource Manager limits were
exceeded. If they aren't exceeded, then nothing is logged. The log
levels are at ERROR log level so that they are shown by default.
The motivation is so that users know when they have exceeded resource
manager limits. To find what is exceeding the limits, they'll need to
turn on debug logging and inspect the errors being logged. This could
collect the specific limits being reached, but that's more complicated
to implement and could result in much longer log messages.
Fixes#8957
The context was only checked while reading data.
Not while writing data to the http connection.
So since the data flow through an io.Pipe the closing didn't flowed through and left the writer open hanging.
Co-authored-by: Antonio Navarro Perez <antnavper@gmail.com>
* Add 'ipfs repo migrate' command
This PR replaces #7658 that was originally contributed by zaibons, in order to move code into a branch and avoid some CI problem.
The command allows the user to run the repo migration without starting the daemon.
resolves#7471
* return non-ErrNeedMigration errors from fsrepo.Open()
Co-authored-by: Gus Eggert <gus@gus.dev>
* pubsub multibase encoding
Adds clarification for pubsub multibase encoding over HTTP RPC for issue https://github.com/ipfs/ipfs-docs/issues/1007
* Grammatical change
* Moved period
* feat: persist limit changes to config
This changes the "ipfs swarm limit" command so that when limit changes
are applied via the command line, they are persisted to the repo
config, so that they remain in effect when the daemon restarts.
Any existing limit.json can be dropped into the IPFS config easily
using something like:
cat ~/.ipfs/config | jq ".Swarm.ResourceMgr.Limits = $(cat limit.json)" | sponge ~/.ipfs/config
This also upgrades to Resource Manager v0.3.0, which exports the config
schema so that we don't have to maintain our own copy of it.
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* feat: add block profiling to collect-profiles.sh
* feat: add more profiles to 'ipfs diag profile'
This adds mutex and block profiles, and brings the command up-to-par
with 'collect-profiles.sh', so that we can remove it.
Profiles are also now collected concurrently, which improves the
runtime from (profile_time * num_profiles) to just (profile_time).
Note that this has a backwards-incompatible change, removing
--cpu-profile-time in favor of the more general --profile-time, which
covers all sampling profiles.
* docs(cli): ipfs diag profile
* add CLI flag to select specific diag collectors
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* update go-libp2p to v0.18.0
* initialize the resource manager
* add resource manager stats/limit commands
* load limit file when building resource manager
* log absent limit file
* write rcmgr to file when IPFS_DEBUG_RCMGR is set
* fix: mark swarm limit|stats as experimental
* feat(cfg): opt-in Swarm.ResourceMgr
This ensures we can safely test the resource manager without impacting
default behavior.
- Resource manager is disabled by default
- Default for Swarm.ResourceMgr.Enabled is false for now
- Swarm.ResourceMgr.Limits allows user to tweak limits per specific
scope in a way that is persisted across restarts
- 'ipfs swarm limit system' outputs human-readable json
- 'ipfs swarm limit system new-limits.json' sets new runtime limits
(but does not change Swarm.ResourceMgr.Limits in the config)
Conventions to make libp2p devs life easier:
- 'IPFS_RCMGR=1 ipfs daemon' overrides the config and enables resource manager
- 'limit.json' overrides implicit defaults from libp2p (if present)
* docs(config): small tweaks
* fix: skip libp2p.ResourceManager if disabled
This ensures 'ipfs swarm limit|stats' work only when enabled.
* fix: use NullResourceManager when disabled
This reverts commit b19f7c9eca.
after clarification feedback from
https://github.com/ipfs/go-ipfs/pull/8680#discussion_r841680182
* style: rename IPFS_RCMGR to LIBP2P_RCMGR
preexisting libp2p toggles use LIBP2P_ prefix
* test: Swarm.ResourceMgr
* fix: location of opt-in limit.json and rcmgr.json.gz
Places these files inside of IPFS_PATH
* Update docs/config.md
* feat: expose rcmgr metrics when enabled (#8785)
* add metrics for the resource manager
* export protocol and service name in Prometheus metrics
* fix: expose rcmgr metrics only when enabled
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* refactor: rcmgr_metrics.go
* refactor: rcmgr_defaults.go
This file defines implicit limit defaults used when Swarm.ResourceMgr.Enabled
We keep vendored copy to ensure go-ipfs is not impacted when go-libp2p
decides to change defaults in any of the future releases.
* refactor: adjustedDefaultLimits
Cleans up the way we initialize defaults and adds a fix for case
when connection manager runs with high limits.
It also hides `Swarm.ResourceMgr.Limits` until we have a better
understanding what syntax makes sense.
* chore: cleanup after a review
* fix: restore go-ipld-prime v0.14.2
* fix: restore go-ds-flatfs v0.5.1
Co-authored-by: Lucas Molas <schomatis@gmail.com>
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* feat: add support for carv2 import
* update to multicodec 0.4.0
* add sharness test for carv2
Co-authored-by: Keenan Nemetz <keenan.nemetz@gmail.com>
* feat(cmds): add deprecated and experimental status
Added programmatic state annotation introduced in
https://github.com/ipfs/go-ipfs-cmds/pull/225
for already deprecated / experimental commands.
* chore: go-ipfs-cmds v0.7.0
Co-authored-by: Lucas Molas <schomatis@gmail.com>
Bubble ErrNotFound improvements.
The is a really nice UX improvement because it know tell you which ipld block is missing instead of a generic "not found".
