* feat: add block profiling to collect-profiles.sh
* feat: add more profiles to 'ipfs diag profile'
This adds mutex and block profiles, and brings the command up-to-par
with 'collect-profiles.sh', so that we can remove it.
Profiles are also now collected concurrently, which improves the
runtime from (profile_time * num_profiles) to just (profile_time).
Note that this has a backwards-incompatible change, removing
--cpu-profile-time in favor of the more general --profile-time, which
covers all sampling profiles.
* docs(cli): ipfs diag profile
* add CLI flag to select specific diag collectors
Co-authored-by: Marcin Rataj <lidel@lidel.org>
Since go1.16, there are built in tools that allow for embeding
filesystem inside the binary. We now make use of the `embed` package to
have all files put into the binary, removing the need to generate the
files and removes dependencies
Co-authored-by: Jorropo <jorropo.pgm@gmail.com>
* fix: report gateway http metrics only when response is successful
* fix(gw): 304 Not Modified as no-op
This fix ensures we don't do any additional work when Etag match
what user already has in their own cache.
Co-authored-by: Marcin Rataj <lidel@lidel.org>
This adds simple check that will scream loud and clear every time
go-libp2p libraries change any of the implicit defaults
related to go-libp2p-resource-manager
* update go-libp2p to v0.18.0
* initialize the resource manager
* add resource manager stats/limit commands
* load limit file when building resource manager
* log absent limit file
* write rcmgr to file when IPFS_DEBUG_RCMGR is set
* fix: mark swarm limit|stats as experimental
* feat(cfg): opt-in Swarm.ResourceMgr
This ensures we can safely test the resource manager without impacting
default behavior.
- Resource manager is disabled by default
- Default for Swarm.ResourceMgr.Enabled is false for now
- Swarm.ResourceMgr.Limits allows user to tweak limits per specific
scope in a way that is persisted across restarts
- 'ipfs swarm limit system' outputs human-readable json
- 'ipfs swarm limit system new-limits.json' sets new runtime limits
(but does not change Swarm.ResourceMgr.Limits in the config)
Conventions to make libp2p devs life easier:
- 'IPFS_RCMGR=1 ipfs daemon' overrides the config and enables resource manager
- 'limit.json' overrides implicit defaults from libp2p (if present)
* docs(config): small tweaks
* fix: skip libp2p.ResourceManager if disabled
This ensures 'ipfs swarm limit|stats' work only when enabled.
* fix: use NullResourceManager when disabled
This reverts commit b19f7c9eca.
after clarification feedback from
https://github.com/ipfs/go-ipfs/pull/8680#discussion_r841680182
* style: rename IPFS_RCMGR to LIBP2P_RCMGR
preexisting libp2p toggles use LIBP2P_ prefix
* test: Swarm.ResourceMgr
* fix: location of opt-in limit.json and rcmgr.json.gz
Places these files inside of IPFS_PATH
* Update docs/config.md
* feat: expose rcmgr metrics when enabled (#8785)
* add metrics for the resource manager
* export protocol and service name in Prometheus metrics
* fix: expose rcmgr metrics only when enabled
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* refactor: rcmgr_metrics.go
* refactor: rcmgr_defaults.go
This file defines implicit limit defaults used when Swarm.ResourceMgr.Enabled
We keep vendored copy to ensure go-ipfs is not impacted when go-libp2p
decides to change defaults in any of the future releases.
* refactor: adjustedDefaultLimits
Cleans up the way we initialize defaults and adds a fix for case
when connection manager runs with high limits.
It also hides `Swarm.ResourceMgr.Limits` until we have a better
understanding what syntax makes sense.
* chore: cleanup after a review
* fix: restore go-ipld-prime v0.14.2
* fix: restore go-ds-flatfs v0.5.1
Co-authored-by: Lucas Molas <schomatis@gmail.com>
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* feat: add support for carv2 import
* update to multicodec 0.4.0
* add sharness test for carv2
Co-authored-by: Keenan Nemetz <keenan.nemetz@gmail.com>
- Document IPFS_LOGGING deprecation
as alias to GOLOG_LOG_LEVEL and expand doc
to include per-subsystem log levels.
- Document IPFS_LOGGING_FMT deprecation
as alias to GOLOG_LOG_FMT.
* add deprecation warning when tracer plugins are loaded
* add response format attribute to span in gateway handler
* add note about tracing's experimental status in godoc
* add nil check for TTL when adding name span attrs
* add basic sharness test for integration with otel collector
* add nil check in UnixFSAPI.processLink
* test: sharness check all json objs for swarm span
* add env var docs to docs/environment-variables.md
* chore: pin the otel collector version
* add tracing spans per response type (#8841)
* docs: tracing with jaeger-ui
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* fix(gw): validate requested CAR version
This adds validation of 'application/vnd.ipld.car;version=n' passed
in the Accept header by HTTP clients to align Gateway behavior with
the spec submitted to IANA.
* test: fix comment in test/sharness/t0118-gateway-car.sh
Co-authored-by: Gus Eggert <gus@gus.dev>
Co-authored-by: Gus Eggert <gus@gus.dev>
The Docker sharness tests were disabled years ago when go-ipfs moved
from Travis to CircleCI. This makes the tweaks necessary to re-enable
them.
The Docker image has since moved to be based on BusyBox which doesn't
have the requisite wget version for the existing tests to work, so
this adds some functionality to the pollEndpoint program to support
polling HTTP endpoints as well.
This test is actually testing is the ipfs binary (from the path).
Using the go-ipfs-http-client code.
But most PR only update the binary. HOWEVER Golang does not check PATH for caching.
So as far as golang knows: go-ipfs-http-client is old, it havn't changed, and it doesn't run it.
That leads us to this test not running on most important PR.
* feat(cmds): add deprecated and experimental status
Added programmatic state annotation introduced in
https://github.com/ipfs/go-ipfs-cmds/pull/225
for already deprecated / experimental commands.
* chore: go-ipfs-cmds v0.7.0
Co-authored-by: Lucas Molas <schomatis@gmail.com>
* feat(gw): response type histogram metrics
- response-type agnostic firstContentBlockGetMetric which counts the
latency til the first content block.
- car/block/file/gen-dir-index duration histogram metrics that show how
long each response type takes
* docs: improve metrics descriptions
* feat: more gw histogram buckets
0.05, 0.1, 0.25, 0.5, 1, 2, 5, 10, 30, 60 secs
as suggested in reviews at https://github.com/ipfs/go-ipfs/pull/8443
Co-authored-by: Marcin Rataj <lidel@lidel.org>
Co-authored-by: Gus Eggert <gus@gus.dev>
Bubble ErrNotFound improvements.
The is a really nice UX improvement because it know tell you which ipld block is missing instead of a generic "not found".
* feat: serveRawBlock implements ?format=block
* feat: serveCar implements ?format=car
* feat(gw): ?format= or Accept HTTP header
- extracted file-like content type responses to separate .go files
- Accept HTTP header with support for application/vnd.ipld.* types
* fix: use .bin for raw block content-disposition
.raw may be handled by something, depending on OS, and .bin
seems to be universally "binary file" across all systems:
https://en.wikipedia.org/wiki/List_of_filename_extensions_(A%E2%80%93E)
* refactor: gateway_handler_unixfs.go
- Moved UnixFS response handling to gateway_handler_unixfs*.go files.
- Removed support for X-Ipfs-Gateway-Prefix (Closes#7702)
* refactor: prefix cleanup and readable paths
- removed dead code after X-Ipfs-Gateway-Prefix is gone
(https://github.com/ipfs/go-ipfs/issues/7702)
- escaped special characters in content paths returned with http.Error
making them both safer and easier to reason about (e.g. when invisible
whitespace Unicode is used)
* fix: add companion ids to allow origins
- fixes#8689
- Adds the chrome-extension ids for ipfs-companion and
ipfs-companion-beta to the allowed origins list, this
allows us to accesss ipfs api from a manifest v3 extension.
- added tests in t0401-api-browser-security.sh
* fix: companion when custom CORS *-Origin is set
Companion extension should be able to access RPC API even when custom
Access-Control-Allow-Origin is set
Co-authored-by: Marcin Rataj <lidel@lidel.org>
