* add deprecation warning when tracer plugins are loaded
* add response format attribute to span in gateway handler
* add note about tracing's experimental status in godoc
* add nil check for TTL when adding name span attrs
* add basic sharness test for integration with otel collector
* add nil check in UnixFSAPI.processLink
* test: sharness check all json objs for swarm span
* add env var docs to docs/environment-variables.md
* chore: pin the otel collector version
* add tracing spans per response type (#8841)
* docs: tracing with jaeger-ui
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* fix(gw): validate requested CAR version
This adds validation of 'application/vnd.ipld.car;version=n' passed
in the Accept header by HTTP clients to align Gateway behavior with
the spec submitted to IANA.
* test: fix comment in test/sharness/t0118-gateway-car.sh
Co-authored-by: Gus Eggert <gus@gus.dev>
Co-authored-by: Gus Eggert <gus@gus.dev>
* feat(cmds): add deprecated and experimental status
Added programmatic state annotation introduced in
https://github.com/ipfs/go-ipfs-cmds/pull/225
for already deprecated / experimental commands.
* chore: go-ipfs-cmds v0.7.0
Co-authored-by: Lucas Molas <schomatis@gmail.com>
* feat(gw): response type histogram metrics
- response-type agnostic firstContentBlockGetMetric which counts the
latency til the first content block.
- car/block/file/gen-dir-index duration histogram metrics that show how
long each response type takes
* docs: improve metrics descriptions
* feat: more gw histogram buckets
0.05, 0.1, 0.25, 0.5, 1, 2, 5, 10, 30, 60 secs
as suggested in reviews at https://github.com/ipfs/go-ipfs/pull/8443
Co-authored-by: Marcin Rataj <lidel@lidel.org>
Co-authored-by: Gus Eggert <gus@gus.dev>
Bubble ErrNotFound improvements.
The is a really nice UX improvement because it know tell you which ipld block is missing instead of a generic "not found".
* feat: serveRawBlock implements ?format=block
* feat: serveCar implements ?format=car
* feat(gw): ?format= or Accept HTTP header
- extracted file-like content type responses to separate .go files
- Accept HTTP header with support for application/vnd.ipld.* types
* fix: use .bin for raw block content-disposition
.raw may be handled by something, depending on OS, and .bin
seems to be universally "binary file" across all systems:
https://en.wikipedia.org/wiki/List_of_filename_extensions_(A%E2%80%93E)
* refactor: gateway_handler_unixfs.go
- Moved UnixFS response handling to gateway_handler_unixfs*.go files.
- Removed support for X-Ipfs-Gateway-Prefix (Closes#7702)
* refactor: prefix cleanup and readable paths
- removed dead code after X-Ipfs-Gateway-Prefix is gone
(https://github.com/ipfs/go-ipfs/issues/7702)
- escaped special characters in content paths returned with http.Error
making them both safer and easier to reason about (e.g. when invisible
whitespace Unicode is used)
* fix: add companion ids to allow origins
- fixes#8689
- Adds the chrome-extension ids for ipfs-companion and
ipfs-companion-beta to the allowed origins list, this
allows us to accesss ipfs api from a manifest v3 extension.
- added tests in t0401-api-browser-security.sh
* fix: companion when custom CORS *-Origin is set
Companion extension should be able to access RPC API even when custom
Access-Control-Allow-Origin is set
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* feat(cmd): add silent option repo gc command
closes#7129
* test(cmd): add test case for silent option for command repo gc
* fix: no emit on server with --silent
This removes unnecessary send to the client that does not care
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* feat: add warning message when the local node is offline for remote pinning
* feat: check is node is online
* feat: add test in t0700-remotepin.sh
* feat: use grep and fix the test
* fix: print to stdout only when --enc=text
This ensures we don't break JSON produced by --enc=json
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* chore(gateway): better logging for the http requests
* chore(gateway): removed defer and add more data to the final log
* chore(gateway): debug logging refactor
* chore(gateway): use debug w/o context when only msg
* doc: add cmd for log level
* chore: add more logs and address fedback
* chore(gateway): log subdomains and from=requestURI, refactor
* chore(gateway): fix debug redirect
* feat: use Swarm.EnableHolePunching flag within libp2p
* docs: Swarm.EnableHolePunching
Co-authored-by: Marcin Rataj <lidel@lidel.org>
Co-authored-by: Adin Schmahmann <adin.schmahmann@gmail.com>
* multibase encoding on pubsub
* emit multibase for json clients
* refactor(pubsub): base64url for all URL args
This makes it easier to reason about.
Also added better helptext to each command explaining how the binary
data is encoded on the wire, and how to process it in userland.
* refactor: remove ndpayload and lenpayload
Those output formats are undocumented and seem to be only used in tests.
This change removes their implementation and replaces it with error
message to use JSON instead.
I also refactored tests to test the --enc=json response format instead
of imaginary one, making tests more useful as they also act as
regression tests for HTTP RPC.
* test(pubsub): go-ipfs-api
Testing against compatible version from
https://github.com/ipfs/go-ipfs-api/pull/255
* refactor: safeTextListEncoder
Making it clear what it does and why
* refactor(pubsub): unify peerids
This ensures `ipfs pubsub sub` returns the same peerids in the `From`
field as `ipfs pubsub peers`.
libp2p already uses base encoding, no need to double wrap or use custom
multibase.
* test(pubsub): go-ipfs-http-client
* refactor(pubsub): make pub command read from a file
We want to send payload in the body as multipart so users can use
existing tools like curl for publishing arbitrary bytes to a topic.
StringArg was created for "one message per line" use case, and if data
has `\n` or `\r\n` byte sequences, it will cause payload to be split. It
is not possible to undo this, because mentioned sequences are lost, so
we are not able to tell if it was `\n` or `\r\n`
We already avoid this problem in `block put` and `dht put` by reading
payload via FileArg which does not mangle binary data and send it as-is.
It feel like `pubsub pub` should be using it in the first place anyway,
so this commit replaces StringArg with FileArg.
This also closes https://github.com/ipfs/go-ipfs/issues/8454
and makes rpc in go-ipfs easier to code against.
* test(pubsub): publishing with line breaks
Making sure we don't see regressions in the future.
Ref. https://github.com/ipfs/go-ipfs/issues/7939
* chore: disable pubsub interop for now
See
344f692d8c
* test: t0322-pubsub-http-rpc.sh
- Adds HTTP RPC regression test that ensures topic is encoded as URL-safe
multibase.
- Moves pubsub tests to live in unique range ./t032x
* fix(ci): js-ipfs with fixed pubsub wire format
uses js-ipfs from https://github.com/ipfs/js-ipfs/pull/3922
until js-ipfs release can ship with dependency on go-ipfs 0.11.0-rc1
Co-authored-by: Marcin Rataj <lidel@lidel.org>
Co-authored-by: Adin Schmahmann <adin.schmahmann@gmail.com>
* plumb through go-datastore context changes
* update go-libp2p to v0.16.0
* use LIBP2P_TCP_REUSEPORT instead of IPFS_REUSEPORT
* use relay config
* making deprecation notice match the go-ipfs-config key
* docs(config): circuit relay v2
* docs(config): fix links and headers
* feat(config): Internal.Libp2pForceReachability
This switches to config that supports setting and reading
Internal.Libp2pForceReachability OptionalString flag
* use configuration option for static relays
* chore: go-ipfs-config v0.18.0
https://github.com/ipfs/go-ipfs-config/releases/tag/v0.18.0
* feat: circuit v1 migration prompt when Swarm.EnableRelayHop is set (#8559)
* exit when Swarm.EnableRelayHop is set
* docs: Experimental.ShardingEnabled migration
This ensures existing users of global sharding experiment get notified
that the flag no longer works + that autosharding happens automatically.
For people who NEED to keep the old behavior (eg. have no time to
migrate today) there is a note about restoring it with
`UnixFSShardingSizeThreshold`.
* chore: add dag-jose code to the cid command output
* add support for setting automatic unixfs sharding threshold from the config
* test: have tests use low cutoff for sharding to mimic old behavior
* test: change error message to match the current error
* test: Add automatic sharding/unsharding tests (#8547)
* test: refactored naming in the sharding sharness tests to make more sense
* ci: set interop test executor to convenience image for Go1.16 + Node
* ci: use interop master
Co-authored-by: Marcin Rataj <lidel@lidel.org>
Co-authored-by: Marten Seemann <martenseemann@gmail.com>
Co-authored-by: Marcin Rataj <lidel@lidel.org>
Co-authored-by: Gus Eggert <gus@gus.dev>
Co-authored-by: Lucas Molas <schomatis@gmail.com>
* chore: replace go-merkledag walk with go-ipld-prime traversal for dag export
This is "safe" now because we can limit duplicate block loads like
go-merkledag does and won't get trapped taking a long time for complex
DAGs. We can do this while we're using an exhaustive selector (like
ExploreAll here) but will need an alternative strategy when we go for
arbitrary selectors.
Cannot reproduce the flakiness at the moment. The report suggests that connections are established on different transports. Adding logging to show what these transports are.
* fix(cli): object add-link: do not allow blocks over BS limit
* refactor: allow-big-block
- renamed override flag to --allow-big-block
- separate tests for default and override behavior
Co-authored-by: Marcin Rataj <lidel@lidel.org>
