* fix: report gateway http metrics only when response is successful
* fix(gw): 304 Not Modified as no-op
This fix ensures we don't do any additional work when Etag match
what user already has in their own cache.
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* add deprecation warning when tracer plugins are loaded
* add response format attribute to span in gateway handler
* add note about tracing's experimental status in godoc
* add nil check for TTL when adding name span attrs
* add basic sharness test for integration with otel collector
* add nil check in UnixFSAPI.processLink
* test: sharness check all json objs for swarm span
* add env var docs to docs/environment-variables.md
* chore: pin the otel collector version
* add tracing spans per response type (#8841)
* docs: tracing with jaeger-ui
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* fix(gw): validate requested CAR version
This adds validation of 'application/vnd.ipld.car;version=n' passed
in the Accept header by HTTP clients to align Gateway behavior with
the spec submitted to IANA.
* test: fix comment in test/sharness/t0118-gateway-car.sh
Co-authored-by: Gus Eggert <gus@gus.dev>
Co-authored-by: Gus Eggert <gus@gus.dev>
* feat(gw): response type histogram metrics
- response-type agnostic firstContentBlockGetMetric which counts the
latency til the first content block.
- car/block/file/gen-dir-index duration histogram metrics that show how
long each response type takes
* docs: improve metrics descriptions
* feat: more gw histogram buckets
0.05, 0.1, 0.25, 0.5, 1, 2, 5, 10, 30, 60 secs
as suggested in reviews at https://github.com/ipfs/go-ipfs/pull/8443
Co-authored-by: Marcin Rataj <lidel@lidel.org>
Co-authored-by: Gus Eggert <gus@gus.dev>
* feat: serveRawBlock implements ?format=block
* feat: serveCar implements ?format=car
* feat(gw): ?format= or Accept HTTP header
- extracted file-like content type responses to separate .go files
- Accept HTTP header with support for application/vnd.ipld.* types
* fix: use .bin for raw block content-disposition
.raw may be handled by something, depending on OS, and .bin
seems to be universally "binary file" across all systems:
https://en.wikipedia.org/wiki/List_of_filename_extensions_(A%E2%80%93E)
* refactor: gateway_handler_unixfs.go
- Moved UnixFS response handling to gateway_handler_unixfs*.go files.
- Removed support for X-Ipfs-Gateway-Prefix (Closes#7702)
* refactor: prefix cleanup and readable paths
- removed dead code after X-Ipfs-Gateway-Prefix is gone
(https://github.com/ipfs/go-ipfs/issues/7702)
- escaped special characters in content paths returned with http.Error
making them both safer and easier to reason about (e.g. when invisible
whitespace Unicode is used)
* fix: add companion ids to allow origins
- fixes#8689
- Adds the chrome-extension ids for ipfs-companion and
ipfs-companion-beta to the allowed origins list, this
allows us to accesss ipfs api from a manifest v3 extension.
- added tests in t0401-api-browser-security.sh
* fix: companion when custom CORS *-Origin is set
Companion extension should be able to access RPC API even when custom
Access-Control-Allow-Origin is set
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* chore(gateway): better logging for the http requests
* chore(gateway): removed defer and add more data to the final log
* chore(gateway): debug logging refactor
* chore(gateway): use debug w/o context when only msg
* doc: add cmd for log level
* chore: add more logs and address fedback
* chore(gateway): log subdomains and from=requestURI, refactor
* chore(gateway): fix debug redirect
* plumb through go-datastore context changes
* update go-libp2p to v0.16.0
* use LIBP2P_TCP_REUSEPORT instead of IPFS_REUSEPORT
* use relay config
* making deprecation notice match the go-ipfs-config key
* docs(config): circuit relay v2
* docs(config): fix links and headers
* feat(config): Internal.Libp2pForceReachability
This switches to config that supports setting and reading
Internal.Libp2pForceReachability OptionalString flag
* use configuration option for static relays
* chore: go-ipfs-config v0.18.0
https://github.com/ipfs/go-ipfs-config/releases/tag/v0.18.0
* feat: circuit v1 migration prompt when Swarm.EnableRelayHop is set (#8559)
* exit when Swarm.EnableRelayHop is set
* docs: Experimental.ShardingEnabled migration
This ensures existing users of global sharding experiment get notified
that the flag no longer works + that autosharding happens automatically.
For people who NEED to keep the old behavior (eg. have no time to
migrate today) there is a note about restoring it with
`UnixFSShardingSizeThreshold`.
* chore: add dag-jose code to the cid command output
* add support for setting automatic unixfs sharding threshold from the config
* test: have tests use low cutoff for sharding to mimic old behavior
* test: change error message to match the current error
* test: Add automatic sharding/unsharding tests (#8547)
* test: refactored naming in the sharding sharness tests to make more sense
* ci: set interop test executor to convenience image for Go1.16 + Node
* ci: use interop master
Co-authored-by: Marcin Rataj <lidel@lidel.org>
Co-authored-by: Marten Seemann <martenseemann@gmail.com>
Co-authored-by: Marcin Rataj <lidel@lidel.org>
Co-authored-by: Gus Eggert <gus@gus.dev>
Co-authored-by: Lucas Molas <schomatis@gmail.com>
Cannot reproduce the flakiness at the moment. The report suggests that connections are established on different transports. Adding logging to show what these transports are.
I believe we figured that these were for "informational purposes", but
really, we _should_ always be able to resolve names to CIDs. If we
can't, there's probably something wrong with the directory.
Namesys is a very useful submodule. Given a ValueStore and a Datastore it can
resolve and publish /ipns/ paths.
This functionality does not need to be sequestered inside go-ipfs as it can
and should be used without IPFS, for example, for implementing lightweight
IPNS publishing services or for resolving /ipns/ paths.
"keystore" extraction was necessary, as there is a dependency to it in
namesys. Keystore is also a useful module by itself within the stack.
Fixes#6537
We used Clear-Site-Data to cushion transition period for local gateway
exposed at http://localhost while we were still figuring out
security-related details.
In the final implementation subdomain gateways are not tied to a
hostname explicitly, which removes the risk of cookies leaking,
removing the need for the header.
Turns out it causes issues for Firefox users, so let's just remove it.
Closes https://github.com/ipfs-shipyard/ipfs-companion/issues/977
This adds localhost ipv6 addresses to the allowlist for use in browser
context and fixes WebUI on ipv6-only deployments: http://[::1]:5001/webui
We were missing CORS/Origin tests for API port so I've added basic ones
and included localhost/127.0.0.1/::1 variants.
This adds proper end-to-end tests for directory listing on Gateway port
that protects us against regressions oni each gw type:
- path gateway
- subdomain gateway
- dnslink website gateway
Tests cover:
- etag/unicode support
- breadcrumbs
- file name column
- hash column
This implements 'attachment' mode triggered then
?filename parameter is accompanied with &download=true
When Content-Disposition: attachment is detected by a modern browser
it will skip rendering and immediately open the "save as" dialog,
making this useful feature for using IPFS gateway as target of
"Download" links on various websites.
Parameter name was suggested in:
https://github.com/ipfs/go-ipfs/pull/4177#issuecomment-414870327
These changes are needed to prepare for the Directory page UI improvements
implemented in https://github.com/ipfs/dir-index-html/issues/37.
- update dir-index-html type structs
- emit gateway URL for root links
- emit CID of each directoryItem
- emit size of directory
- emit breadcrumbs
This ensures implicit defaults are always present, even when
Gateway.PublicGateways is defined in the config.
User still can disable them, but needs to do it per hostname.
License: MIT
Signed-off-by: Marcin Rataj <lidel@lidel.org>
Add support for one or more wildcards in the hostname definition
of a public gateway. This is useful for example to support easily
multiples environment.
Wildcarded hostname are set in the config as for example "*.domain.tld".
While the content of raw files retrieved via the gateway should never
change, the look and feel of the directory index can and will change
between versions of go-ipfs.
Incorporate the hash of assets/bindata.go into the ETag when appropriate
fixes https://github.com/ipfs/go-ipfs/pull/4233#issuecomment-631454543
Basically, there's a trade-off here:
1. We can support directory listings while supporting 404 pages (this PR).
2. If a 404 page is present, directory listings don't work.
Given that option 1 is more flexible and users shouldn't be _too_ confused if
they land on a directory with no index.html page, I've gone with that option.
Files already have an explicit Content-Type set. Be sure to do this for
directory listings as well to avoid a fallback to autodetection in
net/http. That fallback fails when a ResponseWriter is installed that
performs compression.
In the same way that an `index.html` file is rendered, if one is present, when the
requested path is a directory, now an `ipfs-404.html` file is rendered if
the requested file is not present within the specified IPFS object.
`ipfs-404.html` files are looked for in the directory of the requested path and each
parent until one is found, falling back on the well-known 404 error message.
License: MIT
Signed-off-by: JP Hastings-Spital <jphastings@gmail.com>
This commit upgrades go-ipfs-cmds and configures the commands HTTP API Handler
to only allow POST/OPTIONS, disallowing GET and others in the handling of
command requests in the IPFS HTTP API (where before every type of request
method was handled, with GET/POST/PUT/PATCH being equivalent).
The Read-Only commands that the HTTP API attaches to the gateway endpoint will
additional handled GET as they did before (but stop handling PUT,DELETEs).
By limiting the request types we address the possibility that a website
accessed by a browser abuses the IPFS API by issuing GET requests to it which
have no Origin or Referrer set, and are thus bypass CORS and CSRF protections.
This is a breaking change for clients that relay on GET requests against the
HTTP endpoint (usually :5001). Applications integrating on top of the
gateway-read-only API should still work (including cross-domain access).
Co-Authored-By: Steven Allen <steven@stebalien.com>
Co-Authored-By: Marcin Rataj <lidel@lidel.org>
When request is sent to http://localhost:8080/ipfs/$cid response has
HTTP 301 status code and "Location" header with redirect destination at
$cid.ipfs.localhost:8080
Redirect is followed by browsersi, but not by commandline tools.
Status 301 is ignored by curl in default mode: it will print response
and won't follow redirect, user needs to add -L for that.
To fix curl, we return correct payload in body of HTTP 301 response,
but set Clear-Site-Data header to ensure Origin sandbox can't be abused.
This requires a surgical workaround:
If Location header is present in ResponseWriter's Header map,
we ensure http.ServeContent() returns HTTP 301
Context: https://github.com/ipfs/go-ipfs/pull/6982
License: MIT
Signed-off-by: Marcin Rataj <lidel@lidel.org>
Instead of adding a new fake header (that could be spoofed by the client...),
just read the original request URI from the request object.
This also removes support for suborigins. They have never been implemented in
browsers and it looks like efforts have stalled. We can add support back if we
need it but, well, maintaining support was going to be more trouble than it was
worth.
License: MIT
Signed-off-by: Steven Allen <steven@stebalien.com>
We've deprecated this system and have yet to move to a new system. We might as
well remove everything, switch to a new system, then deliberately trace the
entire system.
1. Require files to have known sizes. We can add support for unknown sizes
_later_ but we can't use ServeContent for those files.
2. Replace the `sizeReadSeeker` with a `lazySeeker`. This one makes no
assumptions about how it's used so we're less likely to run into weird bugs.
1. Fix handling of PUT. The simple implementation was the correct
implementation, I have no idea what was going on here.
2. Use MFS everywhere to reduce code duplication and add support for sharded
directories.
3. _Correctly_ block IPNS.
4. Remove the dependency on `core.IpfsNode`.
5. Remove support for putting empty directories with a well-known CID. It was
useless as directories are automatically created.
