This fixes a regression around directory listing and index.html hosting.
Seems that during one of recent refactors code changed and we no longer
check for trailing slash in HTTP request path, but look at content path
instead.
This cleans this up and also ensures dir behavior is the same for
both index.html hosting and dir-index-html (generated listing).
It also adds more tests so we catch any future regressions.
This fixes a regression introduced in 0.13.0, where websites hosted via
index.html placed in UnixFS directory were always returned with
Cache-Control: public, max-age=29030400, immutable
even when loaded from mutable /ipns/ contentPath.
This fix safelists additional headers allowing JS running on websites to
read them when IPFS resource is downloaded via Fetch API.
These headers provide metadata necessary for making smart caching
decisions when IPFS resources are downloaded via Service Worker or a
similar middleware on the edge.
* fix(core/gateway): option to limit directory size listing
* feat(gw): HTMLDirListingLimit
This is alternative take on the way we limit the HTML listing output.
Instead of a hard cut-off, we list up to HTMLDirListingLimit.
When a directory has more items than HTMLDirListingLimit we show
additional header and footer informing user that only $HTMLDirListingLimit
items are listed. This is a better UX.
* fix: 0 disables Gateway.HTMLDirListingLimit
* refactor: Gateway.FastDirIndexThreshold
see explainer in docs/config.md
* refactor: prealoc slices
* docs: Gateway.FastDirIndexThreshold
* refactor: core/corehttp/gateway_handler.go
https://github.com/ipfs/go-ipfs/pull/8853#discussion_r851437088
* docs: apply suggestions from code review
Co-authored-by: Alan Shaw <alan.shaw@protocol.ai>
Co-authored-by: Marcin Rataj <lidel@lidel.org>
Co-authored-by: Alan Shaw <alan.shaw@protocol.ai>
Improves the way we handle If-None-Match header:
- Support for more than one Etag passed in If-None-Match
- Match both strong and weak Etags to maximize caching across
various HTTP clients and libraries (some send weak Etags by default)
- Support for wildcard '*'
- Tests for If-None-Match behavior
* Extract functions from getOrHeadHandler to improve readability and prepare for later refactorings
* Address PR feedback on when to return errors or booleans
* Be explicit about use of *requestError vs error
Since go1.16, there are built in tools that allow for embeding
filesystem inside the binary. We now make use of the `embed` package to
have all files put into the binary, removing the need to generate the
files and removes dependencies
Co-authored-by: Jorropo <jorropo.pgm@gmail.com>
* fix: report gateway http metrics only when response is successful
* fix(gw): 304 Not Modified as no-op
This fix ensures we don't do any additional work when Etag match
what user already has in their own cache.
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* add deprecation warning when tracer plugins are loaded
* add response format attribute to span in gateway handler
* add note about tracing's experimental status in godoc
* add nil check for TTL when adding name span attrs
* add basic sharness test for integration with otel collector
* add nil check in UnixFSAPI.processLink
* test: sharness check all json objs for swarm span
* add env var docs to docs/environment-variables.md
* chore: pin the otel collector version
* add tracing spans per response type (#8841)
* docs: tracing with jaeger-ui
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* fix(gw): validate requested CAR version
This adds validation of 'application/vnd.ipld.car;version=n' passed
in the Accept header by HTTP clients to align Gateway behavior with
the spec submitted to IANA.
* test: fix comment in test/sharness/t0118-gateway-car.sh
Co-authored-by: Gus Eggert <gus@gus.dev>
Co-authored-by: Gus Eggert <gus@gus.dev>
* feat(gw): response type histogram metrics
- response-type agnostic firstContentBlockGetMetric which counts the
latency til the first content block.
- car/block/file/gen-dir-index duration histogram metrics that show how
long each response type takes
* docs: improve metrics descriptions
* feat: more gw histogram buckets
0.05, 0.1, 0.25, 0.5, 1, 2, 5, 10, 30, 60 secs
as suggested in reviews at https://github.com/ipfs/go-ipfs/pull/8443
Co-authored-by: Marcin Rataj <lidel@lidel.org>
Co-authored-by: Gus Eggert <gus@gus.dev>
* feat: serveRawBlock implements ?format=block
* feat: serveCar implements ?format=car
* feat(gw): ?format= or Accept HTTP header
- extracted file-like content type responses to separate .go files
- Accept HTTP header with support for application/vnd.ipld.* types
* fix: use .bin for raw block content-disposition
.raw may be handled by something, depending on OS, and .bin
seems to be universally "binary file" across all systems:
https://en.wikipedia.org/wiki/List_of_filename_extensions_(A%E2%80%93E)
* refactor: gateway_handler_unixfs.go
- Moved UnixFS response handling to gateway_handler_unixfs*.go files.
- Removed support for X-Ipfs-Gateway-Prefix (Closes#7702)
* refactor: prefix cleanup and readable paths
- removed dead code after X-Ipfs-Gateway-Prefix is gone
(https://github.com/ipfs/go-ipfs/issues/7702)
- escaped special characters in content paths returned with http.Error
making them both safer and easier to reason about (e.g. when invisible
whitespace Unicode is used)
* fix: add companion ids to allow origins
- fixes#8689
- Adds the chrome-extension ids for ipfs-companion and
ipfs-companion-beta to the allowed origins list, this
allows us to accesss ipfs api from a manifest v3 extension.
- added tests in t0401-api-browser-security.sh
* fix: companion when custom CORS *-Origin is set
Companion extension should be able to access RPC API even when custom
Access-Control-Allow-Origin is set
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* chore(gateway): better logging for the http requests
* chore(gateway): removed defer and add more data to the final log
* chore(gateway): debug logging refactor
* chore(gateway): use debug w/o context when only msg
* doc: add cmd for log level
* chore: add more logs and address fedback
* chore(gateway): log subdomains and from=requestURI, refactor
* chore(gateway): fix debug redirect
