Commit Graph

3440 Commits

Author SHA1 Message Date
rendaw
a0656402ed
Key import and export cli commands 2020-07-16 22:25:37 +09:00
Petar Maymounkov
b3e5ffc41a
feat: add flag to ipfs key and list to output keys in b36/CIDv1 (#7531)
* add flag to "ipfs key gen" to output keys in b36 CIDv1
* add flag to "ipfs key list" to output keys in b36 CIDv1
* add and modify corresponding sharness tests
2020-07-13 12:08:34 -04:00
Marcin Rataj
231fab811d feat: support ED25519 libp2p-key in subdomains
This:

- adds subdomain gateway support for ED25519 CIDs in a way that fits in
  a single DNS label to enable TLS for every IPNS website.

- cleans up subdomain redirect logic and adds more explicit error
  handling.

TL;DR on router logic:

When CID is longer than 63 characters, router at /ipfs/* and /ipns/*
converts to Base36, and if that does not help, returns a human readable
400 Bad Request error.

Addressing code review:
https://github.com/ipfs/go-ipfs/pull/7441#pullrequestreview-440043209

refactor: use b36 for all libp2p-keys in subdomains
Consensus reached in
https://github.com/ipfs/go-ipfs/pull/7441#discussion_r452372828
https://github.com/ipfs/go-ipfs/pull/7441#discussion_r451477890
https://github.com/ipfs/go-ipfs/pull/7441#discussion_r452500272
2020-07-10 16:25:20 -04:00
Michael Muré
87dfc46e03 feat: support X-Forwarded-Host when doing gateway redirect 2020-07-10 14:13:11 -04:00
Martín Triay
c532cb4d5a
Document add behavior when the daemon is not running 2020-06-27 15:57:18 -03:00
Rafael Ramalho
4c38ea748a chore:bump webui version to 2.10.1 2020-06-23 16:28:38 +01:00
Rafael Ramalho
cc4a136360 chore: bump webui version 2020-06-22 17:39:30 +01:00
Adin Schmahmann
19d6230cde Merge branch 'release' into master 2020-06-19 20:04:30 -04:00
Steven Allen
84341d0c5a fix: use the correct context when resolving dnsaddr links 2020-06-17 19:44:28 -07:00
Steven Allen
5f19f3bbe8 fix: cancel resolve search context
We probably don't need to do this, but it can't hurt.
2020-06-17 19:43:20 -07:00
Steven Allen
3ce725dd96 fix: upgrade warning to error 2020-06-16 15:15:51 -07:00
Steven Allen
e164af1f2a feat: add a transport config section
This way, users can disable transports (especially QUIC), and set muxer/security
transport priorities.
2020-06-16 15:15:47 -07:00
Steven Allen
cee60700b9 fix: use bitswap sessions for ipfs refs
This isn't perfect (we only use sessions after resolving the root cid) but it's
better than what we have. The real solution is #7198 so we can use sessions
everywhere.

(cherry picked from commit 62f61c588d)
2020-06-02 17:36:30 -07:00
Marcin Rataj
5be7e0f8de feat: webui v2.9.0
(cherry picked from commit 0f76ed79f9)
2020-06-02 17:36:30 -07:00
Shotaro Yamada
bd9382fe41 chore: update WebUI to 2.8.0
(cherry picked from commit 58aac04a53)
2020-06-02 17:36:30 -07:00
Steven Allen
0cdde03824
Merge pull request #7389 from ipfs/fix/refs-sessions
fix: use bitswap sessions for ipfs refs
2020-06-02 11:17:12 -07:00
Steven Allen
22dc1ad397
Merge pull request #7397 from ipfs/fix/id-addr-format
fix(commands): print consistent addresses in ipfs id
2020-06-02 10:46:28 -07:00
Steven Allen
3aa86f8ed9 fix(commands): print consistent addresses in ipfs id
Consistently append `/p2p/QmMyId` to addresses when calling `ipfs id Me` and `ipfs id NotMe`.

Fixes https://github.com/ipfs/go-ipfs/issues/7378.
2020-05-29 18:48:53 -07:00
Steven Allen
c58e3e4c7d fix: remove pubsub discovery hack
Pubsub handles this internally now.
2020-05-29 16:42:10 -07:00
Steven Allen
ac8a88d054 fix(pubsub): flood publish
When publishing a value, flood. This ensures that our messages make it to the
network even if some of our peers are bad and/or overloaded.
2020-05-29 16:42:10 -07:00
Steven Allen
62f61c588d fix: use bitswap sessions for ipfs refs
This isn't perfect (we only use sessions after resolving the root cid) but it's
better than what we have. The real solution is #7198 so we can use sessions
everywhere.
2020-05-28 17:15:56 -07:00
Marcin Rataj
0f76ed79f9
feat: webui v2.9.0 2020-05-28 23:04:35 +02:00
Shotaro Yamada
58aac04a53 chore: update WebUI to 2.8.0 2020-05-28 10:53:25 +09:00
Steven Allen
d48b80b30e feat: add noise support
We can add a proper configuration option in a followup patch after the RC.
2020-05-25 23:46:48 -07:00
Steven Allen
978091a626 feat: implement peering service
MVP for #6097

This feature will repeatedly reconnect (with a randomized exponential backoff)
to peers in a set of "peered" peers.

In the future, this should be extended to:

1. Include a CLI for modifying this list at runtime.
2. Include additional options for peers we want to _protect_ but not connect to.
3. Allow configuring timeouts, backoff, etc.
4. Allow groups? Possibly through textile threads.
5. Allow for runtime-only peering rules.
6. Different reconnect policies.

But this MVP should be a significant step forward.
2020-05-25 20:24:41 -07:00
Steven Allen
707b5506ba
Merge pull request #7360 from ipfs/feat/better_dir_etag
Include the git blob id of the dir-index bundle in the ETag
2020-05-25 13:21:55 -07:00
Steven Allen
c17bcbd4e0
Merge pull request #7340 from ipfs/feat/boostrap-when-low
feat: bootstrap in dht when the routing table is empty
2020-05-25 13:01:08 -07:00
Peter Rabbitson
2d5f8b4ebe Include the git blob id of the dir-index bundle in the ETag
While the content of raw files retrieved via the gateway should never
change, the look and feel of the directory index can and will change
between versions of go-ipfs.

Incorporate the hash of assets/bindata.go into the ETag when appropriate
2020-05-25 18:42:07 +00:00
Steven Allen
ff17485cbb feat: bootstrap in dht when the routing table is empty
Otherwise, we could end up with only DHT clients and never re-bootstrap.

I've left the default go-ipfs bootstrapping code in for now as it's technically
possible to disable the DHT entirely.
2020-05-24 22:55:30 -07:00
Steven Allen
4714c0cecc feat: enable QUIC by default
But only if private networks are disabled.
2020-05-22 13:48:58 -07:00
Steven Allen
03c1f769d7 fix: remove strict signature verification option
Signature verification is now always strict.
2020-05-22 13:48:26 -07:00
@RubenKelevra
2dc1f691f1 quic: remove experimental status and add it to the default config 2020-05-22 04:54:19 +02:00
Steven Allen
6a2fe0a20d fix: support directory listings even if a 404 page is present
fixes https://github.com/ipfs/go-ipfs/pull/4233#issuecomment-631454543

Basically, there's a trade-off here:

1. We can support directory listings while supporting 404 pages (this PR).
2. If a 404 page is present, directory listings don't work.

Given that option 1 is more flexible and users shouldn't be _too_ confused if
they land on a directory with no index.html page, I've gone with that option.
2020-05-20 19:11:50 -07:00
Steven Allen
0a8b2f117b chore: update deps
And remove use of mafilter library (now rolled into go-multiaddr).
2020-05-20 17:51:04 -07:00
Steven Allen
febdc3bed6
Merge pull request #7335 from ipfs/chore/update-libp2p
chore: update libp2p
2020-05-20 17:42:15 -07:00
Steven Allen
69ccf99915 chore: update libp2p 2020-05-20 16:56:22 -07:00
Peter Wu
d4952f2a73 fix(gateway): ensure directory listings have Content-Type text/html
Files already have an explicit Content-Type set. Be sure to do this for
directory listings as well to avoid a fallback to autodetection in
net/http. That fallback fails when a ResponseWriter is installed that
performs compression.
2020-05-20 15:20:33 +02:00
Peter Wu
7c7888c4db fix(gateway): fix status code for HEAD on redirects
Report a consistent status code for HEAD requests that end up in a
redirect.
2020-05-20 14:03:58 +02:00
JP Hastings-Spital
dfceafdbd3 Gateway renders pretty 404 pages if available
In the same way that an `index.html` file is rendered, if one is present, when the
requested path is a directory, now an `ipfs-404.html` file is rendered if
the requested file is not present within the specified IPFS object.

`ipfs-404.html` files are looked for in the directory of the requested path and each
parent until one is found, falling back on the well-known 404 error message.

License: MIT
Signed-off-by: JP Hastings-Spital <jphastings@gmail.com>
2020-05-19 14:41:21 -07:00
Steven Allen
5751917f49 feat: add a stats dht command
Currently, it just prints out the routing tables.
2020-05-14 23:32:18 -07:00
Steven Allen
3339ce3b0e
Merge pull request #6774 from MichaelMure/pincoreapi
pin: implement pin/ls with only CoreApi
2020-05-05 18:49:19 -07:00
Michael Muré
6388f5a1c2
pin: honor the context more accurately 2020-05-05 14:19:38 +02:00
Michael Muré
6ef190f5a6 pin: implement pin/ls with only CoreApi 2020-05-04 18:02:50 -07:00
Steven Allen
0189e42bcb
Merge pull request #7061 from RubenKelevra/patch-1
README.md: typo
2020-05-04 17:53:52 -07:00
Gowtham G
e93f2002f5 optimize import order 2020-05-02 14:19:01 +05:30
Gowtham G
214d29ebf8 #7252 - print error message 2020-05-01 22:37:20 +05:30
Gowtham G
9201b1dde6 #7252 - read content directly instead of sending 512bytes 2020-05-01 20:29:32 +05:30
Gowtham G
2e87ac88a3 Fixes #7252 - Uses gabriel-vasile/mimetype to support additional content types 2020-05-01 12:19:36 +05:30
RubenKelevra
a048008c5b fix typo in command texts 2020-04-30 20:36:00 +02:00
Steven Allen
46ae021733 feat: show the absolute path every time
Even for dnslink websites.

fixes https://github.com/ipfs/go-ipfs/issues/7205
2020-04-26 17:19:17 -07:00
Steven Allen
d8bc5c991e feat: always show the hash
Previously, we only showed this /ipns paths. However, knowing the hash of the
current directory is useful regardless.
2020-04-26 16:09:02 -07:00
Dominic Della Valle
048adadb5e reformat log helptext (80 char width) 2020-04-24 12:17:00 -04:00
Dominic Della Valle
05d0573e02 update log helptext to match actual levels 2020-04-24 12:13:34 -04:00
Steven Allen
5d13ac370b fix: fix duplicate block issue in bitswap
1. Increase broadcast timeout.
2. Fix use of sessions when pinning.

This also includes some bitswap changes to reduce some memory allocations and
optimize some hot spots.
2020-04-23 00:31:06 -07:00
Steven Allen
9033ba9cad
Merge pull request #7181 from rex4539/fix-typos
Fix typos and cleanup
2020-04-20 21:43:47 -07:00
Dimitris Apostolou
1e437c7e97
Fix typos and cleanup 2020-04-20 22:00:01 +03:00
@RubenKelevra
8a85c0a344 merge conflicts with upstream 2020-04-18 13:11:56 +02:00
Henrique Dias
135c451997
feat: webui 2.7.5 2020-04-17 20:10:21 +01:00
Marcin Rataj
8290ec11c3
fix: subdomain redirect for dir CIDs
Closes #7164

License: MIT
Signed-off-by: Marcin Rataj <lidel@lidel.org>
2020-04-15 23:50:53 +02:00
Steven Allen
1f23fc000c feat: add autonat config options
1. Enable AutoNATService on _all_ nodes by default. If it's an issue, we can
disable it in RC3 but this will give us the best testing results.
2. Expose options to configure AutoNAT rate limiting.
2020-04-14 20:54:58 -07:00
Marcin Rataj
7d397ac219
feat: webui v2.7.4
(supersedes v2.7.3)

License: MIT
Signed-off-by: Marcin Rataj <lidel@lidel.org>
2020-04-14 17:51:25 +02:00
Steven Allen
28e31e17a8
Merge pull request #7127 from ipfs/feat/dual-dht
feat: introduce the dual WAN/LAN DHT
2020-04-13 19:01:12 -07:00
Steven Allen
d0d508b2e1 fix: fix 'dht query' command to actually return the closest peers
Then improve the tests such that they actually work.
2020-04-13 17:42:03 -07:00
Steven Allen
b64d17b924 properly expose dual dht 2020-04-13 15:58:12 -07:00
Steven Allen
8679954f9b chore: update flatfs
And remove retry logic. This was flatfs specific and we've moved the logic down
into flatfs itself.

This update:

* Retries in more cases when we run out of file descriptors.
* Ensures we don't leak temporary files on batch put.
2020-04-10 16:42:29 -07:00
Steven Allen
e55c688fca fix: run tests in "dht server" mode
Otherwise, we'd need to wait a delay on start for nodes to start becoming
servers.
2020-04-09 23:07:27 -07:00
Steven Allen
bb08f7fe56 assign public IP addresses for tests that need them 2020-04-09 21:41:42 -07:00
Steven Allen
8a002250dc feat: introduce the dual WAN/LAN DHT
And only join the WAN DHT if we're publicly reachable.
2020-04-09 21:41:38 -07:00
Peter Rabbitson
fbe40883e6 Remove PresentInImport tracking as per conversation 2020-04-08 14:54:49 -07:00
Peter Rabbitson
90c58d6cac Dag import functionality only ( no progress )
This still works over "loosely defined" .car files
Please refer to the sharness tests for extra info

We can tighten this up if the sentiment is "Postel was wrong"
2020-04-08 14:54:49 -07:00
Peter Rabbitson
1a85fddfeb For now remove export from the gateway whitelist 2020-04-08 14:54:49 -07:00
Peter Rabbitson
4772ca6134 Add simple byte-counting export progress-bar 2020-04-08 14:54:49 -07:00
Peter Rabbitson
ae5a9e0549 Bring back pipe closure from other end 2020-04-08 14:54:49 -07:00
Peter Rabbitson
380bffa146 Move commented out code together 2020-04-08 14:54:49 -07:00
Peter Rabbitson
73bb0e8353 Check against error veriable 2020-04-08 14:54:49 -07:00
Peter Rabbitson
c2f9849a2b Fastify with go-car version taking a session 2020-04-08 14:54:49 -07:00
Peter Rabbitson
f129f24bd7 Bring back error-prettifyer removed in f6b707d85f 2020-04-08 14:54:49 -07:00
Peter Rabbitson
3dca025f05 Remove pipe close since we are already in error 2020-04-08 14:54:49 -07:00
Peter Rabbitson
7a749ea7cb Less confusing CLI helptext 2020-04-08 14:54:49 -07:00
Peter Rabbitson
a5f19c4f9c Remove unnecessay prefetch 2020-04-08 14:54:49 -07:00
Peter Rabbitson
b0a461ca5c Typo fix 2020-04-08 14:54:49 -07:00
Peter Rabbitson
635e333d60 Dag export command, silent, no progress 2020-04-08 14:54:49 -07:00
Steven Allen
ba631bbab0 fix: start the IPFS dht in server mode by default for now 2020-04-06 18:53:16 -07:00
Steven Allen
56e0e18294 feat: add a dhtserver option 2020-04-06 18:53:16 -07:00
Steven Allen
70fe64581d feat(dht): update to cypress DHT in backwards compatibility mode
This upgrades to the next DHT version, but in backwards compatibility mode.
2020-04-06 18:53:14 -07:00
@RubenKelevra
ae5bd6f649 files.go: fix typos / wording 2020-04-06 01:52:08 +02:00
Hector Sanjuan
7340543615 corehttp: Gateway handler: add Allow headers when returning MethodNotAllowed
Spec says that response with 405 must set Allow headers.
2020-04-05 09:57:57 +02:00
Hector Sanjuan
1b490476e5 HTTP API: Disallow GET requests on API
This commit upgrades go-ipfs-cmds and configures the commands HTTP API Handler
to only allow POST/OPTIONS, disallowing GET and others in the handling of
command requests in the IPFS HTTP API (where before every type of request
method was handled, with GET/POST/PUT/PATCH being equivalent).

The Read-Only commands that the HTTP API attaches to the gateway endpoint will
additional handled GET as they did before (but stop handling PUT,DELETEs).

By limiting the request types we address the possibility that a website
accessed by a browser abuses the IPFS API by issuing GET requests to it which
have no Origin or Referrer set, and are thus bypass CORS and CSRF protections.

This is a breaking change for clients that relay on GET requests against the
HTTP endpoint (usually :5001). Applications integrating on top of the
gateway-read-only API should still work (including cross-domain access).

Co-Authored-By: Steven Allen <steven@stebalien.com>
Co-Authored-By: Marcin Rataj <lidel@lidel.org>
2020-04-05 09:57:57 +02:00
@RubenKelevra
7b750664fb
Revert last change (the default is now printed twice) 2020-04-04 05:34:56 +02:00
Hector Sanjuan
3b30e8e2b0 ipfs files cp: improve docstring with better words.
And align capitalization.
2020-04-02 20:46:31 +02:00
Hector Sanjuan
b5aaf75e25 Fix #4996: Improve help text for "ipfs files cp" 2020-04-01 12:08:48 +02:00
Steven Allen
06ba6d0fc2 fix: remove internal relay discovery
This logic collects a list of known relays by testing every new connection.

It exists so we can dial /p2p-circuit/p2p/QmFoobar addresses (circuit addresses
that don't specify the relay). However, this kind of address is useless outside
of basic demos as a random relay is practically guaranteed to not be connected
to the target peer. Picking a random relay to connect to some peer is almost
_never_ the desired behavior.
2020-03-31 12:24:55 -07:00
Steven Allen
efdb8db276 fix: get rid of shutdown errors
Instead of feeding through the top-level context, feed through a cancel-free
context (that still carries the same context values). Then, when the top-level
context is canceled, call `stop` to shut everything down in-order. Finally,
cancel the inner context to make sure everything has been cleaned up.

Ideally, we just wouldn't use contexts for this. But this is strictly better
than what we have.
2020-03-29 20:48:43 -07:00
Steven Allen
f28b8fce66 feat: tls by default
Switches to TLS as the default security transports.
2020-03-29 19:12:54 -07:00
Steven Allen
ad543f0724 feat: update go-libp2p & go-bitswap
Uses the new libp2p AutoNAT option.
2020-03-24 14:54:09 -07:00
Steven Allen
b5ad3be060 feat: improve key lookup
* Support peer IDs encoded as CIDs.
* Support looking up the identity key by peer ID.
2020-03-23 15:40:48 -07:00
Cornelius Toole
62de9ba9d3 feat(file-ignore): add ignore opts to add cmd
- add the `gitignore` or `ignore options to the add command
2020-03-20 09:22:04 -07:00
Marcin Rataj
f9567a0a0f fix(gateway): curl without redirect on localhost
When request is sent to http://localhost:8080/ipfs/$cid response has
HTTP 301 status code and "Location" header with redirect destination at
$cid.ipfs.localhost:8080

Redirect is followed by browsersi, but not by commandline tools.
Status 301 is ignored by curl in default mode: it will print response
and won't follow redirect, user needs to add -L for that.

To fix curl, we return correct payload in body of HTTP 301 response,
but set Clear-Site-Data header to ensure Origin sandbox can't be abused.

This requires a surgical workaround:
If Location header is present in ResponseWriter's Header map,
we ensure http.ServeContent() returns HTTP 301

Context: https://github.com/ipfs/go-ipfs/pull/6982

License: MIT
Signed-off-by: Marcin Rataj <lidel@lidel.org>
2020-03-18 08:50:40 -07:00
Marcin Rataj
3ecccd6e1d feat(gateway): subdomain and proxy gateway
License: MIT
Signed-off-by: Marcin Rataj <lidel@lidel.org>
2020-03-18 08:50:38 -07:00
Steven Allen
72490f7ed0 gateway: simplify/improve dnslink rewrite handling
Instead of adding a new fake header (that could be spoofed by the client...),
just read the original request URI from the request object.

This also removes support for suborigins. They have never been implemented in
browsers and it looks like efforts have stalled. We can add support back if we
need it but, well, maintaining support was going to be more trouble than it was
worth.

License: MIT
Signed-off-by: Steven Allen <steven@stebalien.com>
2020-03-17 19:34:44 -07:00