coosld/kubo
1
0
Fork 0
mirror of https://github.com/ipfs/kubo.git synced 2026-03-09 18:28:08 +08:00
Code Issues Actions 3 Packages Projects Releases Wiki Activity

Merge pull request #9306 from ipfs/release-v0.16.0

Browse Source 
release v0.16.0
This commit is contained in:
Piotr Galar 2022-10-04 11:40:41 +02:00 committed by GitHub
commit 38117db6fc
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
153 changed files with 4990 additions and 2654 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

10
.circleci/main.yml
View File

@ -37,7 +37,7 @@ default_environment: &default_environment
executors:
golang:
docker:
- image: cimg/go:1.18.5
- image: cimg/go:1.19.1
working_directory: ~/ipfs/kubo
environment:
<<: *default_environment
@ -62,7 +62,7 @@ executors:
E2E_IPFSD_TYPE: go
dockerizer:
docker:
- image: cimg/go:1.18.5
- image: cimg/go:1.19.1
environment:
IMAGE_NAME: ipfs/kubo
WIP_IMAGE_TAG: wip
@ -156,8 +156,8 @@ jobs:
- run: sudo apt update
- run: |
mkdir ~/localgo && cd ~/localgo
wget https://golang.org/dl/go1.18.5.linux-amd64.tar.gz
tar xfz go1.18.5.linux-amd64.tar.gz
wget https://golang.org/dl/go1.19.1.linux-amd64.tar.gz
tar xfz go1.19.1.linux-amd64.tar.gz
echo "export PATH=$(pwd)/go/bin:\$PATH" >> ~/.bashrc
- run: go version
- run: sudo apt install socat net-tools
@ -223,7 +223,7 @@ jobs:
- *store_gomod
interop:
docker:
- image: cimg/go:1.18.5-node
- image: cimg/go:1.19.1-node
parallelism: 4
resource_class: large
steps:

6
.github/workflows/codeql-analysis.yml vendored
View File

@ -31,12 +31,12 @@ jobs:
# Initializes the CodeQL tools for scanning.
- name: Initialize CodeQL
uses: github/codeql-action/init@v1
uses: github/codeql-action/init@v2
with:
languages: ${{ matrix.language }}
- name: Autobuild
uses: github/codeql-action/autobuild@v1
uses: github/codeql-action/autobuild@v2
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@v1
uses: github/codeql-action/analyze@v2

9
.github/workflows/golang-analysis.yml vendored
View File

@ -11,7 +11,7 @@ jobs:
submodules: recursive
- uses: actions/setup-go@v2
with:
go-version: "1.18.x"
go-version: "1.19.x"
- name: Check that go.mod is tidy
uses: protocol/multiple-go-modules@v1.2
with:
@ -22,12 +22,13 @@ jobs:
exit 1
fi
git diff --exit-code -- go.sum go.mod
- name: gofmt
- name: go fmt
if: always() # run this step even if the previous one failed
run: |
out=$(gofmt -s -l .)
out=$(go fmt ./...)
if [[ -n "$out" ]]; then
echo $out | awk '{print "::error file=" $0 ",line=0,col=0::File is not gofmt-ed."}'
echo "Files are not go-fmt-ed:"
echo "$out"
exit 1
fi
- name: go vet

1
CHANGELOG.md
View File

@ -1,5 +1,6 @@
# Kubo Changelogs
- [v0.16](docs/changelogs/v0.16.md)
- [v0.15](docs/changelogs/v0.15.md)
- [v0.14](docs/changelogs/v0.14.md)
- [v0.13](docs/changelogs/v0.13.md)

6
Dockerfile
View File

@ -1,5 +1,5 @@
# Note: when updating the go minor version here, also update the go-channel in snap/snapcraft.yml
FROM golang:1.18.5-buster
FROM golang:1.19.1-buster
LABEL maintainer="Steven Allen <steven@stebalien.com>"
# Install deps
@ -111,10 +111,10 @@ ENV IPFS_LOGGING ""
# 2. The API and Gateway are accessible from outside the container.
ENTRYPOINT ["/sbin/tini", "--", "/usr/local/bin/start_ipfs"]
# Heathcheck for the container
# Healthcheck for the container
# QmUNLLsPACCz1vLxQVkXqqLX5R1X345qqfHbsf67hvA3Nn is the CID of empty folder
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
CMD ipfs dag stat /ipfs/QmUNLLsPACCz1vLxQVkXqqLX5R1X345qqfHbsf67hvA3Nn || exit 1
CMD ipfs dag stat /ipfs/QmUNLLsPACCz1vLxQVkXqqLX5R1X345qqfHbsf67hvA3Nn || exit 1
# Execute the daemon subcommand by default
CMD ["daemon", "--migrate=true", "--agent-version-suffix=docker"]

171
README.md
View File

@ -1,6 +1,3 @@
# kubo
> the oldest IPFS implementation, previously known as "go-ipfs"
![kubo, an IPFS node in Go](https://ipfs.io/ipfs/bafykbzacecaesuqmivkauix25v6i6xxxsvsrtxknhgb5zak3xxsg2nb4dhs2u/ipfs.go.png)
@ -10,15 +7,18 @@
## What is Kubo?
Kubo (go-ipfs) the earliest and most widely used implementation of IPFS.
Kubo was the first is the most widely used IPFS implementation today. Implementing the *Interplanetary Filesystem* - the Web3 standard and contender to replace https. Thus powered by IPLD's data models and the libp2p for network communication. Kubo is written in Go.
It includes:
- an IPFS daemon server
- extensive [command line tooling](https://docs.ipfs.tech/reference/kubo/cli/)
- an [HTTP Gateway](https://github.com/ipfs/specs/tree/main/http-gateways#readme) (`/ipfs/`, `/ipns/`) for serving content to HTTP browsers
- an HTTP RPC API (`/api/v0`) for controlling the daemon node
Featureset
- Runs an IPFS-Node as a network service
- [Command Line Interface](https://docs.ipfs.tech/reference/kubo/cli/) to IPFS-Nodes
- Local [Web2-to-Web3 HTTP Gateway functionality](https://github.com/ipfs/specs/tree/main/http-gateways#readme)
- HTTP RPC API (`/api/v0`) to access and control the daemon
- IPFS's internal Webgui can be used to manage the Kubo nodes
Note: [other implementations exist](https://docs.ipfs.tech/basics/ipfs-implementations/).
### Other implementations
See [List](https://docs.ipfs.tech/basics/ipfs-implementations/)
## What is IPFS?
@ -51,31 +51,30 @@ Before opening an issue, consider using one of the following locations to ensure
- [Install](#install)
- [System Requirements](#system-requirements)
- [Docker](#docker)
- [Native Linux package managers](#native-linux-package-managers)
- [ArchLinux](#archlinux)
- [Official prebuilt binaries](#official-prebuilt-binaries)
- [Updating](#updating)
- [Using ipfs-update](#using-ipfs-update)
- [Downloading builds using IPFS](#downloading-builds-using-ipfs)
- [Unofficial Linux packages](#unofficial-linux-packages)
- [ArchLinux](#arch-linux)
- [Nix](#nix)
- [Solus](#solus)
- [openSUSE](#opensuse)
- [Other package managers](#other-package-managers)
- [Guix](#guix)
- [Snap](#snap)
- [macOS package managers](#macos-package-managers)
- [Unofficial MacOS packages](#unofficial-macos-packages)
- [MacPorts](#macports)
- [Nix](#nix-1)
- [Homebrew](#homebrew)
- [Windows package managers](#windows-package-managers)
- [Homebrew](#homebrew)
- [Unofficial Windows packages](#unofficial-windows-packages)
- [Chocolatey](#chocolatey)
- [Scoop](#scoop)
- [Install prebuilt binaries](#install-prebuilt-binaries)
- [Build from Source](#build-from-source)
- [Install Go](#install-go)
- [Download and Compile IPFS](#download-and-compile-ipfs)
- [Cross Compiling](#cross-compiling)
- [OpenSSL](#openssl)
- [Troubleshooting](#troubleshooting)
- [Updating](#updating)
- [Using ipfs-update](#using-ipfs-update)
- [Downloading builds using IPFS](#downloading-builds-using-ipfs)
- [Getting Started](#getting-started)
- [Usage](#usage)
- [Some things to try](#some-things-to-try)
@ -110,19 +109,67 @@ If your system is resource-constrained, we recommend:
### Docker
[![Docker Image Version (legacy name)](https://img.shields.io/docker/v/ipfs/go-ipfs?color=blue&label=go-ipfs%20docker%20image&logo=docker&sort=semver&style=flat-square&cacheSeconds=3600)](https://hub.docker.com/r/ipfs/go-ipfs/)
<!-- TODO: replace with kubo after we have minimum set of images after kubo 0.14 (stable semver release, 'latest' and 'release' docker tags)
Official images are published at https://hub.docker.com/r/ipfs/kubo/:
[![Docker Image Version (latest semver)](https://img.shields.io/docker/v/ipfs/kubo?color=blue&label=kubo%20docker%20image&logo=docker&sort=semver&style=flat-square&cacheSeconds=3600)](https://hub.docker.com/r/ipfs/kubo/)
-->
More info on how to run kubo (go-ipfs) inside Docker can be found [here](https://docs.ipfs.tech/how-to/run-ipfs-inside-docker/).
More info on how to run Kubo (go-ipfs) inside Docker can be found [here](https://docs.ipfs.tech/how-to/run-ipfs-inside-docker/).
### Native Linux package managers
### Official prebuilt binaries
The official binaries are published at https://dist.ipfs.tech#kubo:
[![dist.ipfs.tech Downloads](https://img.shields.io/github/v/release/ipfs/kubo?label=dist.ipfs.tech&logo=ipfs&style=flat-square&cacheSeconds=3600)](https://dist.ipfs.tech#kubo)
From there:
- Click the blue "Download Kubo" on the right side of the page.
- Open/extract the archive.
- Move kubo (`ipfs`) to your path (`install.sh` can do it for you).
If you are unable to access [dist.ipfs.tech](https://dist.ipfs.tech#kubo), you can also download kubo (go-ipfs) from:
- this project's GitHub [releases](https://github.com/ipfs/kubo/releases/latest) page
- `/ipns/dist.ipfs.tech` at [dweb.link](https://dweb.link/ipns/dist.ipfs.tech#kubo) gateway
#### Updating
##### Using ipfs-update
IPFS has an updating tool that can be accessed through `ipfs update`. The tool is
not installed alongside IPFS in order to keep that logic independent of the main
codebase. To install `ipfs-update` tool, [download it here](https://dist.ipfs.tech/#ipfs-update).
##### Downloading builds using IPFS
List the available versions of Kubo (go-ipfs) implementation:
```console
$ ipfs cat /ipns/dist.ipfs.tech/kubo/versions
```
Then, to view available builds for a version from the previous command (`$VERSION`):
```console
$ ipfs ls /ipns/dist.ipfs.tech/kubo/$VERSION
```
To download a given build of a version:
```console
$ ipfs get /ipns/dist.ipfs.tech/kubo/$VERSION/kubo_$VERSION_darwin-386.tar.gz # darwin 32-bit build
$ ipfs get /ipns/dist.ipfs.tech/kubo/$VERSION/kubo_$VERSION_darwin-amd64.tar.gz # darwin 64-bit build
$ ipfs get /ipns/dist.ipfs.tech/kubo/$VERSION/kubo_$VERSION_freebsd-amd64.tar.gz # freebsd 64-bit build
$ ipfs get /ipns/dist.ipfs.tech/kubo/$VERSION/kubo_$VERSION_linux-386.tar.gz # linux 32-bit build
$ ipfs get /ipns/dist.ipfs.tech/kubo/$VERSION/kubo_$VERSION_linux-amd64.tar.gz # linux 64-bit build
$ ipfs get /ipns/dist.ipfs.tech/kubo/$VERSION/kubo_$VERSION_linux-arm.tar.gz # linux arm build
$ ipfs get /ipns/dist.ipfs.tech/kubo/$VERSION/kubo_$VERSION_windows-amd64.zip # windows 64-bit build
```
### Unofficial Linux packages
- [Arch Linux](#arch-linux)
- [Nix](#nix-linux)
- [Solus](#solus)
- [openSUSE](#openSUSE)
- [openSUSE](#opensuse)
#### Arch Linux
@ -159,11 +206,6 @@ You can also install it through the Solus software center.
[Community Package for go-ipfs](https://software.opensuse.org/package/go-ipfs)
### Other package managers
- [Guix](#guix)
- [Snap](#snap)
#### Guix
GNU's functional package manager, [Guix](https://www.gnu.org/software/guix/), also provides a go-ipfs package:
@ -174,6 +216,11 @@ $ guix package -i go-ipfs
#### Snap
> ⚠️ **SNAP USE IS DISCOURAGED**
>
> If you want something more sophisticated to escape the Snap confinement, we recommend using a different method to install Kubo so that it is not subject to snap confinement.
With snap, in any of the [supported Linux distributions](https://snapcraft.io/docs/core/install):
```
@ -186,14 +233,6 @@ The snap sets `IPFS_PATH` to `SNAP_USER_COMMON`, which is usually `~/snap/ipfs/c
$ sudo mount --bind ~/.ipfs ~/snap/ipfs/common
```
If you want something more sophisticated to escape the snap confinement, we recommend using a different method to install kubo so that it is not subject to snap confinement.
#### macOS package managers
- [MacPorts](#macports)
- [Nix](#nix-macos)
- [Homebrew](#Homebrew)
#### MacPorts
The package [ipfs](https://ports.macports.org/port/ipfs) currently points to kubo (go-ipfs) and is being maintained.
@ -220,7 +259,7 @@ A Homebrew formula [ipfs](https://formulae.brew.sh/formula/ipfs) is maintained t
$ brew install --formula ipfs
```
### Windows package managers
### Unofficial Windows packages
- [Chocolatey](#chocolatey)
- [Scoop](#scoop)
@ -230,7 +269,7 @@ $ brew install --formula ipfs
[![Chocolatey Version](https://img.shields.io/chocolatey/v/go-ipfs?color=00a4ef&label=go-ipfs&logo=windows&style=flat-square&cacheSeconds=3600)](https://chocolatey.org/packages/go-ipfs)
```Powershell
PS> choco install ipfs
PS> choco install go-ipfs
```
#### Scoop
@ -242,18 +281,12 @@ PS> scoop bucket add extras
PS> scoop install go-ipfs
```
### Install prebuilt binaries
### Unofficial macOS packages
[![dist.ipfs.tech Downloads](https://img.shields.io/github/v/release/ipfs/kubo?label=dist.ipfs.tech&logo=ipfs&style=flat-square&cacheSeconds=3600)](https://dweb.link/ipns/dist.ipfs.tech#kubo)
- [MacPorts](#macports)
- [Nix](#nix-macos)
- [Homebrew](#homebrew)
From there:
- Click the blue "Download kubo" on the right side of the page.
- Open/extract the archive.
- Move kubo (`ipfs`) to your path (`install.sh` can do it for you).
If you are unable to access [dist.ipfs.tech](https://dist.ipfs.tech#kubo), you can also download kubo (go-ipfs) from:
- this project's GitHub [releases](https://github.com/ipfs/kubo/releases/latest) page
- `/ipns/dist.ipfs.tech` at [dweb.link](https://dweb.link/ipns/dist.ipfs.tech#kubo) gateway
### Build from Source
@ -323,42 +356,6 @@ dependencies as well.
- Shell command completions can be generated with one of the `ipfs commands completion` subcommands. Read [docs/command-completion.md](docs/command-completion.md) to learn more.
- See the [misc folder](https://github.com/ipfs/kubo/tree/master/misc) for how to connect IPFS to systemd or whatever init system your distro uses.
### Updating
#### Using ipfs-update
IPFS has an updating tool that can be accessed through `ipfs update`. The tool is
not installed alongside IPFS in order to keep that logic independent of the main
codebase. To install `ipfs update`, [download it here](https://dist.ipfs.tech/#ipfs-update).
#### Downloading builds using IPFS
<!-- TODO: rename this section after we figure out if dist.ipfs.tech sgould produce both /go-ipfs/ and /kubo/ -->
List the available versions of kubo (go-ipfs) implementation:
```
$ ipfs cat /ipns/dist.ipfs.tech/go-ipfs/versions
```
Then, to view available builds for a version from the previous command ($VERSION):
```
$ ipfs ls /ipns/dist.ipfs.tech/go-ipfs/$VERSION
```
To download a given build of a version:
```
$ ipfs get /ipns/dist.ipfs.tech/go-ipfs/$VERSION/go-ipfs_$VERSION_darwin-386.tar.gz # darwin 32-bit build
$ ipfs get /ipns/dist.ipfs.tech/go-ipfs/$VERSION/go-ipfs_$VERSION_darwin-amd64.tar.gz # darwin 64-bit build
$ ipfs get /ipns/dist.ipfs.tech/go-ipfs/$VERSION/go-ipfs_$VERSION_freebsd-amd64.tar.gz # freebsd 64-bit build
$ ipfs get /ipns/dist.ipfs.tech/go-ipfs/$VERSION/go-ipfs_$VERSION_linux-386.tar.gz # linux 32-bit build
$ ipfs get /ipns/dist.ipfs.tech/go-ipfs/$VERSION/go-ipfs_$VERSION_linux-amd64.tar.gz # linux 64-bit build
$ ipfs get /ipns/dist.ipfs.tech/go-ipfs/$VERSION/go-ipfs_$VERSION_linux-arm.tar.gz # linux arm build
$ ipfs get /ipns/dist.ipfs.tech/go-ipfs/$VERSION/go-ipfs_$VERSION_windows-amd64.zip # windows 64-bit build
```
## Getting Started
### Usage

9
bin/mkreleaselog
View File

@ -52,7 +52,7 @@ IGNORE_FILES=(
if [[ ${#INCLUDE_MODULES[@]} -gt 0 ]]; then
INCLUDE_REGEX="(${$(printf "|%s" "${INCLUDE_MODULES[@]}"):1})"
else
else
INCLUDE_REGEX="" # "match anything"
fi
@ -149,11 +149,12 @@ pr_link() {
}
ignored_commit() {
local commit="$1"
local repo="$1"
local commit="$2"
local matches
# Check to see if this commit includes any non-ignored files.
matches=$(git -C "$dir" diff-tree --no-commit-id --name-only -r "$commit^" "$commit" \
matches=$(git -C "$repo" diff-tree --no-commit-id --name-only -r "$commit^" "$commit" \
-- "${IGNORE_FILES_PATHSPEC[@]}" | wc -l)
[[ "$matches" -eq 0 ]]
}
@ -175,7 +176,7 @@ release_log() {
"$start..$end" |
while read commit subject; do
# Skip commits that only touch ignored files.
if ignored_commit "$commit"; then
if ignored_commit "$dir" "$commit"; then
continue
fi

2
cmd/ipfs/add_migrations.go
View File

@ -16,7 +16,7 @@ import (
"github.com/ipfs/kubo/core/coreapi"
"github.com/ipfs/kubo/repo/fsrepo/migrations"
"github.com/ipfs/kubo/repo/fsrepo/migrations/ipfsfetcher"
"github.com/libp2p/go-libp2p-core/peer"
"github.com/libp2p/go-libp2p/core/peer"
)
// addMigrations adds any migration downloaded by the fetcher to the IPFS node

28
cmd/ipfs/daemon.go
View File

@ -59,6 +59,7 @@ const (
routingOptionDHTKwd = "dht"
routingOptionDHTServerKwd = "dhtserver"
routingOptionNoneKwd = "none"
routingOptionCustomKwd = "custom"
routingOptionDefaultKwd = "default"
unencryptTransportKwd = "disable-transport-encryption"
unrestrictedApiAccessKwd = "unrestricted-api"
@ -142,14 +143,15 @@ environment variable:
Routing
IPFS by default will use a DHT for content routing. There is a highly
experimental alternative that operates the DHT in a 'client only' mode that
can be enabled by running the daemon as:
IPFS by default will use a DHT for content routing. There is an alternative
that operates the DHT in a 'client only' mode that can be enabled by
running the daemon as:
ipfs daemon --routing=dhtclient
This will later be transitioned into a config option once it gets out of the
'experimental' stage.
Or you can set routing to dhtclient in the config:
ipfs config Routing.Type dhtclient
DEPRECATION NOTICE
@ -400,7 +402,10 @@ func daemonFunc(req *cmds.Request, re cmds.ResponseEmitter, env cmds.Environment
routingOption, _ := req.Options[routingOptionKwd].(string)
if routingOption == routingOptionDefaultKwd {
routingOption = cfg.Routing.Type.WithDefault(routingOptionDHTKwd)
routingOption = cfg.Routing.Type
if routingOption == "" {
routingOption = routingOptionDHTKwd
}
}
switch routingOption {
case routingOptionSupernodeKwd:
@ -413,6 +418,14 @@ func daemonFunc(req *cmds.Request, re cmds.ResponseEmitter, env cmds.Environment
ncfg.Routing = libp2p.DHTServerOption
case routingOptionNoneKwd:
ncfg.Routing = libp2p.NilRouterOption
case routingOptionCustomKwd:
ncfg.Routing = libp2p.ConstructDelegatedRouting(
cfg.Routing.Routers,
cfg.Routing.Methods,
cfg.Identity.PeerID,
cfg.Addresses.Swarm,
cfg.Identity.PrivKey,
)
default:
return fmt.Errorf("unrecognized routing option: %s", routingOption)
}
@ -520,6 +533,7 @@ func daemonFunc(req *cmds.Request, re cmds.ResponseEmitter, env cmds.Environment
"commit": version.CurrentCommit,
}).Set(1)
// TODO(9285): make metrics more configurable
// initialize metrics collector
prometheus.MustRegister(&corehttp.IpfsNodeCollector{Node: node})
@ -812,7 +826,7 @@ func serveHTTPGateway(req *cmds.Request, cctx *oldcmds.Context) (<-chan error, e
}
if len(cfg.Gateway.PathPrefixes) > 0 {
log.Error("Support for X-Ipfs-Gateway-Prefix and Gateway.PathPrefixes is deprecated and will be removed in the next release. Please comment on the issue if you're using this feature: https://github.com/ipfs/kubo/issues/7702")
log.Fatal("Support for custom Gateway.PathPrefixes was removed: https://github.com/ipfs/go-ipfs/issues/7702")
}
node, err := cctx.ConstructNode()

4
cmd/ipfs/pinmfs.go
View File

@ -5,8 +5,8 @@ import (
"fmt"
"time"
"github.com/libp2p/go-libp2p-core/host"
peer "github.com/libp2p/go-libp2p-core/peer"
"github.com/libp2p/go-libp2p/core/host"
peer "github.com/libp2p/go-libp2p/core/peer"
cid "github.com/ipfs/go-cid"
ipld "github.com/ipfs/go-ipld-format"

4
cmd/ipfs/pinmfs_test.go
View File

@ -10,8 +10,8 @@ import (
ipld "github.com/ipfs/go-ipld-format"
merkledag "github.com/ipfs/go-merkledag"
config "github.com/ipfs/kubo/config"
"github.com/libp2p/go-libp2p-core/host"
peer "github.com/libp2p/go-libp2p-core/peer"
"github.com/libp2p/go-libp2p/core/host"
peer "github.com/libp2p/go-libp2p/core/peer"
)
type testPinMFSContext struct {

2
config/bootstrap_peers.go
View File

@ -4,7 +4,7 @@ import (
"errors"
"fmt"
peer "github.com/libp2p/go-libp2p-core/peer"
peer "github.com/libp2p/go-libp2p/core/peer"
ma "github.com/multiformats/go-multiaddr"
)

10
config/config.go
View File

@ -78,11 +78,11 @@ func Path(configroot, extension string) (string, error) {
// Filename returns the configuration file path given a configuration root
// directory and a user-provided configuration file path argument with the
// following rules:
// * If the user-provided configuration file path is empty, use the default one.
// * If the configuration root directory is empty, use the default one.
// * If the user-provided configuration file path is only a file name, use the
// configuration root directory, otherwise use only the user-provided path
// and ignore the configuration root.
// - If the user-provided configuration file path is empty, use the default one.
// - If the configuration root directory is empty, use the default one.
// - If the user-provided configuration file path is only a file name, use the
// configuration root directory, otherwise use only the user-provided path
// and ignore the configuration root.
func Filename(configroot string, userConfigFile string) (string, error) {
if userConfigFile == "" {
return Path(configroot, DefaultConfigFile)

2
config/dns.go
View File

@ -9,7 +9,7 @@ type DNS struct {
// https://en.wikipedia.org/wiki/DNS_over_HTTPS
//
// Example:
// - Custom resolver for ENS: `eth.` → `https://eth.link/dns-query`
// - Custom resolver for ENS: `eth.` → `https://dns.eth.limo/dns-query`
// - Override the default OS resolver: `.` → `https://doh.applied-privacy.net/query`
Resolvers map[string]string
// MaxCacheTTL is the maximum duration DNS entries are valid in the cache.

19
config/gateway.go
View File

@ -8,7 +8,7 @@ type GatewaySpec struct {
// UseSubdomains indicates whether or not this gateway uses subdomains
// for IPFS resources instead of paths. That is: http://CID.ipfs.GATEWAY/...
//
// If this flag is set, any /ipns/$id and/or /ipfs/$id paths in PathPrefixes
// If this flag is set, any /ipns/$id and/or /ipfs/$id paths in Paths
// will be permanently redirected to http://$id.[ipns|ipfs].$gateway/.
//
// We do not support using both paths and subdomains for a single domain
@ -35,22 +35,7 @@ type Gateway struct {
// writing is done through the API, not the gateway.
Writable bool
// PathPrefixes is an array of acceptable url paths that a client can
// specify in X-Ipfs-Path-Prefix header.
//
// The X-Ipfs-Path-Prefix header is used to specify a base path to prepend
// to links in directory listings and for trailing-slash redirects. It is
// intended to be set by a frontend http proxy like nginx.
//
// Example: To mount blog.ipfs.io (a DNSLink site) at ipfs.io/blog
// set PathPrefixes to ["/blog"] and nginx config to translate paths
// and pass Host header (for DNSLink):
// location /blog/ {
// rewrite "^/blog(/.*)$" $1 break;
// proxy_set_header Host blog.ipfs.io;
// proxy_set_header X-Ipfs-Gateway-Prefix /blog;
// proxy_pass http://127.0.0.1:8080;
// }
// PathPrefixes was removed: https://github.com/ipfs/go-ipfs/issues/7702
PathPrefixes []string
// FastDirIndexThreshold is the maximum number of items in a directory

2
config/identity.go
View File

@ -3,7 +3,7 @@ package config
import (
"encoding/base64"
ic "github.com/libp2p/go-libp2p-core/crypto"
ic "github.com/libp2p/go-libp2p/core/crypto"
)
const IdentityTag = "Identity"

8
config/init.go
View File

@ -8,8 +8,8 @@ import (
"time"
"github.com/ipfs/interface-go-ipfs-core/options"
"github.com/libp2p/go-libp2p-core/crypto"
"github.com/libp2p/go-libp2p-core/peer"
"github.com/libp2p/go-libp2p/core/crypto"
"github.com/libp2p/go-libp2p/core/peer"
)
func Init(out io.Writer, nBitsForKeypair int) (*Config, error) {
@ -48,7 +48,9 @@ func InitWithIdentity(identity Identity) (*Config, error) {
},
Routing: Routing{
Type: NewOptionalString("dht"),
Type: "dht",
Methods: nil,
Routers: nil,
},
// setup the node mount points.

2
config/init_test.go
View File

@ -5,7 +5,7 @@ import (
"testing"
"github.com/ipfs/interface-go-ipfs-core/options"
crypto_pb "github.com/libp2p/go-libp2p-core/crypto/pb"
crypto_pb "github.com/libp2p/go-libp2p/core/crypto/pb"
)
func TestCreateIdentity(t *testing.T) {

2
config/peering.go
View File

@ -1,6 +1,6 @@
package config
import "github.com/libp2p/go-libp2p-core/peer"
import "github.com/libp2p/go-libp2p/core/peer"
// Peering configures the peering service.
type Peering struct {

2
config/profile.go
View File

@ -174,7 +174,7 @@ functionality - performance of content discovery and data
fetching may be degraded.
`,
Transform: func(c *Config) error {
c.Routing.Type = NewOptionalString("dhtclient")
c.Routing.Type = "dhtclient"
c.AutoNAT.ServiceMode = AutoNATServiceDisabled
c.Reprovider.Interval = "0"

144
config/routing.go
View File

@ -1,28 +1,101 @@
package config
import (
"encoding/json"
"fmt"
)
// Routing defines configuration options for libp2p routing
type Routing struct {
// Type sets default daemon routing mode.
//
// Can be one of "dht", "dhtclient", "dhtserver", "none", or unset.
Type *OptionalString `json:",omitempty"`
// Can be one of "dht", "dhtclient", "dhtserver", "none", or "custom".
// When "custom" is set, you can specify a list of Routers.
Type string
Routers map[string]Router
Routers Routers
Methods Methods
}
type Router struct {
// Currenly only supported Type is "reframe".
// Currenly supported Types are "reframe", "dht", "parallel", "sequential".
// Reframe type allows to add other resolvers using the Reframe spec:
// https://github.com/ipfs/specs/tree/main/reframe
// In the future we will support "dht" and other Types here.
Type string
Enabled Flag `json:",omitempty"`
Type RouterType
// Parameters are extra configuration that this router might need.
// A common one for reframe router is "Endpoint".
Parameters map[string]string
Parameters interface{}
}
type Routers map[string]RouterParser
type Methods map[MethodName]Method
func (m Methods) Check() error {
// Check supported methods
for _, mn := range MethodNameList {
_, ok := m[mn]
if !ok {
return fmt.Errorf("method name %q is missing from Routing.Methods config param", mn)
}
}
// Check unsupported methods
for k := range m {
seen := false
for _, mn := range MethodNameList {
if mn == k {
seen = true
break
}
}
if seen {
continue
}
return fmt.Errorf("method name %q is not a supported method on Routing.Methods config param", k)
}
return nil
}
type RouterParser struct {
Router
}
func (r *RouterParser) UnmarshalJSON(b []byte) error {
out := Router{}
out.Parameters = &json.RawMessage{}
if err := json.Unmarshal(b, &out); err != nil {
return err
}
raw := out.Parameters.(*json.RawMessage)
var p interface{}
switch out.Type {
case RouterTypeReframe:
p = &ReframeRouterParams{}
case RouterTypeDHT:
p = &DHTRouterParams{}
case RouterTypeSequential:
p = &ComposableRouterParams{}
case RouterTypeParallel:
p = &ComposableRouterParams{}
}
if err := json.Unmarshal(*raw, &p); err != nil {
return err
}
r.Router.Type = out.Type
r.Router.Parameters = p
return nil
}
// Type is the routing type.
@ -30,15 +103,56 @@ type Router struct {
type RouterType string
const (
RouterTypeReframe RouterType = "reframe"
RouterTypeReframe RouterType = "reframe"
RouterTypeDHT RouterType = "dht"
RouterTypeSequential RouterType = "sequential"
RouterTypeParallel RouterType = "parallel"
)
type RouterParam string
type DHTMode string
const (
// RouterParamEndpoint is the URL where the routing implementation will point to get the information.
// Usually used for reframe Routers.
RouterParamEndpoint RouterParam = "Endpoint"
RouterParamPriority RouterParam = "Priority"
DHTModeServer DHTMode = "server"
DHTModeClient DHTMode = "client"
DHTModeAuto DHTMode = "auto"
)
type MethodName string
const (
MethodNameProvide MethodName = "provide"
MethodNameFindProviders MethodName = "find-providers"
MethodNameFindPeers MethodName = "find-peers"
MethodNameGetIPNS MethodName = "get-ipns"
MethodNamePutIPNS MethodName = "put-ipns"
)
var MethodNameList = []MethodName{MethodNameProvide, MethodNameFindPeers, MethodNameFindProviders, MethodNameGetIPNS, MethodNamePutIPNS}
type ReframeRouterParams struct {
// Endpoint is the URL where the routing implementation will point to get the information.
// Usually used for reframe Routers.
Endpoint string
}
type DHTRouterParams struct {
Mode DHTMode
AcceleratedDHTClient bool `json:",omitempty"`
PublicIPNetwork bool
}
type ComposableRouterParams struct {
Routers []ConfigRouter
Timeout *OptionalDuration `json:",omitempty"`
}
type ConfigRouter struct {
RouterName string
Timeout Duration
IgnoreErrors bool
ExecuteAfter *OptionalDuration `json:",omitempty"`
}
type Method struct {
RouterName string
}

195
config/routing_test.go Normal file
View File

@ -0,0 +1,195 @@
package config
import (
"encoding/json"
"testing"
"time"
"github.com/stretchr/testify/require"
)
func TestRouterParameters(t *testing.T) {
require := require.New(t)
sec := time.Second
min := time.Minute
r := Routing{
Type: "custom",
Routers: map[string]RouterParser{
"router-dht": {Router{
Type: RouterTypeDHT,
Parameters: DHTRouterParams{
Mode: "auto",
AcceleratedDHTClient: true,
PublicIPNetwork: false,
},
}},
"router-reframe": {Router{
Type: RouterTypeReframe,
Parameters: ReframeRouterParams{
Endpoint: "reframe-endpoint",
},
}},
"router-parallel": {Router{
Type: RouterTypeParallel,
Parameters: ComposableRouterParams{
Routers: []ConfigRouter{
{
RouterName: "router-dht",
Timeout: Duration{10 * time.Second},
IgnoreErrors: true,
},
{
RouterName: "router-reframe",
Timeout: Duration{10 * time.Second},
IgnoreErrors: false,
ExecuteAfter: &OptionalDuration{&sec},
},
},
Timeout: &OptionalDuration{&min},
}},
},
"router-sequential": {Router{
Type: RouterTypeSequential,
Parameters: ComposableRouterParams{
Routers: []ConfigRouter{
{
RouterName: "router-dht",
Timeout: Duration{10 * time.Second},
IgnoreErrors: true,
},
{
RouterName: "router-reframe",
Timeout: Duration{10 * time.Second},
IgnoreErrors: false,
},
},
Timeout: &OptionalDuration{&min},
}},
},
},
Methods: Methods{
MethodNameFindPeers: {
RouterName: "router-reframe",
},
MethodNameFindProviders: {
RouterName: "router-dht",
},
MethodNameGetIPNS: {
RouterName: "router-sequential",
},
MethodNameProvide: {
RouterName: "router-parallel",
},
MethodNamePutIPNS: {
RouterName: "router-parallel",
},
},
}
out, err := json.Marshal(r)
require.NoError(err)
r2 := &Routing{}
err = json.Unmarshal(out, r2)
require.NoError(err)
require.Equal(5, len(r2.Methods))
dhtp := r2.Routers["router-dht"].Parameters
require.IsType(&DHTRouterParams{}, dhtp)
rp := r2.Routers["router-reframe"].Parameters
require.IsType(&ReframeRouterParams{}, rp)
sp := r2.Routers["router-sequential"].Parameters
require.IsType(&ComposableRouterParams{}, sp)
pp := r2.Routers["router-parallel"].Parameters
require.IsType(&ComposableRouterParams{}, pp)
}
func TestRouterMissingParameters(t *testing.T) {
require := require.New(t)
r := Routing{
Type: "custom",
Routers: map[string]RouterParser{
"router-wrong-reframe": {Router{
Type: RouterTypeReframe,
Parameters: DHTRouterParams{
Mode: "auto",
AcceleratedDHTClient: true,
PublicIPNetwork: false,
},
}},
},
Methods: Methods{
MethodNameFindPeers: {
RouterName: "router-wrong-reframe",
},
MethodNameFindProviders: {
RouterName: "router-wrong-reframe",
},
MethodNameGetIPNS: {
RouterName: "router-wrong-reframe",
},
MethodNameProvide: {
RouterName: "router-wrong-reframe",
},
MethodNamePutIPNS: {
RouterName: "router-wrong-reframe",
},
},
}
out, err := json.Marshal(r)
require.NoError(err)
r2 := &Routing{}
err = json.Unmarshal(out, r2)
require.NoError(err)
require.Empty(r2.Routers["router-wrong-reframe"].Parameters.(*ReframeRouterParams).Endpoint)
}
func TestMethods(t *testing.T) {
require := require.New(t)
methodsOK := Methods{
MethodNameFindPeers: {
RouterName: "router-wrong-reframe",
},
MethodNameFindProviders: {
RouterName: "router-wrong-reframe",
},
MethodNameGetIPNS: {
RouterName: "router-wrong-reframe",
},
MethodNameProvide: {
RouterName: "router-wrong-reframe",
},
MethodNamePutIPNS: {
RouterName: "router-wrong-reframe",
},
}
require.NoError(methodsOK.Check())
methodsMissing := Methods{
MethodNameFindPeers: {
RouterName: "router-wrong-reframe",
},
MethodNameGetIPNS: {
RouterName: "router-wrong-reframe",
},
MethodNameProvide: {
RouterName: "router-wrong-reframe",
},
MethodNamePutIPNS: {
RouterName: "router-wrong-reframe",
},
}
require.Error(methodsMissing.Check())
}

8
config/swarm.go
View File

@ -1,6 +1,6 @@
package config
import rcmgr "github.com/libp2p/go-libp2p-resource-manager"
import rcmgr "github.com/libp2p/go-libp2p/p2p/host/resource-manager"
type SwarmConfig struct {
// AddrFilters specifies a set libp2p addresses that we should never
@ -104,6 +104,8 @@ type Transports struct {
TCP Flag `json:",omitempty"`
Websocket Flag `json:",omitempty"`
Relay Flag `json:",omitempty"`
// except WebTransport which is experimental and optin.
WebTransport Flag `json:",omitempty"`
}
// Security specifies the transports used to encrypt insecure network
@ -136,14 +138,14 @@ type ConnMgr struct {
}
// ResourceMgr defines configuration options for the libp2p Network Resource Manager
// <https://github.com/libp2p/go-libp2p-resource-manager#readme>
// <https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#readme>
type ResourceMgr struct {
// Enables the Network Resource Manager feature, default to on.
Enabled Flag `json:",omitempty"`
Limits *rcmgr.LimitConfig `json:",omitempty"`
// A list of multiaddrs that can bypass normal system limits (but are still
// limited by the allowlist scope). Convenience config around
// https://pkg.go.dev/github.com/libp2p/go-libp2p-resource-manager#Allowlist.Add
// https://pkg.go.dev/github.com/libp2p/go-libp2p/p2p/host/resource-manager#Allowlist.Add
Allowlist []string `json:",omitempty"`
}

32
config/types.go
View File

@ -262,6 +262,38 @@ func (d OptionalDuration) String() string {
var _ json.Unmarshaler = (*OptionalDuration)(nil)
var _ json.Marshaler = (*OptionalDuration)(nil)
type Duration struct {
time.Duration
}
func (d Duration) MarshalJSON() ([]byte, error) {
return json.Marshal(d.String())
}
func (d *Duration) UnmarshalJSON(b []byte) error {
var v interface{}
if err := json.Unmarshal(b, &v); err != nil {
return err
}
switch value := v.(type) {
case float64:
d.Duration = time.Duration(value)
return nil
case string:
var err error
d.Duration, err = time.ParseDuration(value)
if err != nil {
return err
}
return nil
default:
return fmt.Errorf("unable to parse duration, expected a duration string or a float, but got %T", v)
}
}
var _ json.Unmarshaler = (*Duration)(nil)
var _ json.Marshaler = (*Duration)(nil)
// OptionalInteger represents an integer that has a default value
//
// When encoded in json, Default is encoded as "null"

10
core/bootstrap/bootstrap.go
View File

@ -13,11 +13,11 @@ import (
"github.com/jbenet/goprocess"
"github.com/jbenet/goprocess/context"
"github.com/jbenet/goprocess/periodic"
"github.com/libp2p/go-libp2p-core/host"
"github.com/libp2p/go-libp2p-core/network"
"github.com/libp2p/go-libp2p-core/peer"
"github.com/libp2p/go-libp2p-core/peerstore"
"github.com/libp2p/go-libp2p-core/routing"
"github.com/libp2p/go-libp2p/core/host"
"github.com/libp2p/go-libp2p/core/network"
"github.com/libp2p/go-libp2p/core/peer"
"github.com/libp2p/go-libp2p/core/peerstore"
"github.com/libp2p/go-libp2p/core/routing"
)
var log = logging.Logger("bootstrap")

4
core/bootstrap/bootstrap_test.go
View File

@ -3,8 +3,8 @@ package bootstrap
import (
"testing"
"github.com/libp2p/go-libp2p-core/peer"
"github.com/libp2p/go-libp2p-core/test"
"github.com/libp2p/go-libp2p/core/peer"
"github.com/libp2p/go-libp2p/core/test"
)
func TestSubsetWhenMaxIsGreaterThanLengthOfSlice(t *testing.T) {

91
core/commands/add.go
View File

@ -13,6 +13,8 @@ import (
"github.com/cheggaaa/pb"
cmds "github.com/ipfs/go-ipfs-cmds"
files "github.com/ipfs/go-ipfs-files"
ipld "github.com/ipfs/go-ipld-format"
mfs "github.com/ipfs/go-mfs"
coreiface "github.com/ipfs/interface-go-ipfs-core"
"github.com/ipfs/interface-go-ipfs-core/options"
mh "github.com/multiformats/go-multihash"
@ -45,6 +47,7 @@ const (
hashOptionName = "hash"
inlineOptionName = "inline"
inlineLimitOptionName = "inline-limit"
toFilesOptionName = "to-files"
)
const adderOutChanSize = 8
@ -79,6 +82,20 @@ You can now refer to the added file in a gateway, like so:
/ipfs/QmaG4FuMqEBnQNn3C8XJ5bpW8kLs7zq2ZXgHptJHbKDDVx/example.jpg
Files imported with 'ipfs add' are protected from GC (implicit '--pin=true'),
but it is up to you to remember the returned CID to get the data back later.
Passing '--to-files' creates a reference in Files API (MFS), making it easier
to find it in the future:
> ipfs files mkdir -p /myfs/dir
> ipfs add example.jpg --to-files /myfs/dir/
> ipfs files ls /myfs/dir/
example.jpg
See 'ipfs files --help' to learn more about using MFS
for keeping track of added files and directories.
The chunker option, '-s', specifies the chunking strategy that dictates
how to break files into blocks. Blocks with same content can
be deduplicated. Different chunking strategies will produce different
@ -138,7 +155,6 @@ See 'dag export' and 'dag import' for more information.
cmds.BoolOption(onlyHashOptionName, "n", "Only chunk and hash - do not write to disk."),
cmds.BoolOption(wrapOptionName, "w", "Wrap files with a directory object."),
cmds.StringOption(chunkerOptionName, "s", "Chunking algorithm, size-[bytes], rabin-[min]-[avg]-[max] or buzhash").WithDefault("size-262144"),
cmds.BoolOption(pinOptionName, "Pin this object when adding.").WithDefault(true),
cmds.BoolOption(rawLeavesOptionName, "Use raw blocks for leaf nodes."),
cmds.BoolOption(noCopyOptionName, "Add the file using filestore. Implies raw-leaves. (experimental)"),
cmds.BoolOption(fstoreCacheOptionName, "Check the filestore for pre-existing blocks. (experimental)"),
@ -146,6 +162,8 @@ See 'dag export' and 'dag import' for more information.
cmds.StringOption(hashOptionName, "Hash function to use. Implies CIDv1 if not sha2-256. (experimental)").WithDefault("sha2-256"),
cmds.BoolOption(inlineOptionName, "Inline small blocks into CIDs. (experimental)"),
cmds.IntOption(inlineLimitOptionName, "Maximum block size to inline. (experimental)").WithDefault(32),
cmds.BoolOption(pinOptionName, "Pin locally to protect added files from garbage collection.").WithDefault(true),
cmds.StringOption(toFilesOptionName, "Add reference to Files API (MFS) at the provided path."),
},
PreRun: func(req *cmds.Request, env cmds.Environment) error {
quiet, _ := req.Options[quietOptionName].(bool)
@ -186,10 +204,14 @@ See 'dag export' and 'dag import' for more information.
hashFunStr, _ := req.Options[hashOptionName].(string)
inline, _ := req.Options[inlineOptionName].(bool)
inlineLimit, _ := req.Options[inlineLimitOptionName].(int)
toFilesStr, toFilesSet := req.Options[toFilesOptionName].(string)
hashFunCode, ok := mh.Names[strings.ToLower(hashFunStr)]
if !ok {
return fmt.Errorf("unrecognized hash function: %s", strings.ToLower(hashFunStr))
return fmt.Errorf("unrecognized hash function: %q", strings.ToLower(hashFunStr))
}
if _, err := mh.GetHasher(hashFunCode); err != nil {
return err
}
enc, err := cmdenv.GetCidEncoder(req)
@ -235,7 +257,12 @@ See 'dag export' and 'dag import' for more information.
opts = append(opts, nil) // events option placeholder
ipfsNode, err := cmdenv.GetNode(env)
if err != nil {
return err
}
var added int
var fileAddedToMFS bool
addit := toadd.Entries()
for addit.Next() {
_, dir := addit.Node().(files.Directory)
@ -246,7 +273,65 @@ See 'dag export' and 'dag import' for more information.
go func() {
var err error
defer close(events)
_, err = api.Unixfs().Add(req.Context, addit.Node(), opts...)
pathAdded, err := api.Unixfs().Add(req.Context, addit.Node(), opts...)
if err != nil {
errCh <- err
return
}
// creating MFS pointers when optional --to-files is set
if toFilesSet {
if toFilesStr == "" {
toFilesStr = "/"
}
toFilesDst, err := checkPath(toFilesStr)
if err != nil {
errCh <- fmt.Errorf("%s: %w", toFilesOptionName, err)
return
}
dstAsDir := toFilesDst[len(toFilesDst)-1] == '/'
if dstAsDir {
mfsNode, err := mfs.Lookup(ipfsNode.FilesRoot, toFilesDst)
// confirm dst exists
if err != nil {
errCh <- fmt.Errorf("%s: MFS destination directory %q does not exist: %w", toFilesOptionName, toFilesDst, err)
return
}
// confirm dst is a dir
if mfsNode.Type() != mfs.TDir {
errCh <- fmt.Errorf("%s: MFS destination %q is not a directory", toFilesOptionName, toFilesDst)
return
}
// if MFS destination is a dir, append filename to the dir path
toFilesDst += path.Base(addit.Name())
}
// error if we try to overwrite a preexisting file destination
if fileAddedToMFS && !dstAsDir {
errCh <- fmt.Errorf("%s: MFS destination is a file: only one entry can be copied to %q", toFilesOptionName, toFilesDst)
return
}
_, err = mfs.Lookup(ipfsNode.FilesRoot, path.Dir(toFilesDst))
if err != nil {
errCh <- fmt.Errorf("%s: MFS destination parent %q %q does not exist: %w", toFilesOptionName, toFilesDst, path.Dir(toFilesDst), err)
return
}
var nodeAdded ipld.Node
nodeAdded, err = api.Dag().Get(req.Context, pathAdded.Cid())
if err != nil {
errCh <- err
return
}
err = mfs.PutNode(ipfsNode.FilesRoot, toFilesDst, nodeAdded)
if err != nil {
errCh <- fmt.Errorf("%s: cannot put node in path %q: %w", toFilesOptionName, toFilesDst, err)
return
}
fileAddedToMFS = true
}
errCh <- err
}()

2
core/commands/bitswap.go
View File

@ -12,7 +12,7 @@ import (
decision "github.com/ipfs/go-bitswap/decision"
cidutil "github.com/ipfs/go-cidutil"
cmds "github.com/ipfs/go-ipfs-cmds"
peer "github.com/libp2p/go-libp2p-core/peer"
peer "github.com/libp2p/go-libp2p/core/peer"
)
var BitswapCmd = &cmds.Command{

2
core/commands/bootstrap.go
View File

@ -12,7 +12,7 @@ import (
cmds "github.com/ipfs/go-ipfs-cmds"
config "github.com/ipfs/kubo/config"
peer "github.com/libp2p/go-libp2p-core/peer"
peer "github.com/libp2p/go-libp2p/core/peer"
ma "github.com/multiformats/go-multiaddr"
)

1
core/commands/config.go
View File

@ -152,6 +152,7 @@ Set the value of the 'Datastore.Path' key:
// A pattern matches a part if and only if the pattern is "*" or the lowercase pattern equals the lowercase part.
//
// For example:
//
// matchesGlobPrefix("foo.bar", []string{"*", "bar", "baz"}) returns true
// matchesGlobPrefix("foo.bar.baz", []string{"*", "bar"}) returns true
// matchesGlobPrefix("foo.bar", []string{"baz", "*"}) returns false

4
core/commands/dht.go
View File

@ -8,8 +8,8 @@ import (
cmds "github.com/ipfs/go-ipfs-cmds"
"github.com/ipfs/kubo/core/commands/cmdenv"
peer "github.com/libp2p/go-libp2p-core/peer"
routing "github.com/libp2p/go-libp2p-core/routing"
peer "github.com/libp2p/go-libp2p/core/peer"
routing "github.com/libp2p/go-libp2p/core/routing"
)
var ErrNotDHT = errors.New("routing service is not a DHT")

2
core/commands/dht_test.go
View File

@ -6,7 +6,7 @@ import (
"github.com/ipfs/go-namesys"
ipns "github.com/ipfs/go-ipns"
"github.com/libp2p/go-libp2p-core/test"
"github.com/libp2p/go-libp2p/core/test"
)
func TestKeyTranslation(t *testing.T) {

33
core/commands/files.go
View File

@ -581,7 +581,7 @@ const (
var filesReadCmd = &cmds.Command{
Helptext: cmds.HelpText{
Tagline: "Read a file in a given MFS.",
Tagline: "Read a file from MFS.",
ShortDescription: `
Read a specified number of bytes from a file at a given offset. By default,
it will read the entire file similar to the Unix cat.
@ -724,11 +724,16 @@ const (
var filesWriteCmd = &cmds.Command{
Helptext: cmds.HelpText{
Tagline: "Write to a mutable file in a given filesystem.",
Tagline: "Append to (modify) a file in MFS.",
ShortDescription: `
Write data to a file in a given filesystem. This command allows you to specify
a beginning offset to write to. The entire length of the input will be
written.
A low-level MFS command that allows you to append data to a file. If you want
to add a file without modifying an existing one, use 'ipfs add --to-files'
instead.
`,
LongDescription: `
A low-level MFS command that allows you to append data at the end of a file, or
specify a beginning offset within a file to write to. The entire length of the
input will be written.
If the '--create' option is specified, the file will be created if it does not
exist. Nonexistent intermediate directories will not be created unless the
@ -755,6 +760,22 @@ WARNING:
Usage of the '--flush=false' option does not guarantee data durability until
the tree has been flushed. This can be accomplished by running 'ipfs files
stat' on the file or any of its ancestors.
WARNING:
The CID produced by 'files write' will be different from 'ipfs add' because
'ipfs file write' creates a trickle-dag optimized for append-only operations
See '--trickle' in 'ipfs add --help' for more information.
If you want to add a file without modifying an existing one,
use 'ipfs add' with '--to-files':
> ipfs files mkdir -p /myfs/dir
> ipfs add example.jpg --to-files /myfs/dir/
> ipfs files ls /myfs/dir/
example.jpg
See '--to-files' in 'ipfs add --help' for more information.
`,
},
Arguments: []cmds.Argument{
@ -1019,7 +1040,7 @@ func updatePath(rt *mfs.Root, pth string, builder cid.Builder) error {
var filesRmCmd = &cmds.Command{
Helptext: cmds.HelpText{
Tagline: "Remove a file.",
Tagline: "Remove a file from MFS.",
ShortDescription: `
Remove files or directories.

11
core/commands/id.go
View File

@ -15,11 +15,11 @@ import (
cmds "github.com/ipfs/go-ipfs-cmds"
ke "github.com/ipfs/kubo/core/commands/keyencode"
ic "github.com/libp2p/go-libp2p-core/crypto"
"github.com/libp2p/go-libp2p-core/host"
peer "github.com/libp2p/go-libp2p-core/peer"
pstore "github.com/libp2p/go-libp2p-core/peerstore"
kb "github.com/libp2p/go-libp2p-kbucket"
ic "github.com/libp2p/go-libp2p/core/crypto"
"github.com/libp2p/go-libp2p/core/host"
peer "github.com/libp2p/go-libp2p/core/peer"
pstore "github.com/libp2p/go-libp2p/core/peerstore"
identify "github.com/libp2p/go-libp2p/p2p/protocol/identify"
)
@ -52,6 +52,7 @@ If no peer is specified, prints out information for local peers.
<pver>: Protocol version.
<pubkey>: Public key.
<addrs>: Addresses (newline delimited).
<protocols>: Libp2p Protocol registrations (newline delimited).
EXAMPLE:
@ -217,7 +218,7 @@ func printSelf(keyEnc ke.KeyEncoder, node *core.IpfsNode) (interface{}, error) {
info.Protocols = node.PeerHost.Mux().Protocols()
sort.Strings(info.Protocols)
}
info.ProtocolVersion = identify.LibP2PVersion
info.ProtocolVersion = identify.DefaultProtocolVersion
info.AgentVersion = version.GetUserAgentVersion()
return info, nil
}

4
core/commands/keyencode/keyencode.go
View File

@ -2,7 +2,7 @@ package keyencode
import (
cmds "github.com/ipfs/go-ipfs-cmds"
peer "github.com/libp2p/go-libp2p-core/peer"
peer "github.com/libp2p/go-libp2p/core/peer"
mbase "github.com/multiformats/go-multibase"
)
@ -29,7 +29,7 @@ func KeyEncoderFromString(formatLabel string) (KeyEncoder, error) {
func (enc KeyEncoder) FormatID(id peer.ID) string {
if enc.baseEnc == nil {
return peer.Encode(id)
return id.String()
}
if s, err := peer.ToCid(id).StringOfBase(enc.baseEnc.Encoding()); err != nil {
panic(err)

4
core/commands/keystore.go
View File

@ -22,8 +22,8 @@ import (
ke "github.com/ipfs/kubo/core/commands/keyencode"
fsrepo "github.com/ipfs/kubo/repo/fsrepo"
migrations "github.com/ipfs/kubo/repo/fsrepo/migrations"
"github.com/libp2p/go-libp2p-core/crypto"
peer "github.com/libp2p/go-libp2p-core/peer"
"github.com/libp2p/go-libp2p/core/crypto"
peer "github.com/libp2p/go-libp2p/core/peer"
)
var KeyCmd = &cmds.Command{

4
core/commands/name/ipnsps.go
View File

@ -8,8 +8,8 @@ import (
cmds "github.com/ipfs/go-ipfs-cmds"
"github.com/ipfs/kubo/core/commands/cmdenv"
ke "github.com/ipfs/kubo/core/commands/keyencode"
"github.com/libp2p/go-libp2p-core/peer"
record "github.com/libp2p/go-libp2p-record"
"github.com/libp2p/go-libp2p/core/peer"
)
type ipnsPubsubState struct {
@ -100,7 +100,7 @@ var ipnspsSubsCmd = &cmds.Command{
// Not necessarily an error.
continue
}
pid, err := peer.IDFromString(k)
pid, err := peer.IDFromBytes([]byte(k))
if err != nil {
log.Errorf("ipns key not a valid peer ID: %s", err)
continue

2
core/commands/name/publish.go
View File

@ -13,7 +13,7 @@ import (
options "github.com/ipfs/interface-go-ipfs-core/options"
path "github.com/ipfs/interface-go-ipfs-core/path"
ke "github.com/ipfs/kubo/core/commands/keyencode"
peer "github.com/libp2p/go-libp2p-core/peer"
peer "github.com/libp2p/go-libp2p/core/peer"
)
var (

6
core/commands/p2p.go
View File

@ -15,9 +15,9 @@ import (
p2p "github.com/ipfs/kubo/p2p"
cmds "github.com/ipfs/go-ipfs-cmds"
peer "github.com/libp2p/go-libp2p-core/peer"
pstore "github.com/libp2p/go-libp2p-core/peerstore"
protocol "github.com/libp2p/go-libp2p-core/protocol"
peer "github.com/libp2p/go-libp2p/core/peer"
pstore "github.com/libp2p/go-libp2p/core/peerstore"
protocol "github.com/libp2p/go-libp2p/core/protocol"
ma "github.com/multiformats/go-multiaddr"
madns "github.com/multiformats/go-multiaddr-dns"
)

4
core/commands/pin/remotepin.go
View File

@ -23,8 +23,8 @@ import (
config "github.com/ipfs/kubo/config"
"github.com/ipfs/kubo/core/commands/cmdenv"
fsrepo "github.com/ipfs/kubo/repo/fsrepo"
"github.com/libp2p/go-libp2p-core/host"
peer "github.com/libp2p/go-libp2p-core/peer"
"github.com/libp2p/go-libp2p/core/host"
peer "github.com/libp2p/go-libp2p/core/peer"
)
var log = logging.Logger("core/commands/cmdenv")

4
core/commands/ping.go
View File

@ -11,8 +11,8 @@ import (
"github.com/ipfs/kubo/core/commands/cmdenv"
cmds "github.com/ipfs/go-ipfs-cmds"
peer "github.com/libp2p/go-libp2p-core/peer"
pstore "github.com/libp2p/go-libp2p-core/peerstore"
peer "github.com/libp2p/go-libp2p/core/peer"
pstore "github.com/libp2p/go-libp2p/core/peerstore"
ping "github.com/libp2p/go-libp2p/p2p/protocol/ping"
ma "github.com/multiformats/go-multiaddr"
)

4
core/commands/routing.go
View File

@ -15,8 +15,8 @@ import (
ipld "github.com/ipfs/go-ipld-format"
dag "github.com/ipfs/go-merkledag"
path "github.com/ipfs/go-path"
peer "github.com/libp2p/go-libp2p-core/peer"
routing "github.com/libp2p/go-libp2p-core/routing"
peer "github.com/libp2p/go-libp2p/core/peer"
routing "github.com/libp2p/go-libp2p/core/routing"
)
var RoutingCmd = &cmds.Command{

6
core/commands/stat.go
View File

@ -10,9 +10,9 @@ import (
humanize "github.com/dustin/go-humanize"
cmds "github.com/ipfs/go-ipfs-cmds"
metrics "github.com/libp2p/go-libp2p-core/metrics"
peer "github.com/libp2p/go-libp2p-core/peer"
protocol "github.com/libp2p/go-libp2p-core/protocol"
metrics "github.com/libp2p/go-libp2p/core/metrics"
peer "github.com/libp2p/go-libp2p/core/peer"
protocol "github.com/libp2p/go-libp2p/core/protocol"
)
var StatsCmd = &cmds.Command{

4
core/commands/stat_dht.go
View File

@ -9,11 +9,11 @@ import (
cmdenv "github.com/ipfs/kubo/core/commands/cmdenv"
cmds "github.com/ipfs/go-ipfs-cmds"
"github.com/libp2p/go-libp2p-core/network"
pstore "github.com/libp2p/go-libp2p-core/peerstore"
dht "github.com/libp2p/go-libp2p-kad-dht"
"github.com/libp2p/go-libp2p-kad-dht/fullrt"
kbucket "github.com/libp2p/go-libp2p-kbucket"
"github.com/libp2p/go-libp2p/core/network"
pstore "github.com/libp2p/go-libp2p/core/peerstore"
)
type dhtPeerInfo struct {

6
core/commands/swarm.go
View File

@ -21,9 +21,9 @@ import (
"github.com/ipfs/kubo/repo/fsrepo"
cmds "github.com/ipfs/go-ipfs-cmds"
inet "github.com/libp2p/go-libp2p-core/network"
"github.com/libp2p/go-libp2p-core/peer"
rcmgr "github.com/libp2p/go-libp2p-resource-manager"
inet "github.com/libp2p/go-libp2p/core/network"
"github.com/libp2p/go-libp2p/core/peer"
rcmgr "github.com/libp2p/go-libp2p/p2p/host/resource-manager"
ma "github.com/multiformats/go-multiaddr"
madns "github.com/multiformats/go-multiaddr-dns"
mamask "github.com/whyrusleeping/multiaddr-filter"

42
core/core.go
View File

@ -5,7 +5,7 @@ Packages underneath core/ provide a (relatively) stable, low-level API
to carry out most IPFS-related tasks. For more details on the other
interfaces and how core/... fits into the bigger IPFS picture, see:
$ godoc github.com/ipfs/go-ipfs
$ godoc github.com/ipfs/go-ipfs
*/
package core
@ -26,18 +26,18 @@ import (
logging "github.com/ipfs/go-log"
mfs "github.com/ipfs/go-mfs"
goprocess "github.com/jbenet/goprocess"
connmgr "github.com/libp2p/go-libp2p-core/connmgr"
ic "github.com/libp2p/go-libp2p-core/crypto"
p2phost "github.com/libp2p/go-libp2p-core/host"
metrics "github.com/libp2p/go-libp2p-core/metrics"
"github.com/libp2p/go-libp2p-core/network"
peer "github.com/libp2p/go-libp2p-core/peer"
pstore "github.com/libp2p/go-libp2p-core/peerstore"
routing "github.com/libp2p/go-libp2p-core/routing"
ddht "github.com/libp2p/go-libp2p-kad-dht/dual"
pubsub "github.com/libp2p/go-libp2p-pubsub"
psrouter "github.com/libp2p/go-libp2p-pubsub-router"
record "github.com/libp2p/go-libp2p-record"
connmgr "github.com/libp2p/go-libp2p/core/connmgr"
ic "github.com/libp2p/go-libp2p/core/crypto"
p2phost "github.com/libp2p/go-libp2p/core/host"
metrics "github.com/libp2p/go-libp2p/core/metrics"
"github.com/libp2p/go-libp2p/core/network"
peer "github.com/libp2p/go-libp2p/core/peer"
pstore "github.com/libp2p/go-libp2p/core/peerstore"
routing "github.com/libp2p/go-libp2p/core/routing"
"github.com/libp2p/go-libp2p/p2p/discovery/mdns"
p2pbhost "github.com/libp2p/go-libp2p/p2p/host/basic"
ma "github.com/multiformats/go-multiaddr"
@ -87,18 +87,18 @@ type IpfsNode struct {
RecordValidator record.Validator
// Online
PeerHost p2phost.Host `optional:"true"` // the network host (server+client)
Peering *peering.PeeringService `optional:"true"`
Filters *ma.Filters `optional:"true"`
Bootstrapper io.Closer `optional:"true"` // the periodic bootstrapper
Routing irouting.TieredRouter `optional:"true"` // the routing system. recommend ipfs-dht
DNSResolver *madns.Resolver // the DNS resolver
Exchange exchange.Interface // the block exchange + strategy (bitswap)
Namesys namesys.NameSystem // the name system, resolves paths to hashes
Provider provider.System // the value provider system
IpnsRepub *ipnsrp.Republisher `optional:"true"`
GraphExchange graphsync.GraphExchange `optional:"true"`
ResourceManager network.ResourceManager `optional:"true"`
PeerHost p2phost.Host `optional:"true"` // the network host (server+client)
Peering *peering.PeeringService `optional:"true"`
Filters *ma.Filters `optional:"true"`
Bootstrapper io.Closer `optional:"true"` // the periodic bootstrapper
Routing irouting.ProvideManyRouter `optional:"true"` // the routing system. recommend ipfs-dht
DNSResolver *madns.Resolver // the DNS resolver
Exchange exchange.Interface // the block exchange + strategy (bitswap)
Namesys namesys.NameSystem // the name system, resolves paths to hashes
Provider provider.System // the value provider system
IpnsRepub *ipnsrp.Republisher `optional:"true"`
GraphExchange graphsync.GraphExchange `optional:"true"`
ResourceManager network.ResourceManager `optional:"true"`
PubSub *pubsub.PubSub `optional:"true"`
PSRouter *psrouter.PubsubValueStore `optional:"true"`

88
core/core_test.go
View File

@ -16,8 +16,8 @@ import (
"github.com/ipfs/go-ipns"
"github.com/ipfs/kubo/core/node/libp2p"
"github.com/ipfs/kubo/repo"
"github.com/libp2p/go-libp2p-core/crypto"
peer "github.com/libp2p/go-libp2p-core/peer"
"github.com/libp2p/go-libp2p/core/crypto"
peer "github.com/libp2p/go-libp2p/core/peer"
"github.com/stretchr/testify/require"
datastore "github.com/ipfs/go-datastore"
@ -117,8 +117,6 @@ func TestDelegatedRoutingSingle(t *testing.T) {
err = n.Routing.PutValue(ctx, theID, v)
require.NoError(err)
err = n.Routing.PutValue(ctx, theErrorID, v)
require.Error(err)
}
func TestDelegatedRoutingMulti(t *testing.T) {
@ -164,12 +162,6 @@ func TestDelegatedRoutingMulti(t *testing.T) {
require.NoError(err)
require.NotNil(v)
require.Contains(string(v), "RECORD FROM SERVICE 2")
err = n.Routing.PutValue(ctx, theID1, v)
require.Error(err)
err = n.Routing.PutValue(ctx, theID2, v)
require.Error(err)
}
func StartRoutingServer(t *testing.T, d drs.DelegatedRoutingService) string {
@ -187,16 +179,41 @@ func StartRoutingServer(t *testing.T, d drs.DelegatedRoutingService) string {
func GetNode(t *testing.T, reframeURLs ...string) *IpfsNode {
t.Helper()
routers := make(map[string]config.Router)
routers := make(config.Routers)
var routerNames []string
for i, ru := range reframeURLs {
routers[fmt.Sprintf("reframe-%d", i)] = config.Router{
Type: string(config.RouterTypeReframe),
Parameters: map[string]string{
string(config.RouterParamEndpoint): ru,
},
}
rn := fmt.Sprintf("reframe-%d", i)
routerNames = append(routerNames, rn)
routers[rn] =
config.RouterParser{
Router: config.Router{
Type: config.RouterTypeReframe,
Parameters: &config.ReframeRouterParams{
Endpoint: ru,
},
},
}
}
var crs []config.ConfigRouter
for _, rn := range routerNames {
crs = append(crs, config.ConfigRouter{
RouterName: rn,
IgnoreErrors: true,
Timeout: config.Duration{Duration: time.Minute},
})
}
const parallelRouterName = "parallel-router"
routers[parallelRouterName] = config.RouterParser{
Router: config.Router{
Type: config.RouterTypeParallel,
Parameters: &config.ComposableRouterParams{
Routers: crs,
},
},
}
cfg := config.Config{
Identity: testIdentity,
Addresses: config.Addresses{
@ -204,8 +221,25 @@ func GetNode(t *testing.T, reframeURLs ...string) *IpfsNode {
API: []string{"/ip4/127.0.0.1/tcp/0"},
},
Routing: config.Routing{
Type: config.NewOptionalString("none"),
Type: "custom",
Routers: routers,
Methods: config.Methods{
config.MethodNameFindPeers: config.Method{
RouterName: parallelRouterName,
},
config.MethodNameFindProviders: config.Method{
RouterName: parallelRouterName,
},
config.MethodNameGetIPNS: config.Method{
RouterName: parallelRouterName,
},
config.MethodNameProvide: config.Method{
RouterName: parallelRouterName,
},
config.MethodNamePutIPNS: config.Method{
RouterName: parallelRouterName,
},
},
},
}
@ -214,7 +248,19 @@ func GetNode(t *testing.T, reframeURLs ...string) *IpfsNode {
D: syncds.MutexWrap(datastore.NewMapDatastore()),
}
n, err := NewNode(context.Background(), &BuildCfg{Repo: r, Online: true, Routing: libp2p.NilRouterOption})
n, err := NewNode(context.Background(),
&BuildCfg{
Repo: r,
Online: true,
Routing: libp2p.ConstructDelegatedRouting(
cfg.Routing.Routers,
cfg.Routing.Methods,
cfg.Identity.PeerID,
cfg.Addresses.Swarm,
cfg.Identity.PrivKey,
),
},
)
require.NoError(t, err)
return n
@ -240,6 +286,10 @@ func (drs *delegatedRoutingService) FindProviders(ctx context.Context, key cid.C
return nil, errNotSupported
}
func (drs *delegatedRoutingService) Provide(ctx context.Context, req *client.ProvideRequest) (<-chan client.ProvideAsyncResult, error) {
return nil, errNotSupported
}
func (drs *delegatedRoutingService) GetIPNS(ctx context.Context, id []byte) (<-chan client.GetIPNSAsyncResult, error) {
ctx, cancel := context.WithCancel(ctx)
ch := make(chan client.GetIPNSAsyncResult)

10
core/coreapi/coreapi.go
View File

@ -30,13 +30,13 @@ import (
dag "github.com/ipfs/go-merkledag"
coreiface "github.com/ipfs/interface-go-ipfs-core"
"github.com/ipfs/interface-go-ipfs-core/options"
ci "github.com/libp2p/go-libp2p-core/crypto"
p2phost "github.com/libp2p/go-libp2p-core/host"
peer "github.com/libp2p/go-libp2p-core/peer"
pstore "github.com/libp2p/go-libp2p-core/peerstore"
routing "github.com/libp2p/go-libp2p-core/routing"
pubsub "github.com/libp2p/go-libp2p-pubsub"
record "github.com/libp2p/go-libp2p-record"
ci "github.com/libp2p/go-libp2p/core/crypto"
p2phost "github.com/libp2p/go-libp2p/core/host"
peer "github.com/libp2p/go-libp2p/core/peer"
pstore "github.com/libp2p/go-libp2p/core/peerstore"
routing "github.com/libp2p/go-libp2p/core/routing"
madns "github.com/multiformats/go-multiaddr-dns"
"github.com/ipfs/go-namesys"

4
core/coreapi/dht.go
View File

@ -14,8 +14,8 @@ import (
caopts "github.com/ipfs/interface-go-ipfs-core/options"
path "github.com/ipfs/interface-go-ipfs-core/path"
"github.com/ipfs/kubo/tracing"
peer "github.com/libp2p/go-libp2p-core/peer"
routing "github.com/libp2p/go-libp2p-core/routing"
peer "github.com/libp2p/go-libp2p/core/peer"
routing "github.com/libp2p/go-libp2p/core/routing"
"go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/otel/trace"
)

4
core/coreapi/key.go
View File

@ -12,8 +12,8 @@ import (
caopts "github.com/ipfs/interface-go-ipfs-core/options"
path "github.com/ipfs/interface-go-ipfs-core/path"
"github.com/ipfs/kubo/tracing"
crypto "github.com/libp2p/go-libp2p-core/crypto"
peer "github.com/libp2p/go-libp2p-core/peer"
crypto "github.com/libp2p/go-libp2p/core/crypto"
peer "github.com/libp2p/go-libp2p/core/peer"
"go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/otel/trace"
)

8
core/coreapi/name.go
View File

@ -16,8 +16,8 @@ import (
coreiface "github.com/ipfs/interface-go-ipfs-core"
caopts "github.com/ipfs/interface-go-ipfs-core/options"
path "github.com/ipfs/interface-go-ipfs-core/path"
ci "github.com/libp2p/go-libp2p-core/crypto"
peer "github.com/libp2p/go-libp2p-core/peer"
ci "github.com/libp2p/go-libp2p/core/crypto"
peer "github.com/libp2p/go-libp2p/core/peer"
)
type NameAPI CoreAPI
@ -37,6 +37,8 @@ func (e *ipnsEntry) Value() path.Path {
return e.value
}
type requestContextKey string
// Publish announces new IPNS name and returns the new IPNS entry.
func (api *NameAPI) Publish(ctx context.Context, p path.Path, opts ...caopts.NamePublishOption) (coreiface.IpnsEntry, error) {
ctx, span := tracing.Span(ctx, "CoreAPI.NameAPI", "Publish", trace.WithAttributes(attribute.String("path", p.String())))
@ -76,7 +78,7 @@ func (api *NameAPI) Publish(ctx context.Context, p path.Path, opts ...caopts.Nam
if options.TTL != nil {
// nolint: staticcheck // non-backward compatible change
ctx = context.WithValue(ctx, "ipns-publish-ttl", *options.TTL)
ctx = context.WithValue(ctx, requestContextKey("ipns-publish-ttl"), *options.TTL)
}
eol := time.Now().Add(options.ValidTime)

4
core/coreapi/pubsub.go
View File

@ -7,9 +7,9 @@ import (
coreiface "github.com/ipfs/interface-go-ipfs-core"
caopts "github.com/ipfs/interface-go-ipfs-core/options"
"github.com/ipfs/kubo/tracing"
peer "github.com/libp2p/go-libp2p-core/peer"
routing "github.com/libp2p/go-libp2p-core/routing"
pubsub "github.com/libp2p/go-libp2p-pubsub"
peer "github.com/libp2p/go-libp2p/core/peer"
routing "github.com/libp2p/go-libp2p/core/routing"
"go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/otel/trace"
)

8
core/coreapi/swarm.go
View File

@ -7,10 +7,10 @@ import (
coreiface "github.com/ipfs/interface-go-ipfs-core"
"github.com/ipfs/kubo/tracing"
inet "github.com/libp2p/go-libp2p-core/network"
"github.com/libp2p/go-libp2p-core/peer"
pstore "github.com/libp2p/go-libp2p-core/peerstore"
"github.com/libp2p/go-libp2p-core/protocol"
inet "github.com/libp2p/go-libp2p/core/network"
"github.com/libp2p/go-libp2p/core/peer"
pstore "github.com/libp2p/go-libp2p/core/peerstore"
"github.com/libp2p/go-libp2p/core/protocol"
"github.com/libp2p/go-libp2p/p2p/net/swarm"
ma "github.com/multiformats/go-multiaddr"
"go.opentelemetry.io/otel/attribute"

4
core/coreapi/test/api_test.go
View File

@ -22,8 +22,8 @@ import (
coreiface "github.com/ipfs/interface-go-ipfs-core"
"github.com/ipfs/interface-go-ipfs-core/tests"
"github.com/ipfs/kubo/config"
"github.com/libp2p/go-libp2p-core/crypto"
"github.com/libp2p/go-libp2p-core/peer"
"github.com/libp2p/go-libp2p/core/crypto"
"github.com/libp2p/go-libp2p/core/peer"
mocknet "github.com/libp2p/go-libp2p/p2p/net/mock"
)

4
core/corehttp/gateway.go
View File

@ -20,7 +20,6 @@ import (
type GatewayConfig struct {
Headers map[string][]string
Writable bool
PathPrefixes []string
FastDirIndexThreshold int
}
@ -86,7 +85,6 @@ func GatewayOption(writable bool, paths ...string) ServeOption {
gateway := NewGatewayHandler(GatewayConfig{
Headers: headers,
Writable: writable,
PathPrefixes: cfg.Gateway.PathPrefixes,
FastDirIndexThreshold: int(cfg.Gateway.FastDirIndexThreshold.WithDefault(100)),
}, api, offlineApi)
@ -148,7 +146,7 @@ func VersionOption() ServeOption {
mux.HandleFunc("/version", func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "Commit: %s\n", version.CurrentCommit)
fmt.Fprintf(w, "Client Version: %s\n", version.GetUserAgentVersion())
fmt.Fprintf(w, "Protocol Version: %s\n", id.LibP2PVersion)
fmt.Fprintf(w, "Protocol Version: %s\n", id.DefaultProtocolVersion)
})
return mux, nil
}

140
core/corehttp/gateway_handler.go
View File

@ -13,7 +13,6 @@ import (
gopath "path"
"regexp"
"runtime/debug"
"strconv"
"strings"
"time"
@ -26,7 +25,7 @@ import (
"github.com/ipfs/go-path/resolver"
coreiface "github.com/ipfs/interface-go-ipfs-core"
ipath "github.com/ipfs/interface-go-ipfs-core/path"
routing "github.com/libp2p/go-libp2p-core/routing"
routing "github.com/libp2p/go-libp2p/core/routing"
prometheus "github.com/prometheus/client_golang/prometheus"
"go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/otel/trace"
@ -378,23 +377,6 @@ func (i *gatewayHandler) getOrHeadHandler(w http.ResponseWriter, r *http.Request
return
}
// Resolve path to the final DAG node for the ETag
resolvedPath, err := i.api.ResolvePath(r.Context(), contentPath)
switch err {
case nil:
case coreiface.ErrOffline:
webError(w, "ipfs resolve -r "+debugStr(contentPath.String()), err, http.StatusServiceUnavailable)
return
default:
// if Accept is text/html, see if ipfs-404.html is present
if i.servePretty404IfPresent(w, r, contentPath) {
logger.Debugw("serve pretty 404 if present")
return
}
webError(w, "ipfs resolve -r "+debugStr(contentPath.String()), err, http.StatusBadRequest)
return
}
// Detect when explicit Accept header or ?format parameter are present
responseFormat, formatParams, err := customResponseFormat(r)
if err != nil {
@ -402,6 +384,11 @@ func (i *gatewayHandler) getOrHeadHandler(w http.ResponseWriter, r *http.Request
return
}
trace.SpanFromContext(r.Context()).SetAttributes(attribute.String("ResponseFormat", responseFormat))
resolvedPath, contentPath, ok := i.handlePathResolution(w, r, responseFormat, contentPath, logger)
if !ok {
return
}
trace.SpanFromContext(r.Context()).SetAttributes(attribute.String("ResolvedPath", resolvedPath.String()))
// Detect when If-None-Match HTTP header allows returning HTTP 304 Not Modified
@ -450,36 +437,6 @@ func (i *gatewayHandler) getOrHeadHandler(w http.ResponseWriter, r *http.Request
}
}
func (i *gatewayHandler) servePretty404IfPresent(w http.ResponseWriter, r *http.Request, contentPath ipath.Path) bool {
resolved404Path, ctype, err := i.searchUpTreeFor404(r, contentPath)
if err != nil {
return false
}
dr, err := i.api.Unixfs().Get(r.Context(), resolved404Path)
if err != nil {
return false
}
defer dr.Close()
f, ok := dr.(files.File)
if !ok {
return false
}
size, err := f.Size()
if err != nil {
return false
}
log.Debugw("using pretty 404 file", "path", contentPath)
w.Header().Set("Content-Type", ctype)
w.Header().Set("Content-Length", strconv.FormatInt(size, 10))
w.WriteHeader(http.StatusNotFound)
_, err = io.CopyN(w, f, size)
return err == nil
}
func (i *gatewayHandler) postHandler(w http.ResponseWriter, r *http.Request) {
p, err := i.api.Unixfs().Add(r.Context(), files.NewReaderFile(r.Body))
if err != nil {
@ -920,48 +877,6 @@ func customResponseFormat(r *http.Request) (mediaType string, params map[string]
return "", nil, nil
}
func (i *gatewayHandler) searchUpTreeFor404(r *http.Request, contentPath ipath.Path) (ipath.Resolved, string, error) {
filename404, ctype, err := preferred404Filename(r.Header.Values("Accept"))
if err != nil {
return nil, "", err
}
pathComponents := strings.Split(contentPath.String(), "/")
for idx := len(pathComponents); idx >= 3; idx-- {
pretty404 := gopath.Join(append(pathComponents[0:idx], filename404)...)
parsed404Path := ipath.New("/" + pretty404)
if parsed404Path.IsValid() != nil {
break
}
resolvedPath, err := i.api.ResolvePath(r.Context(), parsed404Path)
if err != nil {
continue
}
return resolvedPath, ctype, nil
}
return nil, "", fmt.Errorf("no pretty 404 in any parent folder")
}
func preferred404Filename(acceptHeaders []string) (string, string, error) {
// If we ever want to offer a 404 file for a different content type
// then this function will need to parse q weightings, but for now
// the presence of anything matching HTML is enough.
for _, acceptHeader := range acceptHeaders {
accepted := strings.Split(acceptHeader, ",")
for _, spec := range accepted {
contentType := strings.SplitN(spec, ";", 1)[0]
switch contentType {
case "*/*", "text/*", "text/html":
return "ipfs-404.html", "text/html", nil
}
}
}
return "", "", fmt.Errorf("there is no 404 file for the requested content types")
}
// returns unquoted path with all special characters revealed as \u codes
func debugStr(path string) string {
q := fmt.Sprintf("%+q", path)
@ -971,6 +886,49 @@ func debugStr(path string) string {
return q
}
// Resolve the provided contentPath including any special handling related to
// the requested responseFormat. Returned ok flag indicates if gateway handler
// should continue processing the request.
func (i *gatewayHandler) handlePathResolution(w http.ResponseWriter, r *http.Request, responseFormat string, contentPath ipath.Path, logger *zap.SugaredLogger) (resolvedPath ipath.Resolved, newContentPath ipath.Path, ok bool) {
// Attempt to resolve the provided path.
resolvedPath, err := i.api.ResolvePath(r.Context(), contentPath)
switch err {
case nil:
return resolvedPath, contentPath, true
case coreiface.ErrOffline:
webError(w, "ipfs resolve -r "+debugStr(contentPath.String()), err, http.StatusServiceUnavailable)
return nil, nil, false
default:
// The path can't be resolved.
if isUnixfsResponseFormat(responseFormat) {
// If we have origin isolation (subdomain gw, DNSLink website),
// and response type is UnixFS (default for website hosting)
// check for presence of _redirects file and apply rules defined there.
// See: https://github.com/ipfs/specs/pull/290
if hasOriginIsolation(r) {
resolvedPath, newContentPath, ok, hadMatchingRule := i.serveRedirectsIfPresent(w, r, resolvedPath, contentPath, logger)
if hadMatchingRule {
logger.Debugw("applied a rule from _redirects file")
return resolvedPath, newContentPath, ok
}
}
// if Accept is text/html, see if ipfs-404.html is present
// This logic isn't documented and will likely be removed at some point.
// Any 404 logic in _redirects above will have already run by this time, so it's really an extra fall back
if i.serveLegacy404IfPresent(w, r, contentPath) {
logger.Debugw("served legacy 404")
return nil, nil, false
}
}
// Note: webError will replace http.StatusBadRequest with StatusNotFound if necessary
webError(w, "ipfs resolve -r "+debugStr(contentPath.String()), err, http.StatusBadRequest)
return nil, nil, false
}
}
// Detect 'Cache-Control: only-if-cached' in request and return data if it is already in the local datastore.
// https://github.com/ipfs/specs/blob/main/http-gateways/PATH_GATEWAY.md#cache-control-request-header
func (i *gatewayHandler) handleOnlyIfCached(w http.ResponseWriter, r *http.Request, contentPath ipath.Path, logger *zap.SugaredLogger) (requestHandled bool) {

287
core/corehttp/gateway_handler_unixfs__redirects.go Normal file
View File

@ -0,0 +1,287 @@
package corehttp
import (
"fmt"
"io"
"net/http"
gopath "path"
"strconv"
"strings"
files "github.com/ipfs/go-ipfs-files"
redirects "github.com/ipfs/go-ipfs-redirects-file"
ipath "github.com/ipfs/interface-go-ipfs-core/path"
"go.uber.org/zap"
)
// Resolving a UnixFS path involves determining if the provided `path.Path` exists and returning the `path.Resolved`
// corresponding to that path. For UnixFS, path resolution is more involved.
//
// When a path under requested CID does not exist, Gateway will check if a `_redirects` file exists
// underneath the root CID of the path, and apply rules defined there.
// See sepcification introduced in: https://github.com/ipfs/specs/pull/290
//
// Scenario 1:
// If a path exists, we always return the `path.Resolved` corresponding to that path, regardless of the existence of a `_redirects` file.
//
// Scenario 2:
// If a path does not exist, usually we should return a `nil` resolution path and an error indicating that the path
// doesn't exist. However, a `_redirects` file may exist and contain a redirect rule that redirects that path to a different path.
// We need to evaluate the rule and perform the redirect if present.
//
// Scenario 3:
// Another possibility is that the path corresponds to a rewrite rule (i.e. a rule with a status of 200).
// In this case, we don't perform a redirect, but do need to return a `path.Resolved` and `path.Path` corresponding to
// the rewrite destination path.
//
// Note that for security reasons, redirect rules are only processed when the request has origin isolation.
// See https://github.com/ipfs/specs/pull/290 for more information.
func (i *gatewayHandler) serveRedirectsIfPresent(w http.ResponseWriter, r *http.Request, resolvedPath ipath.Resolved, contentPath ipath.Path, logger *zap.SugaredLogger) (newResolvedPath ipath.Resolved, newContentPath ipath.Path, continueProcessing bool, hadMatchingRule bool) {
redirectsFile := i.getRedirectsFile(r, contentPath, logger)
if redirectsFile != nil {
redirectRules, err := i.getRedirectRules(r, redirectsFile)
if err != nil {
internalWebError(w, err)
return nil, nil, false, true
}
redirected, newPath, err := i.handleRedirectsFileRules(w, r, contentPath, redirectRules)
if err != nil {
err = fmt.Errorf("trouble processing _redirects file at %q: %w", redirectsFile.String(), err)
internalWebError(w, err)
return nil, nil, false, true
}
if redirected {
return nil, nil, false, true
}
// 200 is treated as a rewrite, so update the path and continue
if newPath != "" {
// Reassign contentPath and resolvedPath since the URL was rewritten
contentPath = ipath.New(newPath)
resolvedPath, err = i.api.ResolvePath(r.Context(), contentPath)
if err != nil {
internalWebError(w, err)
return nil, nil, false, true
}
return resolvedPath, contentPath, true, true
}
}
// No matching rule, paths remain the same, continue regular processing
return resolvedPath, contentPath, true, false
}
func (i *gatewayHandler) handleRedirectsFileRules(w http.ResponseWriter, r *http.Request, contentPath ipath.Path, redirectRules []redirects.Rule) (redirected bool, newContentPath string, err error) {
// Attempt to match a rule to the URL path, and perform the corresponding redirect or rewrite
pathParts := strings.Split(contentPath.String(), "/")
if len(pathParts) > 3 {
// All paths should start with /ipfs/cid/, so get the path after that
urlPath := "/" + strings.Join(pathParts[3:], "/")
rootPath := strings.Join(pathParts[:3], "/")
// Trim off the trailing /
urlPath = strings.TrimSuffix(urlPath, "/")
for _, rule := range redirectRules {
// Error right away if the rule is invalid
if !rule.MatchAndExpandPlaceholders(urlPath) {
continue
}
// We have a match!
// Rewrite
if rule.Status == 200 {
// Prepend the rootPath
toPath := rootPath + rule.To
return false, toPath, nil
}
// Or 4xx
if rule.Status == 404 || rule.Status == 410 || rule.Status == 451 {
toPath := rootPath + rule.To
content4xxPath := ipath.New(toPath)
err := i.serve4xx(w, r, content4xxPath, rule.Status)
return true, toPath, err
}
// Or redirect
if rule.Status >= 301 && rule.Status <= 308 {
http.Redirect(w, r, rule.To, rule.Status)
return true, "", nil
}
}
}
// No redirects matched
return false, "", nil
}
func (i *gatewayHandler) getRedirectRules(r *http.Request, redirectsFilePath ipath.Resolved) ([]redirects.Rule, error) {
// Convert the path into a file node
node, err := i.api.Unixfs().Get(r.Context(), redirectsFilePath)
if err != nil {
return nil, fmt.Errorf("could not get _redirects: %w", err)
}
defer node.Close()
// Convert the node into a file
f, ok := node.(files.File)
if !ok {
return nil, fmt.Errorf("could not parse _redirects: %w", err)
}
// Parse redirect rules from file
redirectRules, err := redirects.Parse(f)
if err != nil {
return nil, fmt.Errorf("could not parse _redirects: %w", err)
}
return redirectRules, nil
}
// Returns a resolved path to the _redirects file located in the root CID path of the requested path
func (i *gatewayHandler) getRedirectsFile(r *http.Request, contentPath ipath.Path, logger *zap.SugaredLogger) ipath.Resolved {
// contentPath is the full ipfs path to the requested resource,
// regardless of whether path or subdomain resolution is used.
rootPath := getRootPath(contentPath)
// Check for _redirects file.
// Any path resolution failures are ignored and we just assume there's no _redirects file.
// Note that ignoring these errors also ensures that the use of the empty CID (bafkqaaa) in tests doesn't fail.
path := ipath.Join(rootPath, "_redirects")
resolvedPath, err := i.api.ResolvePath(r.Context(), path)
if err != nil {
return nil
}
return resolvedPath
}
// Returns the root CID Path for the given path
func getRootPath(path ipath.Path) ipath.Path {
parts := strings.Split(path.String(), "/")
return ipath.New(gopath.Join("/", path.Namespace(), parts[2]))
}
func (i *gatewayHandler) serve4xx(w http.ResponseWriter, r *http.Request, content4xxPath ipath.Path, status int) error {
resolved4xxPath, err := i.api.ResolvePath(r.Context(), content4xxPath)
if err != nil {
return err
}
node, err := i.api.Unixfs().Get(r.Context(), resolved4xxPath)
if err != nil {
return err
}
defer node.Close()
f, ok := node.(files.File)
if !ok {
return fmt.Errorf("could not convert node for %d page to file", status)
}
size, err := f.Size()
if err != nil {
return fmt.Errorf("could not get size of %d page", status)
}
log.Debugf("using _redirects: custom %d file at %q", status, content4xxPath)
w.Header().Set("Content-Type", "text/html")
w.Header().Set("Content-Length", strconv.FormatInt(size, 10))
addCacheControlHeaders(w, r, content4xxPath, resolved4xxPath.Cid())
w.WriteHeader(status)
_, err = io.CopyN(w, f, size)
return err
}
func hasOriginIsolation(r *http.Request) bool {
_, gw := r.Context().Value(requestContextKey("gw-hostname")).(string)
_, dnslink := r.Context().Value("dnslink-hostname").(string)
if gw || dnslink {
return true
}
return false
}
func isUnixfsResponseFormat(responseFormat string) bool {
// The implicit response format is UnixFS
return responseFormat == ""
}
// Deprecated: legacy ipfs-404.html files are superseded by _redirects file
// This is provided only for backward-compatibility, until websites migrate
// to 404s managed via _redirects file (https://github.com/ipfs/specs/pull/290)
func (i *gatewayHandler) serveLegacy404IfPresent(w http.ResponseWriter, r *http.Request, contentPath ipath.Path) bool {
resolved404Path, ctype, err := i.searchUpTreeFor404(r, contentPath)
if err != nil {
return false
}
dr, err := i.api.Unixfs().Get(r.Context(), resolved404Path)
if err != nil {
return false
}
defer dr.Close()
f, ok := dr.(files.File)
if !ok {
return false
}
size, err := f.Size()
if err != nil {
return false
}
log.Debugw("using pretty 404 file", "path", contentPath)
w.Header().Set("Content-Type", ctype)
w.Header().Set("Content-Length", strconv.FormatInt(size, 10))
w.WriteHeader(http.StatusNotFound)
_, err = io.CopyN(w, f, size)
return err == nil
}
func (i *gatewayHandler) searchUpTreeFor404(r *http.Request, contentPath ipath.Path) (ipath.Resolved, string, error) {
filename404, ctype, err := preferred404Filename(r.Header.Values("Accept"))
if err != nil {
return nil, "", err
}
pathComponents := strings.Split(contentPath.String(), "/")
for idx := len(pathComponents); idx >= 3; idx-- {
pretty404 := gopath.Join(append(pathComponents[0:idx], filename404)...)
parsed404Path := ipath.New("/" + pretty404)
if parsed404Path.IsValid() != nil {
break
}
resolvedPath, err := i.api.ResolvePath(r.Context(), parsed404Path)
if err != nil {
continue
}
return resolvedPath, ctype, nil
}
return nil, "", fmt.Errorf("no pretty 404 in any parent folder")
}
func preferred404Filename(acceptHeaders []string) (string, string, error) {
// If we ever want to offer a 404 file for a different content type
// then this function will need to parse q weightings, but for now
// the presence of anything matching HTML is enough.
for _, acceptHeader := range acceptHeaders {
accepted := strings.Split(acceptHeader, ",")
for _, spec := range accepted {
contentType := strings.SplitN(spec, ";", 1)[0]
switch contentType {
case "*/*", "text/*", "text/html":
return "ipfs-404.html", "text/html", nil
}
}
}
return "", "", fmt.Errorf("there is no 404 file for the requested content types")
}

2
core/corehttp/gateway_handler_unixfs_dir.go
View File

@ -185,7 +185,7 @@ func (i *gatewayHandler) serveDirectory(ctx context.Context, w http.ResponseWrit
var gwURL string
// Get gateway hostname and build gateway URL.
if h, ok := r.Context().Value("gw-hostname").(string); ok {
if h, ok := r.Context().Value(requestContextKey("gw-hostname")).(string); ok {
gwURL = "//" + h
} else {
gwURL = ""

9
core/corehttp/gateway_handler_unixfs_file.go
View File

@ -37,6 +37,15 @@ func (i *gatewayHandler) serveFile(ctx context.Context, w http.ResponseWriter, r
return
}
if size == 0 {
// We override null files to 200 to avoid issues with fragment caching reverse proxies.
// Also whatever you are asking for, it's cheaper to just give you the complete file (nothing).
// TODO: remove this if clause once https://github.com/golang/go/issues/54794 is fixed in two latest releases of go
w.Header().Set("Content-Type", "text/plain")
w.WriteHeader(http.StatusOK)
return
}
// Lazy seeker enables efficient range-requests and HTTP HEAD responses
content := &lazySeeker{
size: size,

4
core/corehttp/gateway_test.go
View File

@ -25,7 +25,7 @@ import (
nsopts "github.com/ipfs/interface-go-ipfs-core/options/namesys"
ipath "github.com/ipfs/interface-go-ipfs-core/path"
config "github.com/ipfs/kubo/config"
ci "github.com/libp2p/go-libp2p-core/crypto"
ci "github.com/libp2p/go-libp2p/core/crypto"
id "github.com/libp2p/go-libp2p/p2p/protocol/identify"
)
@ -652,7 +652,7 @@ func TestVersion(t *testing.T) {
t.Fatalf("response doesn't contain client version:\n%s", s)
}
if !strings.Contains(s, "Protocol Version: "+id.LibP2PVersion) {
if !strings.Contains(s, "Protocol Version: "+id.DefaultProtocolVersion) {
t.Fatalf("response doesn't contain protocol version:\n%s", s)
}
}

9
core/corehttp/hostname.go
View File

@ -13,7 +13,7 @@ import (
namesys "github.com/ipfs/go-namesys"
core "github.com/ipfs/kubo/core"
coreapi "github.com/ipfs/kubo/core/coreapi"
"github.com/libp2p/go-libp2p-core/peer"
"github.com/libp2p/go-libp2p/core/peer"
dns "github.com/miekg/dns"
mbase "github.com/multiformats/go-multibase"
@ -221,7 +221,8 @@ func HostnameOption() ServeOption {
if !cfg.Gateway.NoDNSLink && isDNSLinkName(r.Context(), coreAPI, host) {
// rewrite path and handle as DNSLink
r.URL.Path = "/ipns/" + stripPort(host) + r.URL.Path
childMux.ServeHTTP(w, withHostnameContext(r, host))
ctx := context.WithValue(r.Context(), requestContextKey("dnslink-hostname"), host)
childMux.ServeHTTP(w, withHostnameContext(r.WithContext(ctx), host))
return
}
@ -242,6 +243,8 @@ type wildcardHost struct {
spec *config.GatewaySpec
}
type requestContextKey string
// Extends request context to include hostname of a canonical gateway root
// (subdomain root or dnslink fqdn)
func withHostnameContext(r *http.Request, hostname string) *http.Request {
@ -250,7 +253,7 @@ func withHostnameContext(r *http.Request, hostname string) *http.Request {
// Host header, subdomain gateways have more comples rules (knownSubdomainDetails)
// More: https://github.com/ipfs/dir-index-html/issues/42
// nolint: staticcheck // non-backward compatible change
ctx := context.WithValue(r.Context(), "gw-hostname", hostname)
ctx := context.WithValue(r.Context(), requestContextKey("gw-hostname"), hostname)
return r.WithContext(ctx)
}

2
core/corehttp/metrics_test.go
View File

@ -7,7 +7,7 @@ import (
"github.com/ipfs/kubo/core"
inet "github.com/libp2p/go-libp2p-core/network"
inet "github.com/libp2p/go-libp2p/core/network"
bhost "github.com/libp2p/go-libp2p/p2p/host/basic"
swarmt "github.com/libp2p/go-libp2p/p2p/net/swarm/testing"
)

4
core/corehttp/p2p_proxy.go
View File

@ -9,10 +9,10 @@ import (
"strings"
core "github.com/ipfs/kubo/core"
peer "github.com/libp2p/go-libp2p-core/peer"
peer "github.com/libp2p/go-libp2p/core/peer"
protocol "github.com/libp2p/go-libp2p-core/protocol"
p2phttp "github.com/libp2p/go-libp2p-http"
protocol "github.com/libp2p/go-libp2p/core/protocol"
)
// P2PProxyOption is an endpoint for proxying a HTTP request to another ipfs peer

2
core/corehttp/p2p_proxy_test.go
View File

@ -7,7 +7,7 @@ import (
"github.com/ipfs/kubo/thirdparty/assert"
protocol "github.com/libp2p/go-libp2p-core/protocol"
protocol "github.com/libp2p/go-libp2p/core/protocol"
)
type TestCase struct {

4
core/corehttp/webui.go
View File

@ -1,11 +1,13 @@
package corehttp
// TODO: move to IPNS
const WebUIPath = "/ipfs/bafybeibozpulxtpv5nhfa2ue3dcjx23ndh3gwr5vwllk7ptoyfwnfjjr4q" // v2.15.1
const WebUIPath = "/ipfs/bafybeiageaoxg6d7npaof6eyzqbwvbubyler7bq44hayik2hvqcggg7d2y" // v2.18.1
// this is a list of all past webUI paths.
var WebUIPaths = []string{
WebUIPath,
"/ipfs/bafybeidb5eryh72zajiokdggzo7yct2d6hhcflncji5im2y5w26uuygdsm",
"/ipfs/bafybeibozpulxtpv5nhfa2ue3dcjx23ndh3gwr5vwllk7ptoyfwnfjjr4q",
"/ipfs/bafybeiednzu62vskme5wpoj4bjjikeg3xovfpp4t7vxk5ty2jxdi4mv4bu",
"/ipfs/bafybeihcyruaeza7uyjd6ugicbcrqumejf6uf353e5etdkhotqffwtguva",
"/ipfs/bafybeiflkjt66aetfgcrgvv75izymd5kc47g6luepqmfq6zsf5w6ueth6y",

6
core/mock/mock.go
View File

@ -16,10 +16,10 @@ import (
config "github.com/ipfs/kubo/config"
"github.com/libp2p/go-libp2p"
"github.com/libp2p/go-libp2p-core/host"
"github.com/libp2p/go-libp2p-core/peer"
pstore "github.com/libp2p/go-libp2p-core/peerstore"
testutil "github.com/libp2p/go-libp2p-testing/net"
"github.com/libp2p/go-libp2p/core/host"
"github.com/libp2p/go-libp2p/core/peer"
pstore "github.com/libp2p/go-libp2p/core/peerstore"
mocknet "github.com/libp2p/go-libp2p/p2p/net/mock"
)

44
core/node/bitswap.go
View File

@ -7,27 +7,32 @@ import (
"github.com/ipfs/go-bitswap/network"
blockstore "github.com/ipfs/go-ipfs-blockstore"
exchange "github.com/ipfs/go-ipfs-exchange-interface"
config "github.com/ipfs/kubo/config"
"github.com/ipfs/kubo/config"
irouting "github.com/ipfs/kubo/routing"
"github.com/libp2p/go-libp2p-core/host"
"github.com/libp2p/go-libp2p/core/host"
"go.uber.org/fx"
"github.com/ipfs/kubo/core/node/helpers"
)
// Docs: https://github.com/ipfs/kubo/blob/master/docs/config.md#internalbitswap
const (
// Docs: https://github.com/ipfs/kubo/blob/master/docs/config.md#internalbitswap
DefaultEngineBlockstoreWorkerCount = 128
DefaultTaskWorkerCount = 8
DefaultEngineTaskWorkerCount = 8
DefaultMaxOutstandingBytesPerPeer = 1 << 20
)
// OnlineExchange creates new LibP2P backed block exchange (BitSwap)
func OnlineExchange(cfg *config.Config, provide bool) interface{} {
return func(mctx helpers.MetricsCtx, lc fx.Lifecycle, host host.Host, rt irouting.TieredRouter, bs blockstore.GCBlockstore) exchange.Interface {
bitswapNetwork := network.NewFromIpfsHost(host, rt)
type bitswapOptionsOut struct {
fx.Out
BitswapOpts []bitswap.Option `group:"bitswap-options,flatten"`
}
// BitswapOptions creates configuration options for Bitswap from the config file
// and whether to provide data.
func BitswapOptions(cfg *config.Config, provide bool) interface{} {
return func() bitswapOptionsOut {
var internalBsCfg config.InternalBitswap
if cfg.Internal.Bitswap != nil {
internalBsCfg = *cfg.Internal.Bitswap
@ -40,13 +45,34 @@ func OnlineExchange(cfg *config.Config, provide bool) interface{} {
bitswap.EngineTaskWorkerCount(int(internalBsCfg.EngineTaskWorkerCount.WithDefault(DefaultEngineTaskWorkerCount))),
bitswap.MaxOutstandingBytesPerPeer(int(internalBsCfg.MaxOutstandingBytesPerPeer.WithDefault(DefaultMaxOutstandingBytesPerPeer))),
}
exch := bitswap.New(helpers.LifecycleCtx(mctx, lc), bitswapNetwork, bs, opts...)
return bitswapOptionsOut{BitswapOpts: opts}
}
}
type onlineExchangeIn struct {
fx.In
Mctx helpers.MetricsCtx
Host host.Host
Rt irouting.ProvideManyRouter
Bs blockstore.GCBlockstore
BitswapOpts []bitswap.Option `group:"bitswap-options"`
}
// OnlineExchange creates new LibP2P backed block exchange (BitSwap).
// Additional options to bitswap.New can be provided via the "bitswap-options"
// group.
func OnlineExchange() interface{} {
return func(in onlineExchangeIn, lc fx.Lifecycle) exchange.Interface {
bitswapNetwork := network.NewFromIpfsHost(in.Host, in.Rt)
exch := bitswap.New(helpers.LifecycleCtx(in.Mctx, lc), bitswapNetwork, in.Bs, in.BitswapOpts...)
lc.Append(fx.Hook{
OnStop: func(ctx context.Context) error {
return exch.Close()
},
})
return exch
}
}

4
core/node/builder.go
View File

@ -15,8 +15,8 @@ import (
ds "github.com/ipfs/go-datastore"
dsync "github.com/ipfs/go-datastore/sync"
cfg "github.com/ipfs/kubo/config"
"github.com/libp2p/go-libp2p-core/crypto"
peer "github.com/libp2p/go-libp2p-core/peer"
"github.com/libp2p/go-libp2p/core/crypto"
peer "github.com/libp2p/go-libp2p/core/peer"
)
type BuildCfg struct {

2
core/node/graphsync.go
View File

@ -6,7 +6,7 @@ import (
"github.com/ipfs/go-graphsync/network"
"github.com/ipfs/go-graphsync/storeutil"
blockstore "github.com/ipfs/go-ipfs-blockstore"
libp2p "github.com/libp2p/go-libp2p-core"
libp2p "github.com/libp2p/go-libp2p/core"
"go.uber.org/fx"
"github.com/ipfs/kubo/core/node/helpers"

12
core/node/groups.go
View File

@ -10,8 +10,8 @@ import (
util "github.com/ipfs/go-ipfs-util"
"github.com/ipfs/go-log"
"github.com/ipfs/kubo/config"
"github.com/libp2p/go-libp2p-core/peer"
pubsub "github.com/libp2p/go-libp2p-pubsub"
"github.com/libp2p/go-libp2p/core/peer"
"github.com/ipfs/kubo/core/node/libp2p"
"github.com/ipfs/kubo/p2p"
@ -142,13 +142,10 @@ func LibP2P(bcfg *BuildCfg, cfg *config.Config) fx.Option {
"If you want to continue running a circuit v1 relay, please use the standalone relay daemon: https://dist.ipfs.tech/#libp2p-relay-daemon (with RelayV1.Enabled: true)")
}
peerChan := make(libp2p.AddrInfoChan)
// Gather all the options
opts := fx.Options(
BaseLibP2P,
fx.Supply(peerChan),
// Services (resource management)
fx.Provide(libp2p.ResourceManager(cfg.Swarm)),
fx.Provide(libp2p.AddrFilters(cfg.Swarm.AddrFilters)),
@ -168,13 +165,11 @@ func LibP2P(bcfg *BuildCfg, cfg *config.Config) fx.Option {
fx.Provide(libp2p.ContentRouting),
fx.Provide(libp2p.BaseRouting(cfg.Experimental.AcceleratedDHTClient)),
fx.Provide(libp2p.DelegatedRouting(cfg.Routing.Routers)),
maybeProvide(libp2p.PubsubRouter, bcfg.getOpt("ipnsps")),
maybeProvide(libp2p.BandwidthCounter, !cfg.Swarm.DisableBandwidthMetrics),
maybeProvide(libp2p.NatPortMap, !cfg.Swarm.DisableNatPortMap),
maybeProvide(libp2p.AutoRelay(cfg.Swarm.RelayClient.StaticRelays, peerChan), enableRelayClient),
maybeInvoke(libp2p.AutoRelayFeeder(cfg.Peering), enableRelayClient),
libp2p.MaybeAutoRelay(cfg.Swarm.RelayClient.StaticRelays, cfg.Peering, enableRelayClient),
autonat,
connmgr,
ps,
@ -293,7 +288,8 @@ func Online(bcfg *BuildCfg, cfg *config.Config) fx.Option {
shouldBitswapProvide := !cfg.Experimental.StrategicProviding
return fx.Options(
fx.Provide(OnlineExchange(cfg, shouldBitswapProvide)),
fx.Provide(BitswapOptions(cfg, shouldBitswapProvide)),
fx.Provide(OnlineExchange()),
maybeProvide(Graphsync, cfg.Experimental.GraphsyncEnabled),
fx.Provide(DNSResolver),
fx.Provide(Namesys(ipnsCacheSize)),

2
core/node/helpers.go
View File

@ -43,7 +43,7 @@ func maybeProvide(opt interface{}, enable bool) fx.Option {
return fx.Options()
}
//nolint unused
// nolint unused
func maybeInvoke(opt interface{}, enable bool) fx.Option {
if enable {
return fx.Invoke(opt)

4
core/node/identity.go
View File

@ -3,8 +3,8 @@ package node
import (
"fmt"
"github.com/libp2p/go-libp2p-core/crypto"
"github.com/libp2p/go-libp2p-core/peer"
"github.com/libp2p/go-libp2p/core/crypto"
"github.com/libp2p/go-libp2p/core/peer"
)
func PeerID(id peer.ID) func() peer.ID {

11
core/node/ipns.go
View File

@ -6,16 +6,15 @@ import (
util "github.com/ipfs/go-ipfs-util"
"github.com/ipfs/go-ipns"
"github.com/libp2p/go-libp2p-core/crypto"
"github.com/libp2p/go-libp2p-core/peerstore"
record "github.com/libp2p/go-libp2p-record"
"github.com/libp2p/go-libp2p/core/crypto"
"github.com/libp2p/go-libp2p/core/peerstore"
madns "github.com/multiformats/go-multiaddr-dns"
irouting "github.com/ipfs/kubo/routing"
"github.com/ipfs/go-namesys"
"github.com/ipfs/go-namesys/republisher"
"github.com/ipfs/kubo/repo"
irouting "github.com/ipfs/kubo/routing"
)
const DefaultIpnsCacheSize = 128
@ -29,8 +28,8 @@ func RecordValidator(ps peerstore.Peerstore) record.Validator {
}
// Namesys creates new name system
func Namesys(cacheSize int) func(rt irouting.TieredRouter, rslv *madns.Resolver, repo repo.Repo) (namesys.NameSystem, error) {
return func(rt irouting.TieredRouter, rslv *madns.Resolver, repo repo.Repo) (namesys.NameSystem, error) {
func Namesys(cacheSize int) func(rt irouting.ProvideManyRouter, rslv *madns.Resolver, repo repo.Repo) (namesys.NameSystem, error) {
return func(rt irouting.ProvideManyRouter, rslv *madns.Resolver, repo repo.Repo) (namesys.NameSystem, error) {
opts := []namesys.Option{
namesys.WithDatastore(repo.Datastore()),
namesys.WithDNSResolver(rslv),

2
core/node/libp2p/addrs.go
View File

@ -4,7 +4,7 @@ import (
"fmt"
"github.com/libp2p/go-libp2p"
"github.com/libp2p/go-libp2p-core/host"
"github.com/libp2p/go-libp2p/core/host"
p2pbhost "github.com/libp2p/go-libp2p/p2p/host/basic"
ma "github.com/multiformats/go-multiaddr"
mamask "github.com/whyrusleeping/multiaddr-filter"

4
core/node/libp2p/discovery.go
View File

@ -4,8 +4,8 @@ import (
"context"
"time"
"github.com/libp2p/go-libp2p-core/host"
"github.com/libp2p/go-libp2p-core/peer"
"github.com/libp2p/go-libp2p/core/host"
"github.com/libp2p/go-libp2p/core/peer"
"github.com/libp2p/go-libp2p/p2p/discovery/mdns"
"go.uber.org/fx"

8
core/node/libp2p/filters.go
View File

@ -1,10 +1,10 @@
package libp2p
import (
"github.com/libp2p/go-libp2p-core/connmgr"
"github.com/libp2p/go-libp2p-core/control"
"github.com/libp2p/go-libp2p-core/network"
"github.com/libp2p/go-libp2p-core/peer"
"github.com/libp2p/go-libp2p/core/connmgr"
"github.com/libp2p/go-libp2p/core/control"
"github.com/libp2p/go-libp2p/core/network"
"github.com/libp2p/go-libp2p/core/peer"
ma "github.com/multiformats/go-multiaddr"
)

8
core/node/libp2p/host.go
View File

@ -4,11 +4,11 @@ import (
"context"
"github.com/libp2p/go-libp2p"
"github.com/libp2p/go-libp2p-core/host"
"github.com/libp2p/go-libp2p-core/peer"
"github.com/libp2p/go-libp2p-core/peerstore"
"github.com/libp2p/go-libp2p-core/routing"
record "github.com/libp2p/go-libp2p-record"
"github.com/libp2p/go-libp2p/core/host"
"github.com/libp2p/go-libp2p/core/peer"
"github.com/libp2p/go-libp2p/core/peerstore"
"github.com/libp2p/go-libp2p/core/routing"
routedhost "github.com/libp2p/go-libp2p/p2p/host/routed"
"github.com/ipfs/kubo/core/node/helpers"

6
core/node/libp2p/hostopt.go
View File

@ -4,9 +4,9 @@ import (
"fmt"
"github.com/libp2p/go-libp2p"
"github.com/libp2p/go-libp2p-core/host"
"github.com/libp2p/go-libp2p-core/peer"
"github.com/libp2p/go-libp2p-core/peerstore"
"github.com/libp2p/go-libp2p/core/host"
"github.com/libp2p/go-libp2p/core/peer"
"github.com/libp2p/go-libp2p/core/peerstore"
)
type HostOption func(id peer.ID, ps peerstore.Peerstore, options ...libp2p.Option) (host.Host, error)

6
core/node/libp2p/libp2p.go
View File

@ -10,9 +10,9 @@ import (
logging "github.com/ipfs/go-log"
"github.com/libp2p/go-libp2p"
"github.com/libp2p/go-libp2p-core/crypto"
"github.com/libp2p/go-libp2p-core/peer"
"github.com/libp2p/go-libp2p-core/peerstore"
"github.com/libp2p/go-libp2p/core/crypto"
"github.com/libp2p/go-libp2p/core/peer"
"github.com/libp2p/go-libp2p/core/peerstore"
"github.com/libp2p/go-libp2p/p2p/net/connmgr"
"go.uber.org/fx"
)

4
core/node/libp2p/peerstore.go
View File

@ -3,8 +3,8 @@ package libp2p
import (
"context"
"github.com/libp2p/go-libp2p-core/peerstore"
"github.com/libp2p/go-libp2p-peerstore/pstoremem"
"github.com/libp2p/go-libp2p/core/peerstore"
"github.com/libp2p/go-libp2p/p2p/host/peerstore/pstoremem"
"go.uber.org/fx"
)

4
core/node/libp2p/pnet.go
View File

@ -9,8 +9,8 @@ import (
"github.com/ipfs/kubo/repo"
"github.com/libp2p/go-libp2p"
"github.com/libp2p/go-libp2p-core/host"
"github.com/libp2p/go-libp2p-core/pnet"
"github.com/libp2p/go-libp2p/core/host"
"github.com/libp2p/go-libp2p/core/pnet"
"go.uber.org/fx"
"golang.org/x/crypto/salsa20"
"golang.org/x/crypto/sha3"

4
core/node/libp2p/pubsub.go
View File

@ -1,9 +1,9 @@
package libp2p
import (
"github.com/libp2p/go-libp2p-core/discovery"
"github.com/libp2p/go-libp2p-core/host"
pubsub "github.com/libp2p/go-libp2p-pubsub"
"github.com/libp2p/go-libp2p/core/discovery"
"github.com/libp2p/go-libp2p/core/host"
"go.uber.org/fx"
"github.com/ipfs/kubo/core/node/helpers"

10
core/node/libp2p/rcmgr.go
View File

@ -14,11 +14,11 @@ import (
"github.com/ipfs/kubo/repo"
"github.com/libp2p/go-libp2p"
"github.com/libp2p/go-libp2p-core/network"
"github.com/libp2p/go-libp2p-core/peer"
"github.com/libp2p/go-libp2p-core/protocol"
rcmgr "github.com/libp2p/go-libp2p-resource-manager"
rcmgrObs "github.com/libp2p/go-libp2p-resource-manager/obs"
"github.com/libp2p/go-libp2p/core/network"
"github.com/libp2p/go-libp2p/core/peer"
"github.com/libp2p/go-libp2p/core/protocol"
rcmgr "github.com/libp2p/go-libp2p/p2p/host/resource-manager"
rcmgrObs "github.com/libp2p/go-libp2p/p2p/host/resource-manager/obs"
"github.com/multiformats/go-multiaddr"
"go.opencensus.io/stats/view"

8
core/node/libp2p/rcmgr_defaults.go
View File

@ -9,7 +9,7 @@ import (
"github.com/ipfs/kubo/config"
"github.com/libp2p/go-libp2p"
rcmgr "github.com/libp2p/go-libp2p-resource-manager"
rcmgr "github.com/libp2p/go-libp2p/p2p/host/resource-manager"
"github.com/wI2L/jsondiff"
)
@ -155,7 +155,7 @@ func checkImplicitDefaults() {
}
if !ok {
log.Errorf("===> OOF! go-libp2p reduced DefaultServiceLimits\n" +
log.Errorf("===> OOF! go-libp2p changed DefaultServiceLimits\n" +
"=> See the aboce reduced values for info.\n" +
"=> go-libp2p SetDefaultServiceLimits update needs a review:\n" +
"Please inspect if changes impact go-ipfs users, and update expectedDefaultServiceLimits in rcmgr_defaults.go to remove this message",
@ -181,7 +181,7 @@ func jsonDiff(old []byte, updated []byte) ([]string, error) {
return changes, nil
}
// https://github.com/libp2p/go-libp2p-resource-manager/blob/v0.1.5/limit_defaults.go#L49
// https://github.com/libp2p/go-libp2p/blob/v0.22.0/p2p/host/resource-manager/limit_defaults.go#L343
const expectedDefaultLimits = `{
"SystemBaseLimit": {
"Streams": 2048,
@ -376,7 +376,7 @@ const expectedDefaultLimits = `{
"ConnsInbound": 1,
"ConnsOutbound": 1,
"FD": 1,
"Memory": 1048576
"Memory": 33554432
},
"ConnLimitIncrease": {
"Streams": 0,

8
core/node/libp2p/rcmgr_logging.go
View File

@ -7,10 +7,10 @@ import (
"time"
"github.com/benbjohnson/clock"
"github.com/libp2p/go-libp2p-core/network"
"github.com/libp2p/go-libp2p-core/peer"
"github.com/libp2p/go-libp2p-core/protocol"
rcmgr "github.com/libp2p/go-libp2p-resource-manager"
"github.com/libp2p/go-libp2p/core/network"
"github.com/libp2p/go-libp2p/core/peer"
"github.com/libp2p/go-libp2p/core/protocol"
rcmgr "github.com/libp2p/go-libp2p/p2p/host/resource-manager"
ma "github.com/multiformats/go-multiaddr"
"go.uber.org/zap"
)

4
core/node/libp2p/rcmgr_logging_test.go
View File

@ -6,8 +6,8 @@ import (
"time"
"github.com/benbjohnson/clock"
"github.com/libp2p/go-libp2p-core/network"
rcmgr "github.com/libp2p/go-libp2p-resource-manager"
"github.com/libp2p/go-libp2p/core/network"
rcmgr "github.com/libp2p/go-libp2p/p2p/host/resource-manager"
ma "github.com/multiformats/go-multiaddr"
"github.com/stretchr/testify/require"
"go.uber.org/zap"

8
core/node/libp2p/rcmgr_metrics.go
View File

@ -4,10 +4,10 @@ import (
"errors"
"strconv"
"github.com/libp2p/go-libp2p-core/network"
"github.com/libp2p/go-libp2p-core/peer"
"github.com/libp2p/go-libp2p-core/protocol"
rcmgr "github.com/libp2p/go-libp2p-resource-manager"
"github.com/libp2p/go-libp2p/core/network"
"github.com/libp2p/go-libp2p/core/peer"
"github.com/libp2p/go-libp2p/core/protocol"
rcmgr "github.com/libp2p/go-libp2p/p2p/host/resource-manager"
"github.com/prometheus/client_golang/prometheus"
)

82
core/node/libp2p/relay.go
View File

@ -1,11 +1,14 @@
package libp2p
import (
"context"
"github.com/ipfs/kubo/config"
"github.com/libp2p/go-libp2p"
"github.com/libp2p/go-libp2p-core/peer"
"github.com/libp2p/go-libp2p/core/peer"
"github.com/libp2p/go-libp2p/p2p/host/autorelay"
"github.com/libp2p/go-libp2p/p2p/protocol/circuitv2/relay"
"go.uber.org/fx"
)
func RelayTransport(enableRelay bool) func() (opts Libp2pOpts, err error) {
@ -43,28 +46,63 @@ func RelayService(enable bool, relayOpts config.RelayService) func() (opts Libp2
}
}
func AutoRelay(staticRelays []string, peerChan <-chan peer.AddrInfo) func() (opts Libp2pOpts, err error) {
return func() (opts Libp2pOpts, err error) {
var autoRelayOpts []autorelay.Option
if len(staticRelays) > 0 {
static := make([]peer.AddrInfo, 0, len(staticRelays))
for _, s := range staticRelays {
var addr *peer.AddrInfo
addr, err = peer.AddrInfoFromString(s)
if err != nil {
return
}
static = append(static, *addr)
}
autoRelayOpts = append(autoRelayOpts, autorelay.WithStaticRelays(static))
autoRelayOpts = append(autoRelayOpts, autorelay.WithCircuitV1Support())
}
if peerChan != nil {
autoRelayOpts = append(autoRelayOpts, autorelay.WithPeerSource(peerChan))
}
opts.Opts = append(opts.Opts, libp2p.EnableAutoRelay(autoRelayOpts...))
return
func MaybeAutoRelay(staticRelays []string, cfgPeering config.Peering, enabled bool) fx.Option {
if !enabled {
return fx.Options()
}
if len(staticRelays) > 0 {
return fx.Provide(func() (opts Libp2pOpts, err error) {
if len(staticRelays) > 0 {
static := make([]peer.AddrInfo, 0, len(staticRelays))
for _, s := range staticRelays {
var addr *peer.AddrInfo
addr, err = peer.AddrInfoFromString(s)
if err != nil {
return
}
static = append(static, *addr)
}
opts.Opts = append(opts.Opts, libp2p.EnableAutoRelay(
autorelay.WithStaticRelays(static),
autorelay.WithCircuitV1Support(),
))
}
return
})
}
peerChan := make(chan peer.AddrInfo)
return fx.Options(
// Provide AutoRelay option
fx.Provide(func() (opts Libp2pOpts, err error) {
opts.Opts = append(opts.Opts, libp2p.EnableAutoRelay(autorelay.WithPeerSource(func(ctx context.Context, numPeers int) <-chan peer.AddrInfo {
// TODO(9257): make this code smarter (have a state and actually try to grow the search outward) instead of a long running task just polling our K cluster.
r := make(chan peer.AddrInfo)
go func() {
defer close(r)
for ; numPeers != 0; numPeers-- {
select {
case v, ok := <-peerChan:
if !ok {
return
}
select {
case r <- v:
case <-ctx.Done():
return
}
case <-ctx.Done():
return
}
}
}()
return r
}, 0)))
return
}),
autoRelayFeeder(cfgPeering, peerChan),
)
}
func HolePunching(flag config.Flag, hasRelayClient bool) func() (opts Libp2pOpts, err error) {

64
core/node/libp2p/routing.go
View File

@ -14,9 +14,6 @@ import (
offroute "github.com/ipfs/go-ipfs-routing/offline"
config "github.com/ipfs/kubo/config"
"github.com/ipfs/kubo/repo"
"github.com/libp2p/go-libp2p-core/host"
"github.com/libp2p/go-libp2p-core/peer"
"github.com/libp2p/go-libp2p-core/routing"
dht "github.com/libp2p/go-libp2p-kad-dht"
ddht "github.com/libp2p/go-libp2p-kad-dht/dual"
"github.com/libp2p/go-libp2p-kad-dht/fullrt"
@ -24,6 +21,9 @@ import (
namesys "github.com/libp2p/go-libp2p-pubsub-router"
record "github.com/libp2p/go-libp2p-record"
routinghelpers "github.com/libp2p/go-libp2p-routing-helpers"
"github.com/libp2p/go-libp2p/core/host"
"github.com/libp2p/go-libp2p/core/peer"
"github.com/libp2p/go-libp2p/core/routing"
"github.com/cenkalti/backoff/v4"
"go.uber.org/fx"
@ -129,39 +129,6 @@ func BaseRouting(experimentalDHTClient bool) interface{} {
}
}
type delegatedRouterOut struct {
fx.Out
Routers []Router `group:"routers,flatten"`
ContentRouter []routing.ContentRouting `group:"content-routers,flatten"`
}
func DelegatedRouting(routers map[string]config.Router) interface{} {
return func() (delegatedRouterOut, error) {
out := delegatedRouterOut{}
for _, v := range routers {
if !v.Enabled.WithDefault(true) {
continue
}
r, err := irouting.RoutingFromConfig(v)
if err != nil {
return out, err
}
out.Routers = append(out.Routers, Router{
Routing: r,
Priority: irouting.GetPriority(v.Parameters),
})
out.ContentRouter = append(out.ContentRouter, r)
}
return out, nil
}
}
type p2pOnlineContentRoutingIn struct {
fx.In
@ -195,24 +162,23 @@ type p2pOnlineRoutingIn struct {
// Routing will get all routers obtained from different methods
// (delegated routers, pub-sub, and so on) and add them all together
// using a TieredRouter.
func Routing(in p2pOnlineRoutingIn) irouting.TieredRouter {
func Routing(in p2pOnlineRoutingIn) irouting.ProvideManyRouter {
routers := in.Routers
sort.SliceStable(routers, func(i, j int) bool {
return routers[i].Priority < routers[j].Priority
})
irouters := make([]routing.Routing, len(routers))
for i, v := range routers {
irouters[i] = v.Routing
var cRouters []*routinghelpers.ParallelRouter
for _, v := range routers {
cRouters = append(cRouters, &routinghelpers.ParallelRouter{
Timeout: 5 * time.Minute,
IgnoreError: true,
Router: v.Routing,
})
}
return irouting.Tiered{
Tiered: routinghelpers.Tiered{
Routers: irouters,
Validator: in.Validator,
},
}
return routinghelpers.NewComposableParallel(cRouters)
}
// OfflineRouting provides a special Router to the routers list when we are creating a offline node.
@ -259,8 +225,8 @@ func PubsubRouter(mctx helpers.MetricsCtx, lc fx.Lifecycle, in p2pPSRoutingIn) (
}, psRouter, nil
}
func AutoRelayFeeder(cfgPeering config.Peering) func(fx.Lifecycle, host.Host, AddrInfoChan, *ddht.DHT) {
return func(lc fx.Lifecycle, h host.Host, peerChan AddrInfoChan, dht *ddht.DHT) {
func autoRelayFeeder(cfgPeering config.Peering, peerChan chan<- peer.AddrInfo) fx.Option {
return fx.Invoke(func(lc fx.Lifecycle, h host.Host, dht *ddht.DHT) {
ctx, cancel := context.WithCancel(context.Background())
done := make(chan struct{})
@ -333,5 +299,5 @@ func AutoRelayFeeder(cfgPeering config.Peering) func(fx.Lifecycle, host.Host, Ad
return nil
},
})
}
})
}

39
core/node/libp2p/routingopt.go
View File

@ -2,14 +2,17 @@ package libp2p
import (
"context"
"github.com/ipfs/go-datastore"
host "github.com/libp2p/go-libp2p-core/host"
"github.com/libp2p/go-libp2p-core/peer"
routing "github.com/libp2p/go-libp2p-core/routing"
"github.com/ipfs/kubo/config"
irouting "github.com/ipfs/kubo/routing"
dht "github.com/libp2p/go-libp2p-kad-dht"
dual "github.com/libp2p/go-libp2p-kad-dht/dual"
record "github.com/libp2p/go-libp2p-record"
routinghelpers "github.com/libp2p/go-libp2p-routing-helpers"
host "github.com/libp2p/go-libp2p/core/host"
"github.com/libp2p/go-libp2p/core/peer"
routing "github.com/libp2p/go-libp2p/core/routing"
)
type RoutingOption func(
@ -46,6 +49,36 @@ func constructDHTRouting(mode dht.ModeOpt) func(
}
}
func ConstructDelegatedRouting(routers config.Routers, methods config.Methods, peerID string, addrs []string, privKey string) func(
ctx context.Context,
host host.Host,
dstore datastore.Batching,
validator record.Validator,
bootstrapPeers ...peer.AddrInfo,
) (routing.Routing, error) {
return func(
ctx context.Context,
host host.Host,
dstore datastore.Batching,
validator record.Validator,
bootstrapPeers ...peer.AddrInfo,
) (routing.Routing, error) {
return irouting.Parse(routers, methods,
&irouting.ExtraDHTParams{
BootstrapPeers: bootstrapPeers,
Host: host,
Validator: validator,
Datastore: dstore,
Context: ctx,
},
&irouting.ExtraReframeParams{
PeerID: peerID,
Addrs: addrs,
PrivKeyB64: privKey,
})
}
}
func constructNilRouting(
ctx context.Context,
host host.Host,

2
core/node/libp2p/smux.go
View File

@ -8,7 +8,7 @@ import (
"github.com/ipfs/kubo/config"
"github.com/libp2p/go-libp2p"
"github.com/libp2p/go-libp2p-core/network"
"github.com/libp2p/go-libp2p/core/network"
"github.com/libp2p/go-libp2p/p2p/muxer/mplex"
"github.com/libp2p/go-libp2p/p2p/muxer/yamux"
)

6
core/node/libp2p/topicdiscovery.go
View File

@ -4,12 +4,12 @@ import (
"math/rand"
"time"
"github.com/libp2p/go-libp2p-core/discovery"
"github.com/libp2p/go-libp2p-core/host"
"github.com/libp2p/go-libp2p/core/discovery"
"github.com/libp2p/go-libp2p/core/host"
"github.com/libp2p/go-libp2p/p2p/discovery/backoff"
disc "github.com/libp2p/go-libp2p/p2p/discovery/routing"
"github.com/libp2p/go-libp2p-core/routing"
"github.com/libp2p/go-libp2p/core/routing"
)
func TopicDiscovery() interface{} {

24
core/node/libp2p/transport.go
View File

@ -5,10 +5,11 @@ import (
"github.com/ipfs/kubo/config"
"github.com/libp2p/go-libp2p"
"github.com/libp2p/go-libp2p-core/metrics"
libp2pquic "github.com/libp2p/go-libp2p/p2p/transport/quic"
"github.com/libp2p/go-libp2p/core/metrics"
quic "github.com/libp2p/go-libp2p/p2p/transport/quic"
"github.com/libp2p/go-libp2p/p2p/transport/tcp"
"github.com/libp2p/go-libp2p/p2p/transport/websocket"
webtransport "github.com/libp2p/go-libp2p/p2p/transport/webtransport"
"go.uber.org/fx"
)
@ -21,7 +22,8 @@ func Transports(tptConfig config.Transports) interface{} {
privateNetworkEnabled := pnet.Fprint != nil
if tptConfig.Network.TCP.WithDefault(true) {
opts.Opts = append(opts.Opts, libp2p.Transport(tcp.NewTCPTransport))
// TODO(9290): Make WithMetrics configurable
opts.Opts = append(opts.Opts, libp2p.Transport(tcp.NewTCPTransport, tcp.WithMetrics()))
}
if tptConfig.Network.Websocket.WithDefault(true) {
@ -30,14 +32,24 @@ func Transports(tptConfig config.Transports) interface{} {
if tptConfig.Network.QUIC.WithDefault(!privateNetworkEnabled) {
if privateNetworkEnabled {
// QUIC was force enabled while the private network was turned on.
// Fail and tell the user.
return opts, fmt.Errorf(
"The QUIC transport does not support private networks. " +
"Please disable Swarm.Transports.Network.QUIC.",
)
}
opts.Opts = append(opts.Opts, libp2p.Transport(libp2pquic.NewTransport))
// TODO(9290): Make WithMetrics configurable
opts.Opts = append(opts.Opts, libp2p.Transport(quic.NewTransport, quic.WithMetrics()))
}
// TODO(9292): Remove the false && to allows it enabled by default
if tptConfig.Network.WebTransport.WithDefault(false && !privateNetworkEnabled) {
if privateNetworkEnabled {
return opts, fmt.Errorf(
"The WebTransport transport does not support private networks. " +
"Please disable Swarm.Transports.Network.WebTransport.",
)
}
opts.Opts = append(opts.Opts, libp2p.Transport(webtransport.New))
}
return opts, nil

4
core/node/peering.go
View File

@ -4,8 +4,8 @@ import (
"context"
"github.com/ipfs/kubo/peering"
"github.com/libp2p/go-libp2p-core/host"
"github.com/libp2p/go-libp2p-core/peer"
"github.com/libp2p/go-libp2p/core/host"
"github.com/libp2p/go-libp2p/core/peer"
"go.uber.org/fx"
)

13
core/node/provider.go
View File

@ -28,13 +28,13 @@ func ProviderQueue(mctx helpers.MetricsCtx, lc fx.Lifecycle, repo repo.Repo) (*q
}
// SimpleProvider creates new record provider
func SimpleProvider(mctx helpers.MetricsCtx, lc fx.Lifecycle, queue *q.Queue, rt irouting.TieredRouter) provider.Provider {
func SimpleProvider(mctx helpers.MetricsCtx, lc fx.Lifecycle, queue *q.Queue, rt irouting.ProvideManyRouter) provider.Provider {
return simple.NewProvider(helpers.LifecycleCtx(mctx, lc), queue, rt)
}
// SimpleReprovider creates new reprovider
func SimpleReprovider(reproviderInterval time.Duration) interface{} {
return func(mctx helpers.MetricsCtx, lc fx.Lifecycle, rt irouting.TieredRouter, keyProvider simple.KeyChanFunc) (provider.Reprovider, error) {
return func(mctx helpers.MetricsCtx, lc fx.Lifecycle, rt irouting.ProvideManyRouter, keyProvider simple.KeyChanFunc) (provider.Reprovider, error) {
return simple.NewReprovider(helpers.LifecycleCtx(mctx, lc), reproviderInterval, rt, keyProvider), nil
}
}
@ -62,12 +62,7 @@ func SimpleProviderSys(isOnline bool) interface{} {
// BatchedProviderSys creates new provider system
func BatchedProviderSys(isOnline bool, reprovideInterval string) interface{} {
return func(lc fx.Lifecycle, cr irouting.TieredRouter, q *q.Queue, keyProvider simple.KeyChanFunc, repo repo.Repo) (provider.System, error) {
r := cr.ProvideMany()
if r == nil {
return nil, fmt.Errorf("BatchedProviderSys requires a content router that supports provideMany")
}
return func(lc fx.Lifecycle, cr irouting.ProvideManyRouter, q *q.Queue, keyProvider simple.KeyChanFunc, repo repo.Repo) (provider.System, error) {
reprovideIntervalDuration := kReprovideFrequency
if reprovideInterval != "" {
dur, err := time.ParseDuration(reprovideInterval)
@ -78,7 +73,7 @@ func BatchedProviderSys(isOnline bool, reprovideInterval string) interface{} {
reprovideIntervalDuration = dur
}
sys, err := batched.New(r, q,
sys, err := batched.New(cr, q,
batched.ReproviderInterval(reprovideIntervalDuration),
batched.Datastore(repo.Datastore()),
batched.KeyProvider(keyProvider))

30
docs/RELEASE_ISSUE_TEMPLATE.md
View File

@ -28,7 +28,7 @@ As usual, this release includes important fixes, some of which may be critical f
## 🔦 Highlights
< top highlights for this release notes >
< top highlights for this release notes. For ANY version (final or RCs) >
## ✅ Release Checklist
@ -36,18 +36,19 @@ For each RC published in each stage:
- version string in `version.go` has been updated (in the `release-vX.Y.Z` branch).
- new commits should be added to the `release-vX.Y.Z` branch from `master` using `git cherry-pick -x ...`
- Note: `release-` branches are protected. You can do all needed updates on a separated branch and when everything is settled push to `release-vX.Y.Z`
- tag commit with `vX.Y.Z-rcN`
- add artifacts to https://dist.ipfs.tech
1. Make a PR against [ipfs/distributions](https://github.com/ipfs/distributions) with local changes produced by `add-version` (see [usage](https://github.com/ipfs/distributions#usage))
2. Wait for PR to build artifacts and generate diff
2. Wait for PR to build artifacts and generate diff (~30min)
3. Inspect results, merge if CI is green and the diff looks ok
4. Wait for `master` branch to build and update DNSLink at https://dist.ipfs.tech
4. Wait for `master` branch to build and update DNSLink at https://dist.ipfs.tech (~30min)
- cut a pre-release on [github](https://github.com/ipfs/kubo/releases) and reuse signed artifacts from https://dist.ipfs.tech/kubo (upload the result of the ipfs/distributions build in the previous step).
- Announce the RC:
- [ ]
- [ ] Create a new post on [IPFS Discourse](https://discuss.ipfs.tech)
- This will automatically post to IPFS Discord #ipfs-chatter
- Examples from the past: [0.14.0](https://discuss.ipfs.io/t/kubo-formerly-go-ipfs-v0-14-0-release-is-out/14794)
- [ ] Pin the topic
- [ ] Pin the topic. You need to be part of the [admin group](https://discuss.ipfs.tech/g/admins) for that.
- [ ] To the _early testers_ listed in [docs/EARLY_TESTERS.md](https://github.com/ipfs/go-ipfs/tree/master/docs/EARLY_TESTERS.md). Do this by copy/pasting their GitHub usernames and checkboxes as a comment so they get a GitHub notification. ([example](https://github.com/ipfs/go-ipfs/issues/8176#issuecomment-909356394))
Checklist:
@ -70,22 +71,31 @@ Checklist:
- [ ] **Stage 1 - Internal Testing**
- [ ] CHANGELOG.md has been updated
- use [`./bin/mkreleaselog`](https://github.com/ipfs/go-ipfs/tree/master/bin/mkreleaselog) to generate a nice starter list
- [ ] Infrastructure Testing:
- you need to install `zsh`.
- [ ] Infrastructure Testing.
- [ ] Open an issue against https://github.com/protocol/bifrost-infra and spell out all that we want (e.g.,mgateways, bootstrapper, and cluster). [example](https://github.com/protocol/bifrost-infra/issues/2046)
- [ ] Deploy new version to a subset of Bootstrappers
- [ ] Deploy new version to a subset of Gateways
- [ ] Deploy new version to a subset of Preload nodes
- [ ] Collect metrics every day. Work with the Infrastructure team to learn of any hiccup
- [ ] Collect [metrics](https://protocollabs.grafana.net/d/8zlhkKTZk/gateway-slis-precomputed?orgId=1) every day. Work with the Infrastructure team to learn of any hiccup
- [ ] IPFS Application Testing - Run the tests of the following applications:
- [ ] [IPFS Desktop](https://github.com/ipfs-shipyard/ipfs-desktop)
- [ ] Ensure the RC is published to [the NPM package](https://www.npmjs.com/package/go-ipfs?activeTab=versions) ([happens automatically, just wait for CI](https://github.com/ipfs/npm-go-ipfs/actions))
- [ ] Upgrade to the RC in [ipfs-desktop](https://github.com/ipfs-shipyard/ipfs-desktop) and push to a branch ([example](https://github.com/ipfs/ipfs-desktop/pull/1826/commits/b0a23db31ce942b46d95965ee6fe770fb24d6bde)), and open a draft PR to track through the final release ([example](https://github.com/ipfs/ipfs-desktop/pull/1826))
- [ ] Ensure CI tests pass, repeat for new RCs
- [ ] [IPFS Companion](https://github.com/ipfs-shipyard/ipfs-companion) - @lidel
- [ ] [IPFS Companion](https://github.com/ipfs-shipyard/ipfs-companion)
- Start kubo daemon of the version to release.
- Start a fresh chromium or chrome instance using `chromium --user-data-dir=$(mktemp -d)`
- Start a fresh firefox instance using `firefox --profile $(mktemp -d)`
- Install IPFS Companion from [vendor-specific store](https://github.com/ipfs/ipfs-companion/#readme).
- Check that the comunication between Kubo daemon and IPFS companion is working properly checking if the number of connected peers changes.
- [ ] **Stage 2 - Community Prod Testing**
- [ ] Documentation
- [ ] Ensure that [CHANGELOG.md](https://github.com/ipfs/go-ipfs/tree/master/CHANGELOG.md) is up to date
- [ ] Add a link from release notes to Discuss post (like we did here: https://github.com/ipfs/kubo/releases/tag/v0.15.0 )
- [ ] Keep the release notes as trim as possible (removing some of the top headers, like we did here: https://github.com/ipfs/kubo/releases/tag/v0.15.0 )
- [ ] Ensure that [README.md](https://github.com/ipfs/go-ipfs/tree/master/README.md) is up to date
- [ ] Update docs by merging the auto-created PR in https://github.com/ipfs/ipfs-docs/pulls (they are auto-created every 12 hours)
- [ ] Update docs by merging the auto-created PR in https://github.com/ipfs/ipfs-docs/pulls (they are auto-created every 12 hours) (only for final releases, not RCs)
- [ ] Invite the wider community through (link to the release issue):
- [ ] [discuss.ipfs.io](https://discuss.ipfs.io/c/announcements)
- [ ] Matrix
@ -143,7 +153,7 @@ The best place to ask your questions about IPFS, how it works and what you can d
### Changelog
< changelog generated by bin/mkreleaselog >
< changelog generated by bin/mkreleaselog > (add it to a separated comment if it is too big)
### ❤️ Contributors

1
docs/changelogs/v0.15.md
View File

@ -27,6 +27,7 @@ We are improving release speed and cadence trying to have a new release every 5
You can now use `blake3` as a valid hash function:
- `ipfs block put --mhtype=blake3`
- `ipfs add --hash=blake3`
It uses a 32 bytes default size.
And verify up to 128 bytes.
Because `blake3` is variable output hash function, you can use a different digest length, set `mhlen`: `ipfs block put --mhtype=blake3 --mhlen=64`, `ipfs add` doesn't have this option yet.

553
docs/changelogs/v0.16.md Normal file
View File

@ -0,0 +1,553 @@
# Kubo changelog v0.16
## v0.16.0
### Overview
Below is an outline of all that is in this release, so you get a sense of all that's included.
- [Kubo changelog v0.16](#kubo-changelog-v016)
- [v0.16.0](#v0160)
- [Overview](#overview)
- [🔦 Highlights](#-highlights)
- [🛣️ More configurable delegated routing system](#️-more-configurable-delegated-routing-system)
- [🌍 WebTransport new experimental Transport](#-webtransport-new-experimental-transport)
- [🗃️ Hardened IPNS record verification](#-hardened-ipns-record-verification)
- [🌉 Web Gateways now support _redirects files](#-web-gateways-now-support-_redirects-files)
- [😻 Add files to MFS with ipfs add --to-files](#-add-files-to-mfs-with-ipfs-add---to-files)
- [Changelog](#changelog)
- [Contributors](#contributors)
### 🔦 Highlights
<!-- TODO -->
#### 🛣️ More configurable delegated routing system
Since Kubo v0.14.0 [Reframe protocol](https://github.com/ipfs/specs/tree/main/reframe#readme) has been supported as a new routing system.
Now, we allow to configure several routers working together, so you can have several `reframe` and `dht` routers making queries. You can use the special `parallel` and `sequential` routers to fill your needs.
Example configuration usage using the [Filecoin Network Indexer](https://docs.cid.contact/filecoin-network-indexer/overview) and the DHT, making first a query to the indexer, and timing out after 3 seconds.
```console
$ ipfs config Routing.Type --json '"custom"'
$ ipfs config Routing.Routers.CidContact --json '{
"Type": "reframe",
"Parameters": {
"Endpoint": "https://cid.contact/reframe"
}
}'
$ ipfs config Routing.Routers.WanDHT --json '{
"Type": "dht",
"Parameters": {
"Mode": "auto",
"PublicIPNetwork": true,
"AcceleratedDHTClient": false
}
}'
$ ipfs config Routing.Routers.ParallelHelper --json '{
"Type": "parallel",
"Parameters": {
"Routers": [
{
"RouterName" : "CidContact",
"IgnoreErrors" : true,
"Timeout": "3s"
},
{
"RouterName" : "WanDHT",
"IgnoreErrors" : false,
"Timeout": "5m",
"ExecuteAfter": "2s"
}
]
}
}'
$ ipfs config Routing.Methods --json '{
"find-peers": {
"RouterName": "ParallelHelper"
},
"find-providers": {
"RouterName": "ParallelHelper"
},
"get-ipns": {
"RouterName": "ParallelHelper"
},
"provide": {
"RouterName": "WanDHT"
},
"put-ipns": {
"RouterName": "ParallelHelper"
}
}'
```
### 🌍 WebTransport new experimental Transport
A new feature of [`go-libp2p`](https://github.com/libp2p/go-libp2p/releases/tag/v0.23.0) is [WebTransport](https://github.com/libp2p/go-libp2p/issues/1717).
For now it is **disabled by default** and considered **experimental**.
If you find issues running it please [report them to us](https://github.com/ipfs/kubo/issues/new).
In the future Kubo will listen on WebTransport by default for anyone already listening on QUIC addresses.
WebTransport is a new transport protocol currently under development by the [IETF](https://datatracker.ietf.org/wg/webtrans/about/) and the [W3C](https://www.w3.org/TR/webtransport/), and [already implemented by Chrome](https://caniuse.com/webtransport).
Conceptually, it’s like WebSocket run over QUIC instead of TCP. Most importantly, it allows browsers to establish (secure!) connections to WebTransport servers without the need for CA-signed certificates,
thereby enabling any js-libp2p node running in a browser to connect to any kubo node, with zero manual configuration involved.
The previous alternative is websocket secure, which require installing a reverse proxy and TLS certificates manually.
#### How to enable WebTransport
Thoses steps are temporary and wont be needed once we make it enabled by default.
1. Enable the WebTransport transport:
`ipfs config Swarm.Transports.Network.WebTransport --json true`
1. Add a listener address for WebTransport to your `Addresses.Swarm` key, for example:
```json
[
"/ip4/0.0.0.0/tcp/4001",
"/ip4/0.0.0.0/udp/4001/quic",
"/ip4/0.0.0.0/udp/4002/quic/webtransport"
]
```
1. Restart your daemon to apply the config changes.
### 🗃️ Hardened IPNS record verification
Records that do not have a valid IPNS V2 signature, or exceed the max size
limit, will no longer pass verification, and will be ignored by Kubo when
resolving `/ipns/{libp2p-key}` content paths.
Kubo continues publishing backward-compatible V1+V2 records that can be
resolved by V1-only (go-ipfs <0.9.0) clients.
More details can be found in _Backward Compatibility_, _Record Creation_, and
_Record Verification_ sections of the [updated IPNS
specification](https://github.com/ipfs/specs/pull/319/files).
### 🌉 Web Gateways now support `_redirects` files
This feature enables support for redirects, single-page applications (SPA),
custom 404 pages, and moving to IPFS-backed website hosting
[without breaking existing HTTP links](https://www.w3.org/Provider/Style/URI).
It is limited to websites hosted in web contexts with unique
[Origins](https://en.wikipedia.org/wiki/Same-origin_policy), such as
[subdomain](https://docs.ipfs.tech/how-to/address-ipfs-on-web/#subdomain-gateway) and
[DNSLink](https://docs.ipfs.tech/how-to/address-ipfs-on-web/#dnslink-gateway) gateways.
Redirect logic is evaluated only if the requested path is not in the DAG.
See more details and usage examples see
[docs.ipfs.tech: _Redirects, custom 404s, and SPA support_](https://docs.ipfs.tech/how-to/websites-on-ipfs/redirects-and-custom-404s/).
### 😻 Add files to MFS with `ipfs add --to-files`
Users no longer need to call `ipfs files cp` after `ipfs add` to create a
reference in [MFS](https://docs.ipfs.tech/concepts/glossary/#mfs), or deal with
low level pins if they do not wish to do so. It is now possible to pass MFS
path in an optional `--to-files` to add data directly to MFS, without creating
a low level pin.
Before (Kubo <0.16.0):
```console
$ ipfs add cat.jpg
QmCID
$ ipfs files cp /ipfs/QmCID /mfs-cats/cat.jpg
$ ipfs pin rm QmCID # removing low level pin, since MFS is protecting from gc
```
Kubo 0.16.0 collapses the above steps into one:
```console
$ ipfs add --pin=false cat.jpg --to-files /mfs-cats/
```
A recursive add to MFS works too (below line will create `/lots-of-cats/` directory in MFS):
```console
$ ipfs add -r ./lots-of-cats/ --to-files /
```
For more information, see `ipfs add --help` and `ipfs files --help`.
### Changelog
<details>
<summary>Full Changelog</summary>
- github.com/ipfs/kubo:
- fix: Set default Methods value to nil
- docs: document remaining 0.16.0 features
- docs: add WebTransport docs ([ipfs/kubo#9308](https://github.com/ipfs/kubo/pull/9308))
- chore: bump version to 0.16.0-rc1
- fix: ensure hasher is registered when using a hashing function
- feat: add webtransport as an optin transport ([ipfs/kubo#9293](https://github.com/ipfs/kubo/pull/9293))
- feat(gateway): _redirects file support (#8890) ([ipfs/kubo#8890](https://github.com/ipfs/kubo/pull/8890))
- docs: fix typo in changelog-v0.16.0.md
- Readme: Rewrite introduction and featureset (#9211) ([ipfs/kubo#9211](https://github.com/ipfs/kubo/pull/9211))
- feat: Delegated routing with custom configuration. (#9274) ([ipfs/kubo#9274](https://github.com/ipfs/kubo/pull/9274))
- Add <protocols> to `ipfs id -h` options (#9229) ([ipfs/kubo#9229](https://github.com/ipfs/kubo/pull/9229))
- chore: bump go-libp2p v0.23.1 ([ipfs/kubo#9285](https://github.com/ipfs/kubo/pull/9285))
- feat(cmds/add): --to-files option automates files cp (#8927) ([ipfs/kubo#8927](https://github.com/ipfs/kubo/pull/8927))
- docs: fix broken ENS DoH example (#9281) ([ipfs/kubo#9281](https://github.com/ipfs/kubo/pull/9281))
- ([ipfs/kubo#9258](https://github.com/ipfs/kubo/pull/9258))
- ([ipfs/kubo#9213](https://github.com/ipfs/kubo/pull/9213))
- docs: small typo in Dockerfile
- feat: ipfs-webui v2.18.1
- feat: ipfs-webui v2.18.0 (#9262) ([ipfs/kubo#9262](https://github.com/ipfs/kubo/pull/9262))
- bump go-libp2p v0.22.0 & go1.18&go1.19 ([ipfs/kubo#9244](https://github.com/ipfs/kubo/pull/9244))
- docs: change windows choco install command to point to go-ipfs
- fix: pass the repo directory into the ignored_commit function
- docs(cmds): daemon: update DHTClient description
- fix(gw): send 200 for empty files
- docs(readme): official vs unofficial packages
- chore: remove Gateway.PathPrefixes
- docs(readme): update Docker section
- docs: fix markdown syntax typo in v0.15's changelog
- chore: Release v0.15.0 ([ipfs/kubo#9236](https://github.com/ipfs/kubo/pull/9236))
- chore: fix undiallable api and gateway files ([ipfs/kubo#9233](https://github.com/ipfs/kubo/pull/9233))
- chore: Bump version to 0.16.0-dev
- github.com/ipfs/go-bitswap (v0.9.0 -> v0.10.2):
- chore: release v0.10.2
- fix: create a copy of the protocol slice in network.processSettings
- chore: release v0.10.1
- fix: incorrect type in the WithTracer polyfill option
- chore: fix incorrect log message when a bad option is passed
- chore: release v0.10.0
- chore: update go-libp2p v0.22.0
- github.com/ipfs/go-cid (v0.2.0 -> v0.3.2):
- chore: release v0.3.2
- Revert "fix: bring back, but deprecate CodecToStr and Codecs"
- chore: release v0.2.1
- fix: bring back, but deprecate CodecToStr and Codecs
- run gofmt -s
- bump go.mod to Go 1.18 and run go fix
- chore: release v0.3.0
- fix: return nil Bytes() if the Cid in undef
- Add MustParse ([ipfs/go-cid#139](https://github.com/ipfs/go-cid/pull/139))
- github.com/ipfs/go-datastore (v0.5.1 -> v0.6.0):
- Release v0.6.0 (#194) ([ipfs/go-datastore#194](https://github.com/ipfs/go-datastore/pull/194))
- feat: add Features + datastore scoping
- chore: Fix comment typo (#191) ([ipfs/go-datastore#191](https://github.com/ipfs/go-datastore/pull/191))
- github.com/ipfs/go-delegated-routing (v0.3.0 -> v0.6.0):
- ([ipfs/go-delegated-routing#53](https://github.com/ipfs/go-delegated-routing/pull/53))
- ([ipfs/go-delegated-routing#52](https://github.com/ipfs/go-delegated-routing/pull/52))
- Release v0.5.2 (#50) ([ipfs/go-delegated-routing#50](https://github.com/ipfs/go-delegated-routing/pull/50))
- Fixed serialisation issue with multiadds (#49) ([ipfs/go-delegated-routing#49](https://github.com/ipfs/go-delegated-routing/pull/49))
- Upgrade to IPLD `0.18.0`
- Release v0.5.0 (#47) ([ipfs/go-delegated-routing#47](https://github.com/ipfs/go-delegated-routing/pull/47))
- feat: use GET for FindProviders (#46) ([ipfs/go-delegated-routing#46](https://github.com/ipfs/go-delegated-routing/pull/46))
- Update provide to take an array of keys, per spec (#45) ([ipfs/go-delegated-routing#45](https://github.com/ipfs/go-delegated-routing/pull/45))
- ([ipfs/go-delegated-routing#44](https://github.com/ipfs/go-delegated-routing/pull/44))
- fix: upgrade edelweiss and rerun 'go generate' (#42) ([ipfs/go-delegated-routing#42](https://github.com/ipfs/go-delegated-routing/pull/42))
- ci: add check to ensure generated files are up-to-date (#41) ([ipfs/go-delegated-routing#41](https://github.com/ipfs/go-delegated-routing/pull/41))
- Add Provide RPC (#37) ([ipfs/go-delegated-routing#37](https://github.com/ipfs/go-delegated-routing/pull/37))
- upgrade to go-log/v2 (#34) ([ipfs/go-delegated-routing#34](https://github.com/ipfs/go-delegated-routing/pull/34))
- github.com/ipfs/go-ipns (v0.1.2 -> v0.3.0):
- fix: require V2 signatures ([ipfs/go-ipns#41](https://github.com/ipfs/go-ipns/pull/41))
- update go-libp2p to v0.22.0, release v0.2.0 (#39) ([ipfs/go-ipns#39](https://github.com/ipfs/go-ipns/pull/39))
- use peer.IDFromBytes instead of peer.IDFromString (#38) ([ipfs/go-ipns#38](https://github.com/ipfs/go-ipns/pull/38))
- sync: update CI config files (#34) ([ipfs/go-ipns#34](https://github.com/ipfs/go-ipns/pull/34))
- github.com/ipfs/go-pinning-service-http-client (v0.1.1 -> v0.1.2):
- chore: release v0.1.2
- fix: send up to nanosecond precision
- refactor: cleanup Sprintf for Bearer token
- sync: update CI config files ([ipfs/go-pinning-service-http-client#21](https://github.com/ipfs/go-pinning-service-http-client/pull/21))
- github.com/ipld/edelweiss (v0.1.4 -> v0.2.0):
- Release v0.2.0 (#60) ([ipld/edelweiss#60](https://github.com/ipld/edelweiss/pull/60))
- feat: add cachable modifier to methods (#48) ([ipld/edelweiss#48](https://github.com/ipld/edelweiss/pull/48))
- adding licenses (#52) ([ipld/edelweiss#52](https://github.com/ipld/edelweiss/pull/52))
- sync: update CI config files ([ipld/edelweiss#56](https://github.com/ipld/edelweiss/pull/56))
- chore: replace deprecated ioutil with io/os ([ipld/edelweiss#59](https://github.com/ipld/edelweiss/pull/59))
- Release v0.1.6
- fix: iterate over BlueMap in deterministic order (#57) ([ipld/edelweiss#57](https://github.com/ipld/edelweiss/pull/57))
- fix: wrap DAG-JSON serialization error (#55) ([ipld/edelweiss#55](https://github.com/ipld/edelweiss/pull/55))
- update examples and harness
- upgrade to go-log/v2 (#53) ([ipld/edelweiss#53](https://github.com/ipld/edelweiss/pull/53))
- github.com/ipld/go-ipld-prime (v0.17.0 -> v0.18.0):
- Prepare v0.18.0
- feat(bindnode): add a BindnodeRegistry utility (#437) ([ipld/go-ipld-prime#437](https://github.com/ipld/go-ipld-prime/pull/437))
- feat(bindnode): support full uint64 range
- chore(bindnode): remove typed functions for options
- chore(bindnode): docs and minor tweaks
- feat(bindnode): make Any converters work for List and Map values
- fix(bindnode): shorten converter option names, minor perf improvements
- fix(bindnode): only custom convert AssignNull for Any converter
- feat(bindnode): pass Null on to nullable custom converters
- chore(bindnode): config helper refactor w/ short-circuit
- feat(bindnode): add AddCustomTypeAnyConverter() to handle `Any` fields
- feat(bindnode): add AddCustomTypeXConverter() options for most scalar kinds
- chore(bindnode): back out of reflection for converters
- feat(bindnode): switch to converter functions instead of type
- feat(bindnode): allow custom type conversions with options
- feat: add release checklist (#442) ([ipld/go-ipld-prime#442](https://github.com/ipld/go-ipld-prime/pull/442))
- github.com/libp2p/go-flow-metrics (v0.0.3 -> v0.1.0):
- introduce an API to set a mock clock (#20) ([libp2p/go-flow-metrics#20](https://github.com/libp2p/go-flow-metrics/pull/20))
- chore: skip slow tests when short testing is specified ([libp2p/go-flow-metrics#16](https://github.com/libp2p/go-flow-metrics/pull/16))
- github.com/libp2p/go-libp2p (v0.21.0 -> v0.23.2):
- release v0.23.2 (#1781) ([libp2p/go-libp2p#1781](https://github.com/libp2p/go-libp2p/pull/1781))
- webtransport: return error before wrapping opened / accepted streams (#1775) ([libp2p/go-libp2p#1775](https://github.com/libp2p/go-libp2p/pull/1775))
- release v0.23.1 (#1773) ([libp2p/go-libp2p#1773](https://github.com/libp2p/go-libp2p/pull/1773))
- websocket: fix nil pointer in tlsClientConf (#1770) ([libp2p/go-libp2p#1770](https://github.com/libp2p/go-libp2p/pull/1770))
- release v0.23.0 (#1764) ([libp2p/go-libp2p#1764](https://github.com/libp2p/go-libp2p/pull/1764))
- noise: switch to proto2, use the new NoiseExtensions protobuf ([libp2p/go-libp2p#1762](https://github.com/libp2p/go-libp2p/pull/1762))
- webtransport: add custom resolver to add SNI (#1761) ([libp2p/go-libp2p#1761](https://github.com/libp2p/go-libp2p/pull/1761))
- swarm: skip dialing WebTransport addresses when we have QUIC addresses (#1756) ([libp2p/go-libp2p#1756](https://github.com/libp2p/go-libp2p/pull/1756))
- webtransport: have the server send the certificates (#1757) ([libp2p/go-libp2p#1757](https://github.com/libp2p/go-libp2p/pull/1757))
- noise: make it possible for the server to send early data (#1750) ([libp2p/go-libp2p#1750](https://github.com/libp2p/go-libp2p/pull/1750))
- swarm: fix selection of transport for dialing (#1653) ([libp2p/go-libp2p#1653](https://github.com/libp2p/go-libp2p/pull/1653))
- autorelay: Add a context.Context to WithPeerSource callback (#1736) ([libp2p/go-libp2p#1736](https://github.com/libp2p/go-libp2p/pull/1736))
- webtransport: add and check the ?type=noise URL parameter (#1749) ([libp2p/go-libp2p#1749](https://github.com/libp2p/go-libp2p/pull/1749))
- webtransport: disable HTTP origin check (#1752) ([libp2p/go-libp2p#1752](https://github.com/libp2p/go-libp2p/pull/1752))
- noise: don't fail handshake when early data is received without handler (#1746) ([libp2p/go-libp2p#1746](https://github.com/libp2p/go-libp2p/pull/1746))
- Add Resolver interface to transport (#1719) ([libp2p/go-libp2p#1719](https://github.com/libp2p/go-libp2p/pull/1719))
- use new /libp2p/go-libp2p/core pkg (#1745) ([libp2p/go-libp2p#1745](https://github.com/libp2p/go-libp2p/pull/1745))
- yamux: pass constructors for peer resource scopes to session constructor (#1739) ([libp2p/go-libp2p#1739](https://github.com/libp2p/go-libp2p/pull/1739))
- tcp: add an option to enable metrics (disabled by default) (#1734) ([libp2p/go-libp2p#1734](https://github.com/libp2p/go-libp2p/pull/1734))
- move go-libp2p-webtransport to p2p/transport/webtransport ([libp2p/go-libp2p#1737](https://github.com/libp2p/go-libp2p/pull/1737))
- autorelay: fix race condition in TestBackoff (#1731) ([libp2p/go-libp2p#1731](https://github.com/libp2p/go-libp2p/pull/1731))
- rcmgr: increase default connection memory limit to 32 MB (#1740) ([libp2p/go-libp2p#1740](https://github.com/libp2p/go-libp2p/pull/1740))
- quic: update quic-go to v0.29.0 (#1723) ([libp2p/go-libp2p#1723](https://github.com/libp2p/go-libp2p/pull/1723))
- noise: implement an API to send and receive early data ([libp2p/go-libp2p#1728](https://github.com/libp2p/go-libp2p/pull/1728))
- identify: make the protocol version configurable (#1724) ([libp2p/go-libp2p#1724](https://github.com/libp2p/go-libp2p/pull/1724))
- Fix threshold calculation (#1722) ([libp2p/go-libp2p#1722](https://github.com/libp2p/go-libp2p/pull/1722))
- connmgr: use clock interface (#1720) ([libp2p/go-libp2p#1720](https://github.com/libp2p/go-libp2p/pull/1720))
- quic: increase the buffer size used for encoding qlogs (#1715) ([libp2p/go-libp2p#1715](https://github.com/libp2p/go-libp2p/pull/1715))
- quic: add a WithMetrics option (#1716) ([libp2p/go-libp2p#1716](https://github.com/libp2p/go-libp2p/pull/1716))
- add default listen addresses for QUIC (#1615) ([libp2p/go-libp2p#1615](https://github.com/libp2p/go-libp2p/pull/1615))
- feat: inject DNS resolver (#1607) ([libp2p/go-libp2p#1607](https://github.com/libp2p/go-libp2p/pull/1607))
- connmgr: prefer peers with no streams when closing connections (#1675) ([libp2p/go-libp2p#1675](https://github.com/libp2p/go-libp2p/pull/1675))
- quic: add DisableReuseport option (#1476) ([libp2p/go-libp2p#1476](https://github.com/libp2p/go-libp2p/pull/1476))
- release v0.22.0 ([libp2p/go-libp2p#1688](https://github.com/libp2p/go-libp2p/pull/1688))
- fix: don't prefer local ports from other addresses when dialing (#1673) ([libp2p/go-libp2p#1673](https://github.com/libp2p/go-libp2p/pull/1673))
- crypto: add better support for alternative backends (#1686) ([libp2p/go-libp2p#1686](https://github.com/libp2p/go-libp2p/pull/1686))
- crypto/secp256k1: Remove btcsuite intermediary. (#1689) ([libp2p/go-libp2p#1689](https://github.com/libp2p/go-libp2p/pull/1689))
- Update resource manager README (#1684) ([libp2p/go-libp2p#1684](https://github.com/libp2p/go-libp2p/pull/1684))
- move go-libp2p-core here ([libp2p/go-libp2p#1683](https://github.com/libp2p/go-libp2p/pull/1683))
- rcmgr: make scaling changes more intuitive (#1685) ([libp2p/go-libp2p#1685](https://github.com/libp2p/go-libp2p/pull/1685))
- move go-eventbus here ([libp2p/go-libp2p#1681](https://github.com/libp2p/go-libp2p/pull/1681))
- basichost: remove usage of MultistreamServerMatcher in test (#1680) ([libp2p/go-libp2p#1680](https://github.com/libp2p/go-libp2p/pull/1680))
- sync: update CI config files (#1678) ([libp2p/go-libp2p#1678](https://github.com/libp2p/go-libp2p/pull/1678))
- move go-libp2p-resource-manager to p2p/host/resource-manager ([libp2p/go-libp2p#1677](https://github.com/libp2p/go-libp2p/pull/1677))
- chore: preallocate slices with known final size (#1679) ([libp2p/go-libp2p#1679](https://github.com/libp2p/go-libp2p/pull/1679))
- autorelay: fix flaky TestMaxAge (#1676) ([libp2p/go-libp2p#1676](https://github.com/libp2p/go-libp2p/pull/1676))
- move go-libp2p-peerstore to p2p/host/peerstore ([libp2p/go-libp2p#1667](https://github.com/libp2p/go-libp2p/pull/1667))
- examples: remove ipfs components from echo (#1672) ([libp2p/go-libp2p#1672](https://github.com/libp2p/go-libp2p/pull/1672))
- chore: update libp2p to v0.21 in examples (#1674) ([libp2p/go-libp2p#1674](https://github.com/libp2p/go-libp2p/pull/1674))
- change the default key type to Ed25519 (#1576) ([libp2p/go-libp2p#1576](https://github.com/libp2p/go-libp2p/pull/1576))
- autorelay: poll for new candidates when needed ([libp2p/go-libp2p#1587](https://github.com/libp2p/go-libp2p/pull/1587))
- examples: fix unresponsive pubsub chat example (#1652) ([libp2p/go-libp2p#1652](https://github.com/libp2p/go-libp2p/pull/1652))
- routed: respect force direct dial context (#1665) ([libp2p/go-libp2p#1665](https://github.com/libp2p/go-libp2p/pull/1665))
- pstoremanager: fix flaky TestClose (#1649) ([libp2p/go-libp2p#1649](https://github.com/libp2p/go-libp2p/pull/1649))
- Allow adding prologue to noise connections (#1663) ([libp2p/go-libp2p#1663](https://github.com/libp2p/go-libp2p/pull/1663))
- connmgr: add nowatchdog go build tag (#1666) ([libp2p/go-libp2p#1666](https://github.com/libp2p/go-libp2p/pull/1666))
- mdns: don't discover ourselves (#1661) ([libp2p/go-libp2p#1661](https://github.com/libp2p/go-libp2p/pull/1661))
- Support generating custom x509 certificates (#1481) ([libp2p/go-libp2p#1481](https://github.com/libp2p/go-libp2p/pull/1481))
- github.com/libp2p/go-libp2p-core (v0.19.1 -> v0.20.1):
- chore: release v0.20.1
- feat: forward crypto/pb
- release v0.20.0
- deprecate this repo
- stop using the deprecated io/ioutil package (#279) ([libp2p/go-libp2p-core#279](https://github.com/libp2p/go-libp2p-core/pull/279))
- use a mock clock in bandwidth tests (#276) ([libp2p/go-libp2p-core#276](https://github.com/libp2p/go-libp2p-core/pull/276))
- remove unused MultistreamSemverMatcher (#277) ([libp2p/go-libp2p-core#277](https://github.com/libp2p/go-libp2p-core/pull/277))
- remove peer.IDFromString (#274) ([libp2p/go-libp2p-core#274](https://github.com/libp2p/go-libp2p-core/pull/274))
- deprecate peer.Encode in favor of peer.ID.String (#275) ([libp2p/go-libp2p-core#275](https://github.com/libp2p/go-libp2p-core/pull/275))
- deprecate peer.ID.Pretty (#273) ([libp2p/go-libp2p-core#273](https://github.com/libp2p/go-libp2p-core/pull/273))
- github.com/libp2p/go-libp2p-kad-dht (v0.17.0 -> v0.18.0):
- update go-libp2p to v0.22.0, release v0.18.0 ([libp2p/go-libp2p-kad-dht#788](https://github.com/libp2p/go-libp2p-kad-dht/pull/788))
- sync: update CI config files (#789) ([libp2p/go-libp2p-kad-dht#789](https://github.com/libp2p/go-libp2p-kad-dht/pull/789))
- github.com/libp2p/go-libp2p-peerstore (v0.7.1 -> v0.8.0):
- release v0.8.0
- deprecate this repo
- fix flaky TestGCDelay (#206) ([libp2p/go-libp2p-peerstore#206](https://github.com/libp2p/go-libp2p-peerstore/pull/206))
- fix flaky EWMA test (#205) ([libp2p/go-libp2p-peerstore#205](https://github.com/libp2p/go-libp2p-peerstore/pull/205))
- github.com/libp2p/go-libp2p-record (v0.1.3 -> v0.2.0):
- update go-libp2p to v0.22.0, release v0.2.0 ([libp2p/go-libp2p-record#50](https://github.com/libp2p/go-libp2p-record/pull/50))
- sync: update CI config files (#47) ([libp2p/go-libp2p-record#47](https://github.com/libp2p/go-libp2p-record/pull/47))
- increase RSA key sizes in tests ([libp2p/go-libp2p-record#44](https://github.com/libp2p/go-libp2p-record/pull/44))
- cleanup: fix staticcheck failures ([libp2p/go-libp2p-record#43](https://github.com/libp2p/go-libp2p-record/pull/43))
- github.com/libp2p/go-libp2p-routing-helpers (v0.2.3 -> v0.4.0):
- ([libp2p/go-libp2p-routing-helpers#62](https://github.com/libp2p/go-libp2p-routing-helpers/pull/62))
- ([libp2p/go-libp2p-routing-helpers#58](https://github.com/libp2p/go-libp2p-routing-helpers/pull/58))
- Update version.json ([libp2p/go-libp2p-routing-helpers#60](https://github.com/libp2p/go-libp2p-routing-helpers/pull/60))
- update go-libp2p to v0.22.0 ([libp2p/go-libp2p-routing-helpers#59](https://github.com/libp2p/go-libp2p-routing-helpers/pull/59))
- sync: update CI config files (#53) ([libp2p/go-libp2p-routing-helpers#53](https://github.com/libp2p/go-libp2p-routing-helpers/pull/53))
- fix staticcheck ([libp2p/go-libp2p-routing-helpers#49](https://github.com/libp2p/go-libp2p-routing-helpers/pull/49))
- fix error handling in Parallel.search ([libp2p/go-libp2p-routing-helpers#48](https://github.com/libp2p/go-libp2p-routing-helpers/pull/48))
- github.com/libp2p/go-libp2p-testing (v0.11.0 -> v0.12.0):
- release v0.12.0 (#67) ([libp2p/go-libp2p-testing#67](https://github.com/libp2p/go-libp2p-testing/pull/67))
- chore: update to go-libp2p v0.22.0 (#66) ([libp2p/go-libp2p-testing#66](https://github.com/libp2p/go-libp2p-testing/pull/66))
- remove the resource manager mocks (#65) ([libp2p/go-libp2p-testing#65](https://github.com/libp2p/go-libp2p-testing/pull/65))
- github.com/libp2p/go-openssl (v0.0.7 -> v0.1.0):
- release v0.1.0 (#31) ([libp2p/go-openssl#31](https://github.com/libp2p/go-openssl/pull/31))
- Fix build with OpenSSL 3.0 (#25) ([libp2p/go-openssl#25](https://github.com/libp2p/go-openssl/pull/25))
- sync: update CI config files ([libp2p/go-openssl#24](https://github.com/libp2p/go-openssl/pull/24))
- Add openssl.DialTimeout(network, addr, timeout, ctx, flags) call ([libp2p/go-openssl#26](https://github.com/libp2p/go-openssl/pull/26))
- Add Ctx.SetMinProtoVersion and Ctx.SetMaxProtoVersion wrappers ([libp2p/go-openssl#27](https://github.com/libp2p/go-openssl/pull/27))
- sync: update CI config files ([libp2p/go-openssl#17](https://github.com/libp2p/go-openssl/pull/17))
- fix: unsafe pointer passing ([libp2p/go-openssl#18](https://github.com/libp2p/go-openssl/pull/18))
- Update test RSA cert ([libp2p/go-openssl#15](https://github.com/libp2p/go-openssl/pull/15))
- Fix tests ([libp2p/go-openssl#16](https://github.com/libp2p/go-openssl/pull/16))
- Address `staticcheck` issues ([libp2p/go-openssl#14](https://github.com/libp2p/go-openssl/pull/14))
- Enabled PEM files with CRLF line endings to be used (#10) ([libp2p/go-openssl#11](https://github.com/libp2p/go-openssl/pull/11))
- github.com/libp2p/zeroconf/v2 (v2.1.1 -> v2.2.0):
- Fix windows libp2p (#29) ([libp2p/zeroconf#29](https://github.com/libp2p/zeroconf/pull/29))
- Fix compatibility with some IoT devices using avahi 0.8-rc1 (#27) ([libp2p/zeroconf#27](https://github.com/libp2p/zeroconf/pull/27))
- Add TTL server option (#23) ([libp2p/zeroconf#23](https://github.com/libp2p/zeroconf/pull/23))
- github.com/lucas-clemente/quic-go (v0.28.0 -> v0.29.1):
- http3: fix double close of chan when using DontCloseRequestStream
- add a logging.NullTracer and logging.NullConnectionTracer ([lucas-clemente/quic-go#3512](https://github.com/lucas-clemente/quic-go/pull/3512))
- add support for providing a custom Connection ID generator via Config (#3452) ([lucas-clemente/quic-go#3452](https://github.com/lucas-clemente/quic-go/pull/3452))
- fix typo in README
- fix datagram support detection (#3511) ([lucas-clemente/quic-go#3511](https://github.com/lucas-clemente/quic-go/pull/3511))
- use a single Go routine to send copies of CONNECTION_CLOSE packets ([lucas-clemente/quic-go#3514](https://github.com/lucas-clemente/quic-go/pull/3514))
- add YoMo to list of projects in README (#3513) ([lucas-clemente/quic-go#3513](https://github.com/lucas-clemente/quic-go/pull/3513))
- http3: fix listening on both QUIC and TCP (#3465) ([lucas-clemente/quic-go#3465](https://github.com/lucas-clemente/quic-go/pull/3465))
- Disable anti-amplification limit by address validation token (#3326) ([lucas-clemente/quic-go#3326](https://github.com/lucas-clemente/quic-go/pull/3326))
- fix typo in README
- implement a new API to let servers control client address verification ([lucas-clemente/quic-go#3501](https://github.com/lucas-clemente/quic-go/pull/3501))
- use a generic streams map for incoming streams ([lucas-clemente/quic-go#3489](https://github.com/lucas-clemente/quic-go/pull/3489))
- fix unreachable code after log.Fatal in fuzzing corpus generator (#3496) ([lucas-clemente/quic-go#3496](https://github.com/lucas-clemente/quic-go/pull/3496))
- use generic Min and Max functions ([lucas-clemente/quic-go#3483](https://github.com/lucas-clemente/quic-go/pull/3483))
- add QPACK (RFC 9204) to the list of supported RFCs (#3485) ([lucas-clemente/quic-go#3485](https://github.com/lucas-clemente/quic-go/pull/3485))
- add a function to distinguish between long and short header packets (#3498) ([lucas-clemente/quic-go#3498](https://github.com/lucas-clemente/quic-go/pull/3498))
- use a generic streams map for outgoing streams (#3488) ([lucas-clemente/quic-go#3488](https://github.com/lucas-clemente/quic-go/pull/3488))
- update golangci-lint action to v3, golangci-lint to v1.48.0 (#3499) ([lucas-clemente/quic-go#3499](https://github.com/lucas-clemente/quic-go/pull/3499))
- use a generic linked list (#3487) ([lucas-clemente/quic-go#3487](https://github.com/lucas-clemente/quic-go/pull/3487))
- drop support for Go 1.16 and 1.17 (#3482) ([lucas-clemente/quic-go#3482](https://github.com/lucas-clemente/quic-go/pull/3482))
- optimize FirstOutstanding in the sent packet history (#3467) ([lucas-clemente/quic-go#3467](https://github.com/lucas-clemente/quic-go/pull/3467))
- update supported RFCs in README (#3456) ([lucas-clemente/quic-go#3456](https://github.com/lucas-clemente/quic-go/pull/3456))
- http3: ignore context after response when using DontCloseRequestStream (#3473) ([lucas-clemente/quic-go#3473](https://github.com/lucas-clemente/quic-go/pull/3473))
- github.com/marten-seemann/webtransport-go (null -> v0.1.1):
- release v0.1.1 (#31) ([marten-seemann/webtransport-go#31](https://github.com/marten-seemann/webtransport-go/pull/31))
- fix double close of chan when using DontCloseRequestStream
- github.com/multiformats/go-base32 (v0.0.4 -> v0.1.0):
- chore: bump version to 0.1.0
- fix: fix staticcheck complaints
- run gofmt -s
- sync: update CI config files (#5) ([multiformats/go-base32#5](https://github.com/multiformats/go-base32/pull/5))
- github.com/multiformats/go-multiaddr (v0.6.0 -> v0.7.0):
- Release v0.7.0 ([multiformats/go-multiaddr#183](https://github.com/multiformats/go-multiaddr/pull/183))
- use decimal numbers for multicodecs ([multiformats/go-multiaddr#184](https://github.com/multiformats/go-multiaddr/pull/184))
- Fix comment on Decapsulate ([multiformats/go-multiaddr#181](https://github.com/multiformats/go-multiaddr/pull/181))
- ([multiformats/go-multiaddr#182](https://github.com/multiformats/go-multiaddr/pull/182))
- sync: update CI config files (#180) ([multiformats/go-multiaddr#180](https://github.com/multiformats/go-multiaddr/pull/180))
- Add webrtc (#179) ([multiformats/go-multiaddr#179](https://github.com/multiformats/go-multiaddr/pull/179))
- github.com/multiformats/go-multicodec (v0.5.0 -> v0.6.0):
- chore: version bump 0.6.0
- fix: replace io/ioutil with io
- bump go.mod to Go 1.18 and run go fix
</details>
### Contributors
| Contributor | Commits | Lines ± | Files Changed |
|-------------|---------|---------|---------------|
| Marten Seemann | 236 | +12637/-24326 | 1152 |
| Raúl Kripalani | 118 | +11626/-4136 | 422 |
| vyzo | 144 | +10129/-3665 | 230 |
| galargh | 9 | +5293/-5298 | 26 |
| Marco Munizaga | 83 | +7502/-3080 | 147 |
| Antonio Navarro Perez | 33 | +4074/-1240 | 78 |
| Steven Allen | 98 | +1974/-1693 | 202 |
| Cole Brown | 57 | +2169/-1338 | 95 |
| Rod Vagg | 21 | +2588/-768 | 56 |
| Gus Eggert | 16 | +2011/-1226 | 36 |
| Yusef Napora | 6 | +2738/-187 | 43 |
| Raúl Kripalani | 2 | +1000/-889 | 18 |
| Łukasz Magiera | 26 | +1312/-500 | 54 |
| Will | 2 | +1593/-200 | 18 |
| Jorropo | 31 | +924/-712 | 204 |
| Juan Batiz-Benet | 2 | +1531/-9 | 21 |
| Jeromy | 14 | +691/-468 | 51 |
| Petar Maymounkov | 4 | +469/-285 | 25 |
| Jeromy Johnson | 24 | +474/-204 | 116 |
| Justin Johnson | 1 | +582/-93 | 7 |
| Aarsh Shah | 24 | +377/-105 | 34 |
| web3-bot | 18 | +246/-228 | 93 |
| Masih H. Derkani | 2 | +197/-213 | 21 |
| Marcin Rataj | 9 | +211/-176 | 16 |
| adam | 4 | +235/-49 | 9 |
| Jakub Sztandera | 9 | +203/-73 | 13 |
| Guilhem Fanton | 1 | +216/-48 | 5 |
| Lucas Molas | 1 | +219/-9 | 3 |
| Peter Argue | 1 | +166/-36 | 3 |
| Vibhav Pant | 4 | +186/-12 | 7 |
| Adrian Lanzafame | 3 | +180/-16 | 5 |
| Lars Gierth | 5 | +151/-41 | 25 |
| João Oliveirinha | 1 | +124/-38 | 11 |
| dignifiedquire | 3 | +122/-33 | 6 |
| Chinmay Kousik | 2 | +128/-4 | 7 |
| Toby | 1 | +89/-36 | 4 |
| Oleg Jukovec | 3 | +111/-14 | 8 |
| Whyrusleeping | 2 | +120/-0 | 6 |
| KevinZønda | 1 | +81/-20 | 2 |
| wzp | 2 | +86/-3 | 2 |
| Benedikt Spies | 1 | +75/-12 | 8 |
| nisainan | 1 | +33/-43 | 12 |
| Tshaka Eric Lekholoane | 1 | +57/-19 | 6 |
| cpuchip | 1 | +65/-6 | 2 |
| Roman Proskuryakov | 2 | +69/-0 | 2 |
| Arceliar | 2 | +36/-28 | 2 |
| Maxim Merzhanov | 1 | +29/-24 | 1 |
| Richard Ramos | 1 | +51/-0 | 2 |
| Dave Collins | 1 | +25/-25 | 4 |
| Leo Balduf | 2 | +37/-10 | 3 |
| David Aronchick | 1 | +42/-0 | 3 |
| Didrik Nordström | 1 | +35/-6 | 1 |
| Vasco Santos | 1 | +20/-20 | 7 |
| Jesse Bouwman | 1 | +19/-21 | 1 |
| Ivan Schasny | 2 | +22/-14 | 4 |
| MGMCN | 1 | +9/-24 | 2 |
| Brian Meek | 1 | +14/-17 | 4 |
| Ian Davis | 3 | +21/-9 | 5 |
| Mars Zuo | 1 | +7/-18 | 1 |
| RubenKelevra | 1 | +10/-10 | 1 |
| mojatter | 1 | +9/-8 | 1 |
| Cory Schwartz | 1 | +0/-17 | 1 |
| Steve Loeppky | 6 | +7/-6 | 6 |
| Matt Joiner | 2 | +10/-3 | 2 |
| Winterhuman | 2 | +7/-5 | 2 |
| Dmitry Yu Okunev | 1 | +5/-7 | 5 |
| corverroos | 1 | +7/-4 | 2 |
| Marcel Gregoriadis | 1 | +9/-0 | 1 |
| Ignacio Hagopian | 2 | +7/-2 | 2 |
| Julien Muret | 1 | +4/-4 | 2 |
| Eclésio Junior | 1 | +8/-0 | 1 |
| Stephan Eberle | 1 | +4/-3 | 1 |
| muXxer | 1 | +3/-3 | 1 |
| eth-limo | 1 | +3/-3 | 2 |
| Russell Dempsey | 2 | +4/-2 | 2 |
| Sergey | 1 | +1/-3 | 1 |
| Jun10ng | 2 | +2/-2 | 2 |
| Jorik Schellekens | 1 | +2/-2 | 1 |
| Eli Wang | 1 | +2/-2 | 1 |
| Andreas Linde | 1 | +4/-0 | 1 |
| whyrusleeping | 1 | +2/-1 | 1 |
| xiabin | 1 | +1/-1 | 1 |
| star | 1 | +0/-2 | 1 |
| fanweixiao | 1 | +1/-1 | 1 |
| dbadoy4874 | 1 | +1/-1 | 1 |
| bigs | 1 | +1/-1 | 1 |
| Tarun Bansal | 1 | +1/-1 | 1 |
| Mikerah | 1 | +1/-1 | 1 |
| Mike Goelzer | 1 | +2/-0 | 1 |
| Max Inden | 1 | +1/-1 | 1 |
| Kevin Mai-Husan Chia | 1 | +1/-1 | 1 |
| John B Nelson | 1 | +1/-1 | 1 |
| Eli Bailey | 1 | +1/-1 | 1 |
| Bryan Stenson | 1 | +1/-1 | 1 |
| Alex Stokes | 1 | +1/-1 | 1 |
| Abirdcfly | 1 | +1/-1 | 1 |

240
docs/config.md
View File

@ -105,8 +105,8 @@ config file at runtime.
- [`Routing`](#routing)
- [`Routing.Routers`](#routingrouters)
- [`Routing.Routers: Type`](#routingrouters-type)
- [`Routing.Routers: Enabled`](#routingrouters-enabled)
- [`Routing.Routers: Parameters`](#routingrouters-parameters)
- [`Routing: Methods`](#routing-methods)
- [`Routing.Type`](#routingtype)
- [`Swarm`](#swarm)
- [`Swarm.AddrFilters`](#swarmaddrfilters)
@ -148,6 +148,8 @@ config file at runtime.
- [`Swarm.Transports.Network.Websocket`](#swarmtransportsnetworkwebsocket)
- [`Swarm.Transports.Network.QUIC`](#swarmtransportsnetworkquic)
- [`Swarm.Transports.Network.Relay`](#swarmtransportsnetworkrelay)
- [`Swarm.Transports.Network.WebTransport`](#swarmtransportsnetworkwebtransport)
- [`How to enable WebTransport`](#how-to-enable-webtransport)
- [`Swarm.Transports.Security`](#swarmtransportssecurity)
- [`Swarm.Transports.Security.TLS`](#swarmtransportssecuritytls)
- [`Swarm.Transports.Security.SECIO`](#swarmtransportssecuritysecio)
@ -205,7 +207,7 @@ documented in `ipfs config profile --help`.
Configures the node to use the flatfs datastore. Flatfs is the default datastore.
This is the most battle-tested and reliable datastore.
This is the most battle-tested and reliable datastore.
You should use this datastore if:
- You need a very simple and very reliable datastore, and you trust your
@ -223,9 +225,9 @@ documented in `ipfs config profile --help`.
Configures the node to use the experimental badger datastore. Keep in mind that this **uses an outdated badger 1.x**.
Use this datastore if some aspects of performance,
Use this datastore if some aspects of performance,
especially the speed of adding many gigabytes of files, are critical. However, be aware that:
- This datastore will not properly reclaim space when your datastore is
smaller than several gigabytes. If you run IPFS with `--enable-gc`, you plan on storing very little data in
your IPFS node, and disk usage is more critical than performance, consider using
@ -358,6 +360,7 @@ Supported Transports:
* tcp/ip{4,6} - `/ipN/.../tcp/...`
* websocket - `/ipN/.../tcp/.../ws`
* quic - `/ipN/.../udp/.../quic`
* webtransport (*experiemental*) - `/ipN/.../udp/.../quic/webtransport` - require using a different port than the QUIC listener for now
Default:
```json
@ -681,40 +684,7 @@ Type: `bool`
### `Gateway.PathPrefixes`
**DEPRECATED:** see [kubo#7702](https://github.com/ipfs/kubo/issues/7702)
<!--
An array of acceptable url paths that a client can specify in X-Ipfs-Path-Prefix
header.
The X-Ipfs-Path-Prefix header is used to specify a base path to prepend to links
in directory listings and for trailing-slash redirects. It is intended to be set
by a frontend http proxy like nginx.
Example: We mount `blog.ipfs.io` (a dnslink page) at `ipfs.io/blog`.
**.ipfs/config**
```json
"Gateway": {
"PathPrefixes": ["/blog"],
}
```
**nginx_ipfs.conf**
```nginx
location /blog/ {
rewrite "^/blog(/.*)$" $1 break;
proxy_set_header Host blog.ipfs.io;
proxy_set_header X-Ipfs-Gateway-Prefix /blog;
proxy_pass http://127.0.0.1:8080;
}
```
-->
Default: `[]`
Type: `array[string]`
**REMOVED:** see [go-ipfs#7702](https://github.com/ipfs/go-ipfs/issues/7702)
### `Gateway.PublicGateways`
@ -837,17 +807,17 @@ Below is a list of the most common public gateway setups.
}'
```
- **Backward-compatible:** this feature enables automatic redirects from content paths to subdomains:
`http://dweb.link/ipfs/{cid}` → `http://{cid}.ipfs.dweb.link`
- **X-Forwarded-Proto:** if you run Kubo behind a reverse proxy that provides TLS, make it add a `X-Forwarded-Proto: https` HTTP header to ensure users are redirected to `https://`, not `http://`. It will also ensure DNSLink names are inlined to fit in a single DNS label, so they work fine with a wildcart TLS cert ([details](https://github.com/ipfs/in-web-browsers/issues/169)). The NGINX directive is `proxy_set_header X-Forwarded-Proto "https";`.:
`http://dweb.link/ipfs/{cid}` → `https://{cid}.ipfs.dweb.link`
`http://dweb.link/ipns/your-dnslink.site.example.com` → `https://your--dnslink-site-example-com.ipfs.dweb.link`
- **X-Forwarded-Host:** we also support `X-Forwarded-Host: example.com` if you want to override subdomain gateway host from the original request:
`http://dweb.link/ipfs/{cid}` → `http://{cid}.ipfs.example.com`
@ -872,7 +842,7 @@ Below is a list of the most common public gateway setups.
Disable fetching of remote data (`NoFetch: true`) and resolving DNSLink at unknown hostnames (`NoDNSLink: true`).
Then, enable DNSLink gateway only for the specific hostname (for which data
is already present on the node), without exposing any content-addressing `Paths`:
```console
$ ipfs config --json Gateway.NoFetch true
$ ipfs config --json Gateway.NoDNSLink true
@ -904,7 +874,7 @@ Type: `string` (base64 encoded)
This section includes internal knobs for various subsystems to allow advanced users with big or private infrastructures to fine-tune some behaviors without the need to recompile Kubo.
**Be aware that making informed change here requires in-depth knowledge and most users should leave these untouched. All knobs listed here are subject to breaking changes between versions.**
**Be aware that making informed change here requires in-depth knowledge and most users should leave these untouched. All knobs listed here are subject to breaking changes between versions.**
### `Internal.Bitswap`
@ -1324,20 +1294,11 @@ It specifies the routing type that will be created.
Currently supported types:
- `reframe` (delegated routing based on the [reframe protocol](https://github.com/ipfs/specs/tree/main/reframe#readme))
- <del>`dht`</del> (WIP, custom DHT will be added in a future release)
- `dht`
- `parallel` and `sequential`: Helpers that can be used to run several routers sequentially or in parallel.
Type: `string`
#### `Routing.Routers: Enabled`
**EXPERIMENTAL: `Routing.Routers` configuration may change in future release**
Optional flag to disable the specified router without removing it from the configuration file.
Default: `true`
Type: `flag` (`null`/missing will apply the default)
#### `Routing.Routers: Parameters`
**EXPERIMENTAL: `Routing.Routers` configuration may change in future release**
@ -1346,7 +1307,26 @@ Parameters needed to create the specified router. Supported params per router ty
Reframe:
- `Endpoint` (mandatory): URL that will be used to connect to a specified router.
- `Priority` (optional): Priority is used when making a routing request. Small numbers represent more important routers. The default priority is 100000.
DHT:
- `"Mode"`: Mode used by the DHT. Possible values: "server", "client", "auto"
- `"AcceleratedDHTClient"`: Set to `true` if you want to use the experimentalDHT.
- `"PublicIPNetwork"`: Set to `true` to create a `WAN` DHT. Set to `false` to create a `LAN` DHT.
Parallel:
- `Routers`: A list of routers that will be executed in parallel:
- `Name:string`: Name of the router. It should be one of the previously added to `Routers` list.
- `Timeout:duration`: Local timeout. It accepts strings compatible with Go `time.ParseDuration(string)` (`10s`, `1m`, `2h`). Time will start counting when this specific router is called, and it will stop when the router returns, or we reach the specified timeout.
- `ExecuteAfter:duration`: Providing this param will delay the execution of that router at the specified time. It accepts strings compatible with Go `time.ParseDuration(string)` (`10s`, `1m`, `2h`).
- `IgnoreErrors:bool`: It will specify if that router should be ignored if an error occurred.
- `Timeout:duration`: Global timeout. It accepts strings compatible with Go `time.ParseDuration(string)` (`10s`, `1m`, `2h`).
Sequential:
- `Routers`: A list of routers that will be executed in order:
- `Name:string`: Name of the router. It should be one of the previously added to `Routers` list.
- `Timeout:duration`: Local timeout. It accepts strings compatible with Go `time.ParseDuration(string)`. Time will start counting when this specific router is called, and it will stop when the router returns, or we reach the specified timeout.
- `IgnoreErrors:bool`: It will specify if that router should be ignored if an error occurred.
- `Timeout:duration`: Global timeout. It accepts strings compatible with Go `time.ParseDuration(string)`.
**Examples:**
@ -1367,13 +1347,106 @@ Default: `{}` (use the safe implicit defaults)
Type: `object[string->string]`
### `Routing: Methods`
`Methods:map` will define which routers will be executed per method. The key will be the name of the method: `"provide"`, `"find-providers"`, `"find-peers"`, `"put-ipns"`, `"get-ipns"`. All methods must be added to the list.
The value will contain:
- `RouterName:string`: Name of the router. It should be one of the previously added to `Routing.Routers` list.
Type: `object[string->object]`
**Examples:**
To use the previously added `CidContact` reframe router on all methods:
```console
$ ipfs config Routing.Methods --json '{
"find-peers": {
"RouterName": "CidContact"
},
"find-providers": {
"RouterName": "CidContact"
},
"get-ipns": {
"RouterName": "CidContact"
},
"provide": {
"RouterName": "CidContact"
},
"put-ipns": {
"RouterName": "CidContact"
}
}'
```
Complete example using 3 Routers, reframe, DHT and parallel.
```
$ ipfs config Routing.Type --json '"custom"'
$ ipfs config Routing.Routers.CidContact --json '{
"Type": "reframe",
"Parameters": {
"Endpoint": "https://cid.contact/reframe"
}
}'
$ ipfs config Routing.Routers.WanDHT --json '{
"Type": "dht",
"Parameters": {
"Mode": "auto",
"PublicIPNetwork": true,
"AcceleratedDHTClient": false
}
}'
$ ipfs config Routing.Routers.ParallelHelper --json '{
"Type": "parallel",
"Parameters": {
"Routers": [
{
"RouterName" : "CidContact",
"IgnoreErrors" : true,
"Timeout": "3s"
},
{
"RouterName" : "WanDHT",
"IgnoreErrors" : false,
"Timeout": "5m",
"ExecuteAfter": "2s"
}
]
}
}'
ipfs config Routing.Methods --json '{
"find-peers": {
"RouterName": "ParallelHelper"
},
"find-providers": {
"RouterName": "ParallelHelper"
},
"get-ipns": {
"RouterName": "ParallelHelper"
},
"provide": {
"RouterName": "WanDHT"
},
"put-ipns": {
"RouterName": "ParallelHelper"
}
}'
```
### `Routing.Type`
There are two core routing options: "none" and "dht" (default).
There are three core routing options: "none", "dht" (default) and "custom".
* If set to "none", your node will use _no_ routing system. You'll have to
explicitly connect to peers that have the content you're looking for.
* If set to "dht" (or "dhtclient"/"dhtserver"), your node will use the IPFS DHT.
* If set to "custom", `Routing.Routers` will be used.
When the DHT is enabled, it can operate in two modes: client and server.
@ -1549,7 +1622,7 @@ Type: `optionalInteger`
#### `Swarm.RelayService.ReservationTTL`
Duration of a new or refreshed reservation.
Duration of a new or refreshed reservation.
Default: `"1h"`
@ -1788,7 +1861,7 @@ Configuration section for libp2p _network_ transports. Transports enabled in
this section will be used for dialing. However, to receive connections on these
transports, multiaddrs for these transports must be added to `Addresses.Swarm`.
Supported transports are: QUIC, TCP, WS, and Relay.
Supported transports are: QUIC, TCP, WS, Relay and WebTransport.
Each field in this section is a `flag`.
@ -1864,6 +1937,49 @@ Listen Addresses:
* This transport is special. Any node that enables this transport can receive
inbound connections on this transport, without specifying a listen address.
#### `Swarm.Transports.Network.WebTransport`
A new feature of [`go-libp2p`](https://github.com/libp2p/go-libp2p/releases/tag/v0.23.0)
is the [WebTransport](https://github.com/libp2p/go-libp2p/issues/1717) transport.
This is a spiritual descendant of WebSocket but over `HTTP/3`.
Since this runs on top of `HTTP/3` it uses `QUIC` under the hood.
We expect it to perform worst than `QUIC` because of the extra overhead,
this transport is really meant at agents that cannot do `TCP` or `QUIC` (like browsers).
For now it is **disabled by default** and considered **experimental**.
If you find issues running it please [report them to us](https://github.com/ipfs/kubo/issues/new).
In the future Kubo will listen on WebTransport by default for anyone already listening on QUIC addresses.
WebTransport is a new transport protocol currently under development by the IETF and the W3C, and already implemented by Chrome.
Conceptually, it’s like WebSocket run over QUIC instead of TCP. Most importantly, it allows browsers to establish (secure!) connections to WebTransport servers without the need for CA-signed certificates,
thereby enabling any js-libp2p node running in a browser to connect to any kubo node, with zero manual configuration involved.
The previous alternative is websocket secure, which require installing a reverse proxy and TLS certificates manually.
Default: Disabled
Type: `flag`
#### How to enable WebTransport
Thoses steps are temporary and wont be needed once we make it enabled by default.
1. Enable the WebTransport transport:
`ipfs config Swarm.Transports.Network.WebTransport --json true`
1. Add a listener address for WebTransport to your `Addresses.Swarm` key, for example:
```json
[
"/ip4/0.0.0.0/tcp/4001",
"/ip4/0.0.0.0/udp/4001/quic",
"/ip4/0.0.0.0/udp/4002/quic/webtransport"
]
```
1. Restart your daemon to apply the config changes.
### `Swarm.Transports.Security`
Configuration section for libp2p _security_ transports. Transports enabled in
@ -1961,7 +2077,7 @@ Example:
{
"DNS": {
"Resolvers": {
"eth.": "https://eth.link/dns-query",
"eth.": "https://dns.eth.limo/dns-query",
"crypto.": "https://resolver.unstoppable.io/dns-query",
"libre.": "https://ns1.iriseden.fr/dns-query",
".": "https://cloudflare-dns.com/dns-query"

462
docs/delegated-routing.md Normal file
View File

@ -0,0 +1,462 @@
# New multi-router configuration system
- Start Date: 2022-08-15
- Related Issues:
- https://github.com/ipfs/kubo/issues/9188
- https://github.com/ipfs/kubo/issues/9079
## Summary
Previously we only used DHT for content routing and content providing. After kubo-0.14.0 release we added support for [delegated routing using Reframe protocol](https://github.com/ipfs/kubo/pull/8997).
Now we need a better way to add different routers using different protocols like Reframe or DHT, and be able to configure them to cover different use cases.
## Motivation
The actual routing implementation is not enough. Some users needs to have more options when configuring the routing system. The new implementations should be able to:
- [x] Be user-friendly and easy enough to configure, but also versatile
- [x] Configurable Router execution order
- [x] Delay some of the Router methods execution when they will be executed on parallel
- [x] Configure which method of a giving router will be used
- [x] Mark some router methods as mandatory to make the execution fails if that method fails
## Detailed design
### Configuration file description
The `Routing` configuration section will contain the following keys:
#### Type
`Type` will be still in use to avoid complexity for the user that only wants to use Kubo with the default behavior. We are going to add a new type, `custom`, that will use the new router systems. `none` type will deactivate **all** routers, default dht and delegated ones.
#### Routers
`Routers` will be a key-value list of routers that will be available to use. The key is the router name and the value is all the needed configurations for that router. the `Type` will define the routing kind. The main router types will be `reframe` and `dht`, but we will implement two special routers used to execute a set of routers in parallel or sequentially: `parallel` router and `sequential` router.
Depending on the routing type, it will use different parameters:
##### Reframe
Params:
- `"Endpoint"`: URL endpoint implementing Reframe protocol.
##### DHT
Params:
- `"Mode"`: Mode used by the DHT. Possible values: "server", "client", "auto"
- `"AcceleratedDHTClient"`: Set to `true` if you want to use the experimentalDHT.
- `"PublicIPNetwork"`: Set to `true` to create a `WAN` DHT. Set to `false` to create a `LAN` DHT.
##### Parallel
Params:
- `Routers`: A list of routers that will be executed in parallel:
- `Name:string`: Name of the router. It should be one of the previously added to `Routers` list.
- `Timeout:duration`: Local timeout. It accepts strings compatible with Go `time.ParseDuration(string)`. Time will start counting when this specific router is called, and it will stop when the router returns, or we reach the specified timeout.
- `ExecuteAfter:duration`: Providing this param will delay the execution of that router at the specified time. It accepts strings compatible with Go `time.ParseDuration(string)`.
- `IgnoreErrors:bool`: It will specify if that router should be ignored if an error occurred.
- `Timeout:duration`: Global timeout. It accepts strings compatible with Go `time.ParseDuration(string)`.
##### Sequential
Params:
- `Routers`: A list of routers that will be executed in order:
- `Name:string`: Name of the router. It should be one of the previously added to `Routers` list.
- `Timeout:duration`: Local timeout. It accepts strings compatible with Go `time.ParseDuration(string)`. Time will start counting when this specific router is called, and it will stop when the router returns, or we reach the specified timeout.
- `IgnoreErrors:bool`: It will specify if that router should be ignored if an error occurred.
- `Timeout:duration`: Global timeout. It accepts strings compatible with Go `time.ParseDuration(string)`.
#### Methods
`Methods:map` will define which routers will be executed per method. The key will be the name of the method: `"provide"`, `"find-providers"`, `"find-peers"`, `"put-ipns"`, `"get-ipns"`. All methods must be added to the list. This will make configuration discoverable giving good errors to the user if a method is missing.
The value will contain:
- `RouterName:string`: Name of the router. It should be one of the previously added to `Routers` list.
#### Configuration file example:
```json
"Routing": {
"Type": "custom",
"Routers": {
"storetheindex": {
"Type": "reframe",
"Parameters": {
"Endpoint": "https://cid.contact/reframe"
}
},
"dht-lan": {
"Type": "dht",
"Parameters": {
"Mode": "server",
"PublicIPNetwork": false,
"AcceleratedDHTClient": false
}
},
"dht-wan": {
"Type": "dht",
"Parameters": {
"Mode": "auto",
"PublicIPNetwork": true,
"AcceleratedDHTClient": false
}
},
"find-providers-router": {
"Type": "parallel",
"Parameters": {
"Routers": [
{
"RouterName": "dht-lan",
"IgnoreErrors": true
},
{
"RouterName": "dht-wan"
},
{
"RouterName": "storetheindex"
}
]
}
},
"provide-router": {
"Type": "parallel",
"Parameters": {
"Routers": [
{
"RouterName": "dht-lan",
"IgnoreErrors": true
},
{
"RouterName": "dht-wan",
"ExecuteAfter": "100ms",
"Timeout": "100ms"
},
{
"RouterName": "storetheindex",
"ExecuteAfter": "100ms"
}
]
}
},
"get-ipns-router": {
"Type": "sequential",
"Parameters": {
"Routers": [
{
"RouterName": "dht-lan",
"IgnoreErrors": true
},
{
"RouterName": "dht-wan",
"Timeout": "300ms"
},
{
"RouterName": "storetheindex",
"Timeout": "300ms"
}
]
}
},
"put-ipns-router": {
"Type": "parallel",
"Parameters": {
"Routers": [
{
"RouterName": "dht-lan"
},
{
"RouterName": "dht-wan"
},
{
"RouterName": "storetheindex"
}
]
}
}
},
"Methods": {
"find-providers": {
"RouterName": "find-providers-router"
},
"provide": {
"RouterName": "provide-router"
},
"get-ipns": {
"RouterName": "get-ipns-router"
},
"put-ipns": {
"RouterName": "put-ipns-router"
}
}
}
```
Added YAML for clarity:
```yaml
---
Type: custom
Routers:
storetheindex:
Type: reframe
Parameters:
Endpoint: https://cid.contact/reframe
dht-lan:
Type: dht
Parameters:
Mode: server
PublicIPNetwork: false
AcceleratedDHTClient: false
dht-wan:
Type: dht
Parameters:
Mode: auto
PublicIPNetwork: true
AcceleratedDHTClient: false
find-providers-router:
Type: parallel
Parameters:
Routers:
- RouterName: dht-lan
IgnoreErrors: true
- RouterName: dht-wan
- RouterName: storetheindex
provide-router:
Type: parallel
Parameters:
Routers:
- RouterName: dht-lan
IgnoreErrors: true
- RouterName: dht-wan
ExecuteAfter: 100ms
Timeout: 100ms
- RouterName: storetheindex
ExecuteAfter: 100ms
get-ipns-router:
Type: sequential
Parameters:
Routers:
- RouterName: dht-lan
IgnoreErrors: true
- RouterName: dht-wan
Timeout: 300ms
- RouterName: storetheindex
Timeout: 300ms
put-ipns-router:
Type: parallel
Parameters:
Routers:
- RouterName: dht-lan
- RouterName: dht-wan
- RouterName: storetheindex
Methods:
find-providers:
RouterName: find-providers-router
provide:
RouterName: provide-router
get-ipns:
RouterName: get-ipns-router
put-ipns:
RouterName: put-ipns-router
```
### Error cases
- If any of the routers fails, the output will be an error by default.
- You can use `IgnoreErrors:true` to ignore errors for a specific router output
- To avoid any error at the output, you must ignore all router errors.
### Implementation Details
#### Methods
All routers must implement the `routing.Routing` interface:
```go=
type Routing interface {
ContentRouting
PeerRouting
ValueStore
Bootstrap(context.Context) error
}
```
All methods involved:
```go=
type Routing interface {
Provide(context.Context, cid.Cid, bool) error
FindProvidersAsync(context.Context, cid.Cid, int) <-chan peer.AddrInfo
FindPeer(context.Context, peer.ID) (peer.AddrInfo, error)
PutValue(context.Context, string, []byte, ...Option) error
GetValue(context.Context, string, ...Option) ([]byte, error)
SearchValue(context.Context, string, ...Option) (<-chan []byte, error)
Bootstrap(context.Context) error
}
```
We can configure which methods will be used per routing implementation. Methods names used in the configuration file will be:
- `Provide`: `"provide"`
- `FindProvidersAsync`: `"find-providers"`
- `FindPeer`: `"find-peers"`
- `PutValue`: `"put-ipns"`
- `GetValue`, `SearchValue`: `"get-ipns"`
- `Bootstrap`: It will be always executed when needed.
#### Routers
We need to implement the `parallel` and `sequential` routers and stop using `routinghelpers.Tiered` router implementation.
Add cycle detection to avoid to user some headaches.
Also we need to implement an internal router, that will define the router used per method.
#### Other considerations
- We need to refactor how DHT routers are created to be able to use and add any amount of custom DHT routers.
- We need to add a new `custom` router type to be able to use the new routing system.
- Bitswap WANT broadcasting is not included on this document, but it can be added in next iterations.
- This document will live in docs/design-notes for historical reasons and future reference.
## Test fixtures
As test fixtures we can add different use cases here and see how the configuration will look like.
### Mimic previous dual DHT config
```json
"Routing": {
"Type": "custom",
"Routers": {
"dht-lan": {
"Type": "dht",
"Parameters": {
"Mode": "server",
"PublicIPNetwork": false
}
},
"dht-wan": {
"Type": "dht",
"Parameters": {
"Mode": "auto",
"PublicIPNetwork": true
}
},
"parallel-dht-strict": {
"Type": "parallel",
"Parameters": {
"Routers": [
{
"RouterName": "dht-lan"
},
{
"RouterName": "dht-wan"
}
]
}
},
"parallel-dht": {
"Type": "parallel",
"Parameters": {
"Routers": [
{
"RouterName": "dht-lan",
"IgnoreError": true
},
{
"RouterName": "dht-wan"
}
]
}
}
},
"Methods": {
"provide": {
"RouterName": "dht-wan"
},
"find-providers": {
"RouterName": "parallel-dht-strict"
},
"find-peers": {
"RouterName": "parallel-dht-strict"
},
"get-ipns": {
"RouterName": "parallel-dht"
},
"put-ipns": {
"RouterName": "parallel-dht"
}
}
}
```
Yaml representation for clarity:
```yaml
---
Type: custom
Routers:
dht-lan:
Type: dht
Parameters:
Mode: server
PublicIPNetwork: false
dht-wan:
Type: dht
Parameters:
Mode: auto
PublicIPNetwork: true
parallel-dht-strict:
Type: parallel
Parameters:
Routers:
- RouterName: dht-lan
- RouterName: dht-wan
parallel-dht:
Type: parallel
Parameters:
Routers:
- RouterName: dht-lan
IgnoreError: true
- RouterName: dht-wan
Methods:
provide:
RouterName: dht-wan
find-providers:
RouterName: parallel-dht-strict
find-peers:
RouterName: parallel-dht-strict
get-ipns:
RouterName: parallel-dht
put-ipns:
RouterName: parallel-dht
```
### Compatibility
~~We need to create a config migration using [fs-repo-migrations](https://github.com/ipfs/fs-repo-migrations). We should remove the `Routing.Type` param and add the configuration specified [previously](#Mimic-previous-dual-DHT-config).~~
We don't need to create any config migration! To avoid to the users the hassle of understanding how the new routing system works, we are gonna keep the old behavior. We will add the Type `custom` to make available the new Routing system.
### Security
No new security implications or considerations were found.
### Alternatives
I got ideas from all of the following links to create this design document:
- https://github.com/ipfs/kubo/issues/9079#issuecomment-1211288268
- https://github.com/ipfs/kubo/issues/9157
- https://github.com/ipfs/kubo/issues/9079#issuecomment-1205000253
- https://www.notion.so/pl-strflt/Delegated-Routing-Thoughts-very-very-WIP-0543bc51b1bd4d63a061b0f28e195d38
- https://gist.github.com/guseggert/effa027ff4cbadd7f67598efb6704d12
### Copyright
Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).

Some files were not shown because too many files have changed in this diff Show More