This also means that rb-pinning-service-api is no longer required for
running remote pinning tests. This alone saves at least 3 minutes in
test runtime in CI because we don't need to checkout the repo, build
the Docker image, run it, etc.
Instead this implements a simple pinning service in Go that the test
runs in-process, with a callback that can be used to control the async
behavior of the pinning service (e.g. simulate work happening
asynchronously like transitioning from "queued" -> "pinning" ->
"pinned").
This also adds an environment variable to Kubo to control the MFS
remote pin polling interval, so that we don't have to wait 30 seconds
in the test for MFS changes to be repinned. This is purely for tests
so I don't think we should document this.
This entire test suite runs in around 2.5 sec on my laptop, compared to
the existing 3+ minutes in CI.
New multi-router configuration system based on https://hackmd.io/G1KRDEX5T3qyfoBMkIrBew#Methods
- Added a new routing type: "custom"
- Added specific struct types for different Routers (instead of map[string]interface{})
- Added `Duration` config type, to make easier time string parsing
- Added config documentation.
- Use the latest go-delegated-routing library version with GET support.
- Added changelog notes for this feature.
It:
- closes#9157
- closes#9079
- closes#9186
The file contains the gateway your node is hosting in the http://<host>:<port> RFC 3986 format.
Structurally it works exactly the same as the API file.
* Delegated Routing.
Implementation of Reframe specs (https://github.com/ipfs/specs/blob/master/REFRAME.md) using go-delegated-routing library.
* Requested changes.
* Init using op string
* Separate possible ContentRouters for TopicDiscovery.
If we don't do this, we have a ciclic dependency creating TieredRouter.
Now we can create first all possible content routers, and after that,
create Routers.
* Set dht default routing type
* Add tests and remove uneeded code
* Add documentation.
* docs: Routing.Routers
* Requested changes.
Signed-off-by: Antonio Navarro Perez <antnavper@gmail.com>
* Add some documentation on new fx functions.
* Add changelog entry and integration tests
* test: sharness for 'dht' in 'routing' commands
Since 'routing' is currently the same as 'dht' (minus query command)
we need to test both, that way we won't have unnoticed divergence
in the default behavior.
* test(sharness): delegated routing via reframe URL
* Add more tests for delegated routing.
* If any put operation fails, the tiered router will fail.
* refactor: Routing.Routers: Parameters.Endpoint
As agreed in https://github.com/ipfs/kubo/pull/8997#issuecomment-1175684716
* Try to improve CHANGELOG entry.
* chore: update reframe spec link
* Update go-delegated-routing dependency
* Fix config error test
* use new changelog format
* Remove port conflict
* go mod tidy
* ProviderManyWrapper to ProviderMany
* Update docs/changelogs/v0.14.md
Co-authored-by: Adin Schmahmann <adin.schmahmann@gmail.com>
Co-authored-by: Marcin Rataj <lidel@lidel.org>
Co-authored-by: Adin Schmahmann <adin.schmahmann@gmail.com>
* feat: add block profiling to collect-profiles.sh
* feat: add more profiles to 'ipfs diag profile'
This adds mutex and block profiles, and brings the command up-to-par
with 'collect-profiles.sh', so that we can remove it.
Profiles are also now collected concurrently, which improves the
runtime from (profile_time * num_profiles) to just (profile_time).
Note that this has a backwards-incompatible change, removing
--cpu-profile-time in favor of the more general --profile-time, which
covers all sampling profiles.
* docs(cli): ipfs diag profile
* add CLI flag to select specific diag collectors
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* add deprecation warning when tracer plugins are loaded
* add response format attribute to span in gateway handler
* add note about tracing's experimental status in godoc
* add nil check for TTL when adding name span attrs
* add basic sharness test for integration with otel collector
* add nil check in UnixFSAPI.processLink
* test: sharness check all json objs for swarm span
* add env var docs to docs/environment-variables.md
* chore: pin the otel collector version
* add tracing spans per response type (#8841)
* docs: tracing with jaeger-ui
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* fix: installation without sudo
I think it's good practice that users don't need to run `sudo`
to install something just to try it out. With this change, the
local bin directory is tried first, which usually is also in the
`PATH`. This way the installation script can be run without
`sudo` and should still work.
* fix: try local path last and support spaces in $HOME
Try the `$HOME/.local/bin` path last, so that the script is backwards
compatible. Also make sure that it works even if there are spaces in
the directory set by `$HOME`.
* Allow pubsub and namesys-pubsub to be enabled via config
Signed-off-by: Joe Holden <jwh@zorins.us>
Co-authored-by: Marcin Rataj <lidel@lidel.org>
Co-authored-by: Adin Schmahmann <adin.schmahmann@gmail.com>
* Expose additional migration APIs
Expose migration APIs for reading migration config and creating migration fetchers. This allows implementation of commands and external applications that want to retrieve migrations according to the Migration portion of the IPFS config.
This change also moves some functionality that is specific to fetching migrations via IPFS into the `ipfsfetcher` package.
* pinmfs: log error if pin failed
* refactor: log pre-existing remote mfs pins
This cleans up debug logging and ensures we use %q for data read from
remote services. We log pre-existing pins for MFS root and print ERROR
if pin for current root has status "failed", otherwise, we log to debug.
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* Enable downloading migrations over IPFS
There are now options in the config file that control how migrations are downloaded. This includes enabling downloading migrations using IPFS by (when migrations are required) spinning up a temporary node for fetching the migrations before running them. There is also a config option to decide what to do with the migrations binaries once they are downloaded (e.g. cache or pin them in your node, or just throw out the data).
Co-authored-by: Steven Allen <steven@stebalien.com>
The InitializeKeyspace functionality is go-ipfs specific but lives in the
go-namesys repo.
It is only called from initializeIpnsKeyspace, and therefore the code is
directly added to this function. The original function will be removed from
go-namesys.
Namesys is a very useful submodule. Given a ValueStore and a Datastore it can
resolve and publish /ipns/ paths.
This functionality does not need to be sequestered inside go-ipfs as it can
and should be used without IPFS, for example, for implementing lightweight
IPNS publishing services or for resolving /ipns/ paths.
"keystore" extraction was necessary, as there is a dependency to it in
namesys. Keystore is also a useful module by itself within the stack.
Fixes#6537
We've had a reliable and enabled by default TLS implementation since
0.4.23 (over a year ago) and turned off SECIO in September of last year.
We might as well remove support entirely in the next release and
encourage users to upgrade their networks.
Noise is faster, anyways.
The code in this PR finds the necessary mirgations, downloads the latest version of them from the distribution site, unpacks the executables, and runs the migrations in order.
This code is also used to build the ipfs-update tool and the fs-repo-migrations tool. Note: the fs-repo-migrations tool is only used to run stand-alone migrations now and is not used by either go-ipfs or ipfs-update.
Additional utility is provided by this PR, that is not specific to migrations:
- Find local ipfs directory
- Get current repo version
- Check for ipfs daemon availability
- Get version information about any distribution on distribution site
- Fetch and unpack any binary executable over ipfs or http
* Improve error message when running key commands that must be run when the daemon is not already running
Fixes Issue #7814
`ipfs key export` now does a PreRun check like `ipfs key rotate` was to give a better error to the user then "someone else has the lock" in the event that the daemon is running while trying to execute these offline-only commands.
While unlikely the "someone else has the lock" error can still be shown if two processes try and grab the repo lock at the same time.
This PreRun function is also exported so it can be used by `ipfs init` where it was originally copied from.
* Added more `ipfs key` command tests
When daemon is running:
- Test that import works
- Test that export fails
- Test that rotate fails
* add support for choosing a peer key type (e.g. RSA or Ed25519) when initializing the repo
* test all variants of ipfs init: RSA, Ed25519 and default
* update subdomain gateway sharness test to publish IPNS using RSA and
Ed25519 keys
* use default identity bit lengths defined in config repo instead of
having separate defaults in go-ipfs
* update config repo dependency
Co-authored-by: Will Scott <will@cypherpunk.email>
Co-authored-by: Petar Maymounkov <petarm@gmail.com>
Adds `ipfs_info` prometheus metric with version and commit info
```prometheus
ipfs_info{commit="9ea7c6a11-dirty",version="0.5.0-dev"} 1
```
This follows the same pattern as go and other systems, adding a gauge metric that is set to 1, with the version info addeds as labels.
This is a common pattern for prometheus. It lets operators merge version info into other prometheus metrics by multiplying it with the other stat, as described in https://www.robustperception.io/exposing-the-software-version-to-prometheus
For example, we already expose the go version info as
```prometheus
go_info{version="go1.12.9"} 1
```
License: MIT
Signed-off-by: Oli Evans <oli@tableflip.io>
