* Initial pass at Telemetry plugin Currently, IP Shipyard, with the help of Probelab, monitor and extract Amino/IPFS public network metrics with the use of DHT crawlers and bootstrappers (via peerlog plugin). For example, we log all peer IDs seen and their AgentVersion/Addresses obtained from the `identify` protocol, which provides insights into protocol usage, total number of peers etc. We would like to increase the ability to obtain more insights from the network by collecting some more information in the future, but also to give users more control over this collection (i.e. opt-out). The information collected will not allow unique identification of anyone and is only used for aggregation. Now, this PR explores a way of moving in this direction: * A new "telemetry" fx plugin is in charge of dealing with telemetry * The FX plugin allows to plug and make decisions / take actions during the setup phase: * We can inspect whether we are using Private Networks before the libp2p.Host has been initialized. * We can send telemetry after the libp2p Host is initialized. * Everything is self-contained. Custom builds can remove the plugin altogether without needing to surgically edit the code. As for behaviour: * The user can opt-in/out via EnvVar, file in the repo path or plugin configuration. * Users on private networks or with custom bootstrappers are detected, offered a wall of text explaining why we need telemetry and invited to opt-in. Opt-out happens otherwise on a timeout (with no input). Their preferences are stored. * Users on standard settings are opted-in by default. This is the status quo in Kubo already, except they don't get a chance to opt out. The telemetry libp2p protocol is yet to be defined, but expect something similar to identify, with a protobuf being pushed to bootstrappers or to a specific telemetry node that we define. In the case of pnets, this will be done with a temporary peer. * checkpoint * telemetry plugin: second pass * On first run it generates a UUID and shows a message to the user. * UUID is persistend to "telemetry_uuid" * Sends telemetry 1 minute after boot and every 24h * LogEvent is the thing containing all the telemetry that is sent * Opt-out possible via env-var or plugin configuration * Telemetry: add changelog and environment variable documentation * docs: improved daemon message making it more obvious nothing was sent yet and that user had 15m to out-out plus some debug logs that confirm opt-out * refactor: rename IPFS_TELEMETRY_MODE to IPFS_TELEMETRY * fix: add User-Agent header to telemetry requests --------- Co-authored-by: Andrew Gillis <11790789+gammazero@users.noreply.github.com> Co-authored-by: Marcin Rataj <lidel@lidel.org>
6.7 KiB
Plugins
Since 0.4.11 Kubo has an experimental plugin system that allows augmenting the daemons functionality without recompiling.
When an IPFS node is started, it will load plugins from the $IPFS_PATH/plugins
directory (by default ~/.ipfs/plugins).
Table of Contents
Plugin Types
Plugins can implement one or more plugin types, defined in the plugin package.
IPLD
IPLD plugins add support for additional formats to ipfs dag and other IPLD
related commands.
Datastore
Datastore plugins add support for additional datastore backends.
Tracer
(experimental)
Tracer plugins allow injecting an opentracing backend into Kubo.
Daemon
Daemon plugins are started when the Kubo daemon is started and are given an instance of the CoreAPI. This should make it possible to build an ipfs-based application without IPC and without forking Kubo.
Note: We eventually plan to make Kubo usable as a library. However, this plugin type is likely the best interim solution.
fx (experimental)
Fx plugins let you customize the fx dependency graph and configuration,
by customizing thefx.Options that are passed to fx when the Kubo node is initialized.
For example, you can override an interface such as exchange.Interface
or pin.Pinner with a custom implementation by appending an option like
fx.Decorate(func() exchange.Interface { return customExchange }).
Fx supports some advanced customization. Simple interface replacements like above are unlikely to break in the future,
but the more invasive your changes, the more likely they are to break between releases. Kubo cannot guarantee backwards
compatibility for fx customizations.
Fx options are applied across every execution of the ipfs binary, including:
- Repo initialization
- Daemon
- Applying migrations
- etc.
So if you plug in a blockservice that disallows non-allowlisted CIDs, then this may break migrations that fetch migration code over the IPFS network.
Internal
(never stable)
Internal plugins are like daemon plugins except that they can access, replace, and modify all internal state. Use this plugin type to extend Kubo in arbitrary ways. However, be aware that your plugin will likely break every time Kubo updated.
Configuration
Plugins can be configured in the Plugins section of the config file. Here,
plugins can be:
- Passed an arbitrary config object via the
Configfield. - Disabled via the
Disabledfield.
Example:
{
// ...
"Plugins": {
"Plugins": {
// plugin named "plugin-foo"
"plugin-foo": {
"Config": { /* arbitrary json */ }
},
// plugin named "plugin-bar"
"plugin-bar": {
"Disabled": true // "plugin-bar" will not be loaded
}
}
}
}
Available Plugins
| Name | Type | Preloaded | Description |
|---|---|---|---|
| git | IPLD | x | An IPLD format for git objects. |
| badgerds | Datastore | x | A high performance but experimental datastore. |
| flatfs | Datastore | x | A stable filesystem-based datastore. |
| levelds | Datastore | x | A stable, flexible datastore backend. |
| jaeger | Tracing | An opentracing backend. | |
| telemetry | Telemetry | x | Collects anonymized usage data for Kubo development. |
- Preloaded plugins are built into the Kubo binary and do not need to be installed separately. At the moment, all in-tree plugins are preloaded.
Installing Plugins
Kubo supports two types of plugins: External and Preloaded.
- External plugins must be installed in
$IPFS_PATH/plugins/(usually~/.ipfs/plugins/). - Preloaded plugins are built-into the Kubo when it's compiled.
External Plugin
The advantage of an external plugin is that it can be built, packaged, and installed independently of Kubo. Unfortunately, this method is only supported on Linux and MacOS at the moment. Users of other operating systems should follow the instructions for preloaded plugins.
In-tree
To build plugins included in plugin/plugins, run:
kubo$ make build_plugins
kubo$ ls plugin/plugins/*.so
To install, copy desired plugins to $IPFS_PATH/plugins. For example:
kubo$ mkdir -p ~/.ipfs/plugins/
kubo$ cp plugin/plugins/git.so ~/.ipfs/plugins/
kubo$ chmod +x ~/.ipfs/plugins/git.so # ensure plugin is executable
Finally, restart daemon if it is running.
Out-of-tree
To build out-of-tree plugins, use the plugin's Makefile if provided. Otherwise, you can manually build the plugin by running:
myplugin$ go build -buildmode=plugin -o myplugin.so myplugin.go
Finally, as with in-tree plugins:
- Install the plugin in
$IPFS_PATH/plugins. - Mark the plugin as executable (
chmod +x $IPFS_PATH/plugins/myplugin.so). - Restart your IPFS daemon (if running).
Preloaded Plugins
The advantages of preloaded plugins are:
- They're bundled with the Kubo binary.
- They work on all platforms.
To preload a Kubo plugin:
- Add the plugin to the preload list:
plugin/loader/preload_list - Build ipfs
kubo$ make build
You can also preload an in-tree but disabled-by-default plugin by adding it to the IPFS_PLUGINS variable. For example, to enable plugins foo, bar, and baz:
kubo$ make build IPFS_PLUGINS="foo bar baz"
Creating A Plugin
To create your own out-of-tree plugin, use the example plugin as a starting point. When you're ready, submit a PR adding it to the list of available plugins.