* multibase encoding on pubsub
* emit multibase for json clients
* refactor(pubsub): base64url for all URL args
This makes it easier to reason about.
Also added better helptext to each command explaining how the binary
data is encoded on the wire, and how to process it in userland.
* refactor: remove ndpayload and lenpayload
Those output formats are undocumented and seem to be only used in tests.
This change removes their implementation and replaces it with error
message to use JSON instead.
I also refactored tests to test the --enc=json response format instead
of imaginary one, making tests more useful as they also act as
regression tests for HTTP RPC.
* test(pubsub): go-ipfs-api
Testing against compatible version from
https://github.com/ipfs/go-ipfs-api/pull/255
* refactor: safeTextListEncoder
Making it clear what it does and why
* refactor(pubsub): unify peerids
This ensures `ipfs pubsub sub` returns the same peerids in the `From`
field as `ipfs pubsub peers`.
libp2p already uses base encoding, no need to double wrap or use custom
multibase.
* test(pubsub): go-ipfs-http-client
* refactor(pubsub): make pub command read from a file
We want to send payload in the body as multipart so users can use
existing tools like curl for publishing arbitrary bytes to a topic.
StringArg was created for "one message per line" use case, and if data
has `\n` or `\r\n` byte sequences, it will cause payload to be split. It
is not possible to undo this, because mentioned sequences are lost, so
we are not able to tell if it was `\n` or `\r\n`
We already avoid this problem in `block put` and `dht put` by reading
payload via FileArg which does not mangle binary data and send it as-is.
It feel like `pubsub pub` should be using it in the first place anyway,
so this commit replaces StringArg with FileArg.
This also closes https://github.com/ipfs/go-ipfs/issues/8454
and makes rpc in go-ipfs easier to code against.
* test(pubsub): publishing with line breaks
Making sure we don't see regressions in the future.
Ref. https://github.com/ipfs/go-ipfs/issues/7939
* chore: disable pubsub interop for now
See
344f692d8c
* test: t0322-pubsub-http-rpc.sh
- Adds HTTP RPC regression test that ensures topic is encoded as URL-safe
multibase.
- Moves pubsub tests to live in unique range ./t032x
* fix(ci): js-ipfs with fixed pubsub wire format
uses js-ipfs from https://github.com/ipfs/js-ipfs/pull/3922
until js-ipfs release can ship with dependency on go-ipfs 0.11.0-rc1
Co-authored-by: Marcin Rataj <lidel@lidel.org>
Co-authored-by: Adin Schmahmann <adin.schmahmann@gmail.com>
* plumb through go-datastore context changes
* update go-libp2p to v0.16.0
* use LIBP2P_TCP_REUSEPORT instead of IPFS_REUSEPORT
* use relay config
* making deprecation notice match the go-ipfs-config key
* docs(config): circuit relay v2
* docs(config): fix links and headers
* feat(config): Internal.Libp2pForceReachability
This switches to config that supports setting and reading
Internal.Libp2pForceReachability OptionalString flag
* use configuration option for static relays
* chore: go-ipfs-config v0.18.0
https://github.com/ipfs/go-ipfs-config/releases/tag/v0.18.0
* feat: circuit v1 migration prompt when Swarm.EnableRelayHop is set (#8559)
* exit when Swarm.EnableRelayHop is set
* docs: Experimental.ShardingEnabled migration
This ensures existing users of global sharding experiment get notified
that the flag no longer works + that autosharding happens automatically.
For people who NEED to keep the old behavior (eg. have no time to
migrate today) there is a note about restoring it with
`UnixFSShardingSizeThreshold`.
* chore: add dag-jose code to the cid command output
* add support for setting automatic unixfs sharding threshold from the config
* test: have tests use low cutoff for sharding to mimic old behavior
* test: change error message to match the current error
* test: Add automatic sharding/unsharding tests (#8547)
* test: refactored naming in the sharding sharness tests to make more sense
* ci: set interop test executor to convenience image for Go1.16 + Node
* ci: use interop master
Co-authored-by: Marcin Rataj <lidel@lidel.org>
Co-authored-by: Marten Seemann <martenseemann@gmail.com>
Co-authored-by: Marcin Rataj <lidel@lidel.org>
Co-authored-by: Gus Eggert <gus@gus.dev>
Co-authored-by: Lucas Molas <schomatis@gmail.com>
* chore: replace go-merkledag walk with go-ipld-prime traversal for dag export
This is "safe" now because we can limit duplicate block loads like
go-merkledag does and won't get trapped taking a long time for complex
DAGs. We can do this while we're using an exhaustive selector (like
ExploreAll here) but will need an alternative strategy when we go for
arbitrary selectors.
* fix(cli): object add-link: do not allow blocks over BS limit
* refactor: allow-big-block
- renamed override flag to --allow-big-block
- separate tests for default and override behavior
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* feat: report block count on `dag import`
* fix: clean-up dag import message format
* Only print stats when --stats flag is passed
This applies to both text and json output encoding.
- Stats data is now contained within a Stats datastructure
- Stats are printed after root so that first line of output is the same as previously, even when stats are output using --stats
* fix sharness test
* Add PayloadBytesCount to stats
* Attempt to stabilize flaky tests
* Rename PayloadBytesCount to BlockBytesCount
* Correctly calculate size or imported dag
* Use RawSize of original block for import bytes calc
* test: dag import without --stats
basic regression tests for the default output (text and json)
Co-authored-by: gammazero <gammazero@users.noreply.github.com>
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* Add a transcoder command to multibase
In order to more easily facilitate the conversion
between multibase formats, include a transcode command
to avoid `multibase decode | multibase encode`
* Example code needed go mod tidy
Co-authored-by: gammazero <gammazero@users.noreply.github.com>
* feat: switch to using go-ipld-prime for codecs, path resolution, and the `dag put/get` commands
* fix: `dag put/get` not roundtripping due to an extra new line being added (https://github.com/ipfs/go-ipfs/issues/3503)
More detailed information is in the CHANGELOG.md file. Very high level:
* IPLD codecs (and their plugins) must use go-ipld-prime
* Added support for the dag-json codec
* `dag get/put` use IPLD codec names from the multicodec table
* `dag get` defaults to dag-json output instead of json, but may output with other codecs
* Data model pathing can be achieved using the /ipld prefix. For example, you can use `/ipld/QmFoo/Links/0/Hash` to traverse through a DagPB node
* With `dag get/put` the DagPB field names have been changed to match the ones in the protobuf listed in the specification
Co-authored-by: hannahhoward <hannah@hannahhoward.net>
Co-authored-by: Daniel Martí <mvdan@mvdan.cc>
Co-authored-by: acruikshank <acruikshank@example.com>
Co-authored-by: Steven Allen <steven@stebalien.com>
Co-authored-by: Will Scott <will.scott@protocol.ai>
Co-authored-by: Will Scott <will@cypherpunk.email>
Co-authored-by: Rod Vagg <rod@vagg.org>
Co-authored-by: Adin Schmahmann <adin.schmahmann@gmail.com>
Co-authored-by: Eric Myhre <hash@exultant.us>
This should replace the "collect-profiles.sh" script and allow users to
easily collect profiles.
At the moment, it just dumps all profiles into a single zip file. It
does this server-side so it's easy fetch them with curl.
In the future, it would be nice to add:
1. Progress indicators (cpu profiles take 30 seconds).
2. An option to specify which profiles to collect.
But we can handle that later.
Unfortunately, this command doesn't produce symbolized svgs as I didn't
want to depend on having a local go compiler.
These are missing some of the features of the current hand-rolled
completions, but:
1. Are less buggy.
2. Cover _all_ commands.
3. Don't need to be manually maintained (which we never do anyways).
fixes#4551fixes#8033
The batched provider system is enabled when the experimental AcceleratedDHTClient is enabled
There is also an `ipfs stats provide` command which gives stats about the providing/reproviding system when the batched provider system is enabled
Export does not require repo lock and it is safe to do even when ipfs
daemon is running. This enables apps like Brave browser to do
import/export without stopping/starting daemon.
Ref. https://github.com/brave/brave-browser/issues/15422
- add dag/export to read-only root. it's ready only, so why the heck not!
- test that i didn't expose other dag subcommands
§
License: MIT
Signed-off-by: Oli Evans <oli@tableflip.io>
