IPLD Prime In IPFS: Target Merge Branch (#7976)

* feat: switch to using go-ipld-prime for codecs, path resolution, and the `dag put/get` commands
* fix: `dag put/get` not roundtripping due to an extra new line being added (https://github.com/ipfs/go-ipfs/issues/3503)

More detailed information is in the CHANGELOG.md file. Very high level:
* IPLD codecs (and their plugins) must use go-ipld-prime
* Added support for the dag-json codec
* `dag get/put` use IPLD codec names from the multicodec table
* `dag get` defaults to dag-json output instead of json, but may output with other codecs
* Data model pathing can be achieved using the /ipld prefix. For example, you can use `/ipld/QmFoo/Links/0/Hash` to traverse through a DagPB node
* With `dag get/put` the DagPB field names have been changed to match the ones in the protobuf listed in the specification

Co-authored-by: hannahhoward <hannah@hannahhoward.net>
Co-authored-by: Daniel Martí <mvdan@mvdan.cc>
Co-authored-by: acruikshank <acruikshank@example.com>
Co-authored-by: Steven Allen <steven@stebalien.com>
Co-authored-by: Will Scott <will.scott@protocol.ai>
Co-authored-by: Will Scott <will@cypherpunk.email>
Co-authored-by: Rod Vagg <rod@vagg.org>
Co-authored-by: Adin Schmahmann <adin.schmahmann@gmail.com>
Co-authored-by: Eric Myhre <hash@exultant.us>
This commit is contained in:
Hannah Howard 2021-08-17 10:32:49 -07:00 committed by GitHub
parent 360aff45a7
commit f63a997c35
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
19 changed files with 522 additions and 233 deletions

View File

@ -1,5 +1,28 @@
# go-ipfs changelog
## v0.10.0 TBD
**IPLD Levels Up**
The handling of data serialization as well as many aspects of DAG traversal and pathing have been migrated from older libraries, including [go-merkledag](https://github.com/ipfs/go-merkledag) and [go-ipld-format](https://github.com/ipfs/go-ipld-format) to the new **[go-ipld-prime](https://github.com/ipld/go-ipld-prime)** library and its components. This allows us to use many of the newer tools afforded by go-ipld-prime, stricter and more uniform codec implementations, support for additional (pluggable) codecs, and some minor performance improvements.
This is significant refactor of a core component that touches many parts of IPFS, and does come with some **breaking changes**:
* **IPLD plugins**:
* The `PluginIPLD` interface has been changed to utilize go-ipld-prime. There is a demonstration of the change in the [bundled git plugin](./plugin/plugins/git/).
* **The semantics of `dag put` and `dag get` change**:
* `dag get` now takes the `format` option which accepts a multicodec name used to encode the output. By default this is `dag-json`. Users may notice differences from the previously plain Go JSON output, particularly where bytes are concerned which are now encoded using a form similar to CIDs: `{"/":{"bytes":"unpadded-base64-bytes"}}` rather than the previously Go-specific plain padded base64 string. See the [dag-json specification](https://ipld.io/specs/codecs/dag-json/spec/) for an explanation of these forms.
* `dag get` no longer prints an additional new-line character at the end of the encoded block output. This means that the output as presented by `dag get` are the exact bytes of the requested node. A round-trip of such bytes back in through `dag put` using the same codec should result in the same CID.
* `dag put` uses the `input-enc` option to specify the multicodec name of the format data is being provided in, and the `format` option to specify the multicodec multicodec name of the format the data should be stored in. These formerly defaulted to `json` and `cbor` respectively. They now default to `dag-json` and `dag-cbor` respectively but may be changed to any supported codec (bundled or loaded via plugin) by its [multicodec name](https://github.com/multiformats/multicodec/blob/master/table.csv).
* The `json` and `cbor` multicodec names (as used by `input-enc` and `format` options) are now no longer aliases for `dag-json` and `dag-cbor` respectively. Instead, they now refer to their proper [multicodec](https://github.com/multiformats/multicodec/blob/master/table.csv) types. `cbor` refers to a plain CBOR format, which will not encode CIDs and does not have strict deterministic encoding rules. `json` is a plain JSON format, which also won't encode CIDs and will encode bytes in the Go-specific padded base64 string format rather than the dag-json method of byte encoding. See https://ipld.io/specs/codecs/ for more information on IPLD codecs.
* The **dag-pb codec**, which is used to encode UnixFS data for IPFS, is now represented in a form via the `dag` API that mirrors the protobuf schema used to define the binary format and unifies the implementations and specification of dag-pb across the IPLD and IPFS stacks. Previously, additional layers of code within IPFS between protobuf serialization and UnixFS handling for file and directory data, obscured the forms that are described by the protobuf representation. Much of this code has now been replaced and there are fewer layers of transformation. This means that interacting with dag-pb data via the `dag` API will use different forms:
* Previously, using `dag get` on a dag-pb block would present the block serialized as JSON as `{"data":"padded-base64-bytes","links":[{"Name":"foo","Size":100,"Cid":{"/":"Qm..."}},...]}`.
* Using the dag-pb data model specification and the new default dag-json codec for output, this will now be serialized as: `{"Data":{"/":{"bytes":"unpadded-base64-bytes"}},"Links":[{"Name":"foo","Tsize":100,"Hash":{"/":"Qm..."}},...]}`. Aside from the change in byte formatting, most field names have changed: `data``Data`, `links``Links`, `Size``Tsize`, `Cid``Hash`. Note that this output can be changed now using the `--format` option to specify an alternative codec.
* Using `dag put` and a `format` option of `dag-pb` now requires that the input conform to this dag-pb specified form. Previously, input using `{"data":"...","links":[...]}` was accepted, now it must be `{"Data":"...","Links":[...]}`.
* Previously it was not possible to use paths to navigate to any of these properties of a dag-pb node, the only possible paths were named links, e.g. `dag get QmFoo/NamedLink` where `NamedLink` was one of the links whose name was `NamedLink`. This functionality remains the same, but by prefixing the path with `/ipld/` we enter data model pathing semantics and can `dag get /ipld/QmFoo/Links/0/Hash` to navigate to links or `/ipld/QmFoo/Data` to simply retrieve the data section of the node, for example.
* See the [dag-pb specification](https://ipld.io/specs/codecs/dag-pb/) for details on the codec and its data model representation.
* See this [detailed write-up](https://github.com/ipld/ipld/blob/master/design/tricky-choices/dag-pb-forms-impl-and-use.md) for further background on these changes.
## v0.9.1 2021-07-20
This is a small bug fix release resolving the following issues:

View File

@ -77,10 +77,10 @@ into an object of the specified format.
cmds.FileArg("object data", true, true, "The object to put").EnableStdin(),
},
Options: []cmds.Option{
cmds.StringOption("format", "f", "Format that the object will be added as.").WithDefault("cbor"),
cmds.StringOption("input-enc", "Format that the input object will be.").WithDefault("json"),
cmds.StringOption("format", "f", "Format that the object will be added as.").WithDefault("dag-cbor"),
cmds.StringOption("input-enc", "Format that the input object will be.").WithDefault("dag-json"),
cmds.BoolOption("pin", "Pin this object when adding."),
cmds.StringOption("hash", "Hash function to use").WithDefault(""),
cmds.StringOption("hash", "Hash function to use").WithDefault("sha2-256"),
},
Run: dagPut,
Type: OutputObject{},
@ -108,6 +108,9 @@ format.
Arguments: []cmds.Argument{
cmds.StringArg("ref", true, false, "The object to get").EnableStdin(),
},
Options: []cmds.Option{
cmds.StringOption("format", "f", "Format that the object will be serialized as.").WithDefault("dag-json"),
},
Run: dagGet,
}

View File

@ -1,11 +1,18 @@
package dagcmd
import (
"strings"
"fmt"
"io"
"github.com/ipfs/go-ipfs/core/commands/cmdenv"
ipldlegacy "github.com/ipfs/go-ipld-legacy"
"github.com/ipfs/interface-go-ipfs-core/path"
"github.com/ipld/go-ipld-prime"
"github.com/ipld/go-ipld-prime/multicodec"
"github.com/ipld/go-ipld-prime/traversal"
mc "github.com/multiformats/go-multicodec"
cmds "github.com/ipfs/go-ipfs-cmds"
)
@ -15,6 +22,12 @@ func dagGet(req *cmds.Request, res cmds.ResponseEmitter, env cmds.Environment) e
return err
}
format, _ := req.Options["format"].(string)
var fCodec mc.Code
if err := fCodec.Set(format); err != nil {
return err
}
rp, err := api.ResolvePath(req.Context, path.New(req.Arguments[0]))
if err != nil {
return err
@ -25,14 +38,34 @@ func dagGet(req *cmds.Request, res cmds.ResponseEmitter, env cmds.Environment) e
return err
}
var out interface{} = obj
universal, ok := obj.(ipldlegacy.UniversalNode)
if !ok {
return fmt.Errorf("%T is not a valid IPLD node", obj)
}
finalNode := universal.(ipld.Node)
if len(rp.Remainder()) > 0 {
rem := strings.Split(rp.Remainder(), "/")
final, _, err := obj.Resolve(rem)
remainderPath := ipld.ParsePath(rp.Remainder())
finalNode, err = traversal.Get(finalNode, remainderPath)
if err != nil {
return err
}
out = final
}
return cmds.EmitOnce(res, &out)
encoder, err := multicodec.LookupEncoder(uint64(fCodec))
if err != nil {
return fmt.Errorf("invalid encoding: %s - %s", format, err)
}
r, w := io.Pipe()
go func() {
defer w.Close()
if err := encoder(finalNode, w); err != nil {
_ = w.CloseWithError(err)
}
}()
return res.Emit(r)
}

View File

@ -1,16 +1,28 @@
package dagcmd
import (
"bytes"
"fmt"
"math"
blocks "github.com/ipfs/go-block-format"
"github.com/ipfs/go-cid"
"github.com/ipfs/go-ipfs/core/commands/cmdenv"
"github.com/ipfs/go-ipfs/core/coredag"
ipldlegacy "github.com/ipfs/go-ipld-legacy"
"github.com/ipld/go-ipld-prime/multicodec"
basicnode "github.com/ipld/go-ipld-prime/node/basic"
cmds "github.com/ipfs/go-ipfs-cmds"
files "github.com/ipfs/go-ipfs-files"
ipld "github.com/ipfs/go-ipld-format"
mh "github.com/multiformats/go-multihash"
mc "github.com/multiformats/go-multicodec"
// Expected minimal set of available format/ienc codecs.
_ "github.com/ipld/go-codec-dagpb"
_ "github.com/ipld/go-ipld-prime/codec/cbor"
_ "github.com/ipld/go-ipld-prime/codec/dagcbor"
_ "github.com/ipld/go-ipld-prime/codec/dagjson"
_ "github.com/ipld/go-ipld-prime/codec/json"
_ "github.com/ipld/go-ipld-prime/codec/raw"
)
func dagPut(req *cmds.Request, res cmds.ResponseEmitter, env cmds.Environment) error {
@ -24,16 +36,33 @@ func dagPut(req *cmds.Request, res cmds.ResponseEmitter, env cmds.Environment) e
hash, _ := req.Options["hash"].(string)
dopin, _ := req.Options["pin"].(bool)
// mhType tells inputParser which hash should be used. MaxUint64 means 'use
// default hash' (sha256 for cbor, sha1 for git..)
mhType := uint64(math.MaxUint64)
var icodec mc.Code
if err := icodec.Set(ienc); err != nil {
return err
}
var fcodec mc.Code
if err := fcodec.Set(format); err != nil {
return err
}
var mhType mc.Code
if err := mhType.Set(hash); err != nil {
return err
}
if hash != "" {
var ok bool
mhType, ok = mh.Names[hash]
if !ok {
return fmt.Errorf("%s in not a valid multihash name", hash)
}
cidPrefix := cid.Prefix{
Version: 1,
Codec: uint64(fcodec),
MhType: uint64(mhType),
MhLength: -1,
}
decoder, err := multicodec.LookupDecoder(uint64(icodec))
if err != nil {
return err
}
encoder, err := multicodec.LookupEncoder(uint64(fcodec))
if err != nil {
return err
}
var adder ipld.NodeAdder = api.Dag()
@ -48,22 +77,36 @@ func dagPut(req *cmds.Request, res cmds.ResponseEmitter, env cmds.Environment) e
if file == nil {
return fmt.Errorf("expected a regular file")
}
nds, err := coredag.ParseInputs(ienc, format, file, mhType, -1)
node := basicnode.Prototype.Any.NewBuilder()
if err := decoder(node, file); err != nil {
return err
}
n := node.Build()
bd := bytes.NewBuffer([]byte{})
if err := encoder(n, bd); err != nil {
return err
}
blockCid, err := cidPrefix.Sum(bd.Bytes())
if err != nil {
return err
}
if len(nds) == 0 {
return fmt.Errorf("no node returned from ParseInputs")
blk, err := blocks.NewBlockWithCid(bd.Bytes(), blockCid)
if err != nil {
return err
}
ln := ipldlegacy.LegacyNode{
Block: blk,
Node: n,
}
for _, nd := range nds {
err := b.Add(req.Context, nd)
if err != nil {
return err
}
if err := b.Add(req.Context, &ln); err != nil {
return err
}
cid := nds[0].Cid()
cid := ln.Cid()
if err := res.Emit(&OutputObject{Cid: cid}); err != nil {
return err
}

View File

@ -17,6 +17,7 @@ import (
"github.com/ipfs/go-ipfs-pinner"
bserv "github.com/ipfs/go-blockservice"
"github.com/ipfs/go-fetcher"
"github.com/ipfs/go-graphsync"
bstore "github.com/ipfs/go-ipfs-blockstore"
exchange "github.com/ipfs/go-ipfs-exchange-interface"
@ -24,7 +25,6 @@ import (
ipld "github.com/ipfs/go-ipld-format"
logging "github.com/ipfs/go-log"
mfs "github.com/ipfs/go-mfs"
resolver "github.com/ipfs/go-path/resolver"
goprocess "github.com/jbenet/goprocess"
connmgr "github.com/libp2p/go-libp2p-core/connmgr"
ic "github.com/libp2p/go-libp2p-core/crypto"
@ -70,18 +70,19 @@ type IpfsNode struct {
PNetFingerprint libp2p.PNetFingerprint `optional:"true"` // fingerprint of private network
// Services
Peerstore pstore.Peerstore `optional:"true"` // storage for other Peer instances
Blockstore bstore.GCBlockstore // the block store (lower level)
Filestore *filestore.Filestore `optional:"true"` // the filestore blockstore
BaseBlocks node.BaseBlocks // the raw blockstore, no filestore wrapping
GCLocker bstore.GCLocker // the locker used to protect the blockstore during gc
Blocks bserv.BlockService // the block service, get/add blocks.
DAG ipld.DAGService // the merkle dag service, get/add objects.
Resolver *resolver.Resolver // the path resolution system
Reporter *metrics.BandwidthCounter `optional:"true"`
Discovery discovery.Service `optional:"true"`
FilesRoot *mfs.Root
RecordValidator record.Validator
Peerstore pstore.Peerstore `optional:"true"` // storage for other Peer instances
Blockstore bstore.GCBlockstore // the block store (lower level)
Filestore *filestore.Filestore `optional:"true"` // the filestore blockstore
BaseBlocks node.BaseBlocks // the raw blockstore, no filestore wrapping
GCLocker bstore.GCLocker // the locker used to protect the blockstore during gc
Blocks bserv.BlockService // the block service, get/add blocks.
DAG ipld.DAGService // the merkle dag service, get/add objects.
IPLDFetcherFactory fetcher.Factory `name:"ipldFetcher"` // fetcher that paths over the IPLD data model
UnixFSFetcherFactory fetcher.Factory `name:"unixfsFetcher"` // fetcher that interprets UnixFS data
Reporter *metrics.BandwidthCounter `optional:"true"`
Discovery discovery.Service `optional:"true"`
FilesRoot *mfs.Root
RecordValidator record.Validator
// Online
PeerHost p2phost.Host `optional:"true"` // the network host (server+client)

View File

@ -19,11 +19,12 @@ import (
"fmt"
bserv "github.com/ipfs/go-blockservice"
"github.com/ipfs/go-ipfs-blockstore"
"github.com/ipfs/go-ipfs-exchange-interface"
"github.com/ipfs/go-fetcher"
blockstore "github.com/ipfs/go-ipfs-blockstore"
exchange "github.com/ipfs/go-ipfs-exchange-interface"
offlinexch "github.com/ipfs/go-ipfs-exchange-offline"
"github.com/ipfs/go-ipfs-pinner"
"github.com/ipfs/go-ipfs-provider"
pin "github.com/ipfs/go-ipfs-pinner"
provider "github.com/ipfs/go-ipfs-provider"
offlineroute "github.com/ipfs/go-ipfs-routing/offline"
ipld "github.com/ipfs/go-ipld-format"
dag "github.com/ipfs/go-merkledag"
@ -55,13 +56,14 @@ type CoreAPI struct {
baseBlocks blockstore.Blockstore
pinning pin.Pinner
blocks bserv.BlockService
dag ipld.DAGService
peerstore pstore.Peerstore
peerHost p2phost.Host
recordValidator record.Validator
exchange exchange.Interface
blocks bserv.BlockService
dag ipld.DAGService
ipldFetcherFactory fetcher.Factory
unixFSFetcherFactory fetcher.Factory
peerstore pstore.Peerstore
peerHost p2phost.Host
recordValidator record.Validator
exchange exchange.Interface
namesys namesys.NameSystem
routing routing.Routing
@ -167,8 +169,10 @@ func (api *CoreAPI) WithOptions(opts ...options.ApiOption) (coreiface.CoreAPI, e
baseBlocks: n.BaseBlocks,
pinning: n.Pinning,
blocks: n.Blocks,
dag: n.DAG,
blocks: n.Blocks,
dag: n.DAG,
ipldFetcherFactory: n.IPLDFetcherFactory,
unixFSFetcherFactory: n.UnixFSFetcherFactory,
peerstore: n.Peerstore,
peerHost: n.PeerHost,

View File

@ -8,10 +8,10 @@ import (
"github.com/ipfs/go-namesys/resolve"
"github.com/ipfs/go-cid"
"github.com/ipfs/go-fetcher"
ipld "github.com/ipfs/go-ipld-format"
ipfspath "github.com/ipfs/go-path"
"github.com/ipfs/go-path/resolver"
uio "github.com/ipfs/go-unixfs/io"
ipfspathresolver "github.com/ipfs/go-path/resolver"
coreiface "github.com/ipfs/interface-go-ipfs-core"
path "github.com/ipfs/interface-go-ipfs-core/path"
)
@ -49,23 +49,19 @@ func (api *CoreAPI) ResolvePath(ctx context.Context, p path.Path) (path.Resolved
return nil, err
}
var resolveOnce resolver.ResolveOnce
switch ipath.Segments()[0] {
case "ipfs":
resolveOnce = uio.ResolveUnixfsOnce
case "ipld":
resolveOnce = resolver.ResolveSingle
default:
if ipath.Segments()[0] != "ipfs" && ipath.Segments()[0] != "ipld" {
return nil, fmt.Errorf("unsupported path namespace: %s", p.Namespace())
}
r := &resolver.Resolver{
DAG: api.dag,
ResolveOnce: resolveOnce,
var dataFetcher fetcher.Factory
if ipath.Segments()[0] == "ipld" {
dataFetcher = api.ipldFetcherFactory
} else {
dataFetcher = api.unixFSFetcherFactory
}
resolver := ipfspathresolver.NewBasicResolver(dataFetcher)
node, rest, err := r.ResolveToLastNode(ctx, ipath)
node, rest, err := resolver.ResolveToLastNode(ctx, ipath)
if err != nil {
return nil, err
}

View File

@ -3,12 +3,13 @@ package node
import (
"context"
"fmt"
"github.com/ipfs/go-bitswap"
"github.com/ipfs/go-bitswap/network"
"github.com/ipfs/go-blockservice"
"github.com/ipfs/go-cid"
"github.com/ipfs/go-datastore"
"github.com/ipfs/go-fetcher"
bsfetcher "github.com/ipfs/go-fetcher/impl/blockservice"
"github.com/ipfs/go-filestore"
blockstore "github.com/ipfs/go-ipfs-blockstore"
exchange "github.com/ipfs/go-ipfs-exchange-interface"
@ -18,6 +19,11 @@ import (
"github.com/ipfs/go-merkledag"
"github.com/ipfs/go-mfs"
"github.com/ipfs/go-unixfs"
"github.com/ipfs/go-unixfsnode"
dagpb "github.com/ipld/go-codec-dagpb"
"github.com/ipld/go-ipld-prime"
basicnode "github.com/ipld/go-ipld-prime/node/basic"
"github.com/ipld/go-ipld-prime/schema"
"github.com/libp2p/go-libp2p-core/host"
"github.com/libp2p/go-libp2p-core/routing"
"go.uber.org/fx"
@ -80,6 +86,26 @@ func (s *syncDagService) Session(ctx context.Context) format.NodeGetter {
return merkledag.NewSession(ctx, s.DAGService)
}
type fetchersOut struct {
fx.Out
IPLDFetcher fetcher.Factory `name:"ipldFetcher"`
UnixfsFetcher fetcher.Factory `name:"unixfsFetcher"`
}
// FetcherConfig returns a fetcher config that can build new fetcher instances
func FetcherConfig(bs blockservice.BlockService) fetchersOut {
ipldFetcher := bsfetcher.NewFetcherConfig(bs)
ipldFetcher.PrototypeChooser = dagpb.AddSupportToChooser(func(lnk ipld.Link, lnkCtx ipld.LinkContext) (ipld.NodePrototype, error) {
if tlnkNd, ok := lnkCtx.LinkNode.(schema.TypedLinkNode); ok {
return tlnkNd.LinkTargetNodePrototype(), nil
}
return basicnode.Prototype.Any, nil
})
unixFSFetcher := ipldFetcher.WithReifier(unixfsnode.Reify)
return fetchersOut{IPLDFetcher: ipldFetcher, UnixfsFetcher: unixFSFetcher}
}
// Dag creates new DAGService
func Dag(bs blockservice.BlockService) format.DAGService {
return merkledag.NewDAGService(bs)

View File

@ -18,7 +18,6 @@ import (
offline "github.com/ipfs/go-ipfs-exchange-offline"
offroute "github.com/ipfs/go-ipfs-routing/offline"
"github.com/ipfs/go-path/resolver"
uio "github.com/ipfs/go-unixfs/io"
"go.uber.org/fx"
)
@ -294,7 +293,7 @@ func Offline(cfg *config.Config) fx.Option {
var Core = fx.Options(
fx.Provide(BlockService),
fx.Provide(Dag),
fx.Provide(resolver.NewBasicResolver),
fx.Provide(FetcherConfig),
fx.Provide(Pinning),
fx.Provide(Files),
)

View File

@ -5,12 +5,12 @@ import (
"fmt"
"time"
"github.com/ipfs/go-fetcher"
"github.com/ipfs/go-ipfs-pinner"
"github.com/ipfs/go-ipfs-provider"
"github.com/ipfs/go-ipfs-provider/batched"
q "github.com/ipfs/go-ipfs-provider/queue"
"github.com/ipfs/go-ipfs-provider/simple"
ipld "github.com/ipfs/go-ipld-format"
"github.com/libp2p/go-libp2p-core/routing"
"github.com/multiformats/go-multihash"
"go.uber.org/fx"
@ -172,7 +172,12 @@ func SimpleProviders(reprovideStrategy string, reprovideInterval string) fx.Opti
}
func pinnedProviderStrategy(onlyRoots bool) interface{} {
return func(pinner pin.Pinner, dag ipld.DAGService) simple.KeyChanFunc {
return simple.NewPinnedProvider(onlyRoots, pinner, dag)
type input struct {
fx.In
Pinner pin.Pinner
IPLDFetcher fetcher.Factory `name:"ipldFetcher"`
}
return func(in input) simple.KeyChanFunc {
return simple.NewPinnedProvider(onlyRoots, in.Pinner, in.IPLDFetcher)
}
}

View File

@ -10,16 +10,18 @@ import (
"os"
"syscall"
core "github.com/ipfs/go-ipfs/core"
mdag "github.com/ipfs/go-merkledag"
path "github.com/ipfs/go-path"
ft "github.com/ipfs/go-unixfs"
uio "github.com/ipfs/go-unixfs/io"
fuse "bazil.org/fuse"
fs "bazil.org/fuse/fs"
"github.com/ipfs/go-cid"
core "github.com/ipfs/go-ipfs/core"
ipld "github.com/ipfs/go-ipld-format"
logging "github.com/ipfs/go-log"
mdag "github.com/ipfs/go-merkledag"
path "github.com/ipfs/go-path"
"github.com/ipfs/go-path/resolver"
ft "github.com/ipfs/go-unixfs"
uio "github.com/ipfs/go-unixfs/io"
cidlink "github.com/ipld/go-ipld-prime/linking/cid"
)
var log = logging.Logger("fuse/ipfs")
@ -65,20 +67,41 @@ func (s *Root) Lookup(ctx context.Context, name string) (fs.Node, error) {
return nil, fuse.ENOENT
}
nd, err := s.Ipfs.Resolver.ResolvePath(ctx, p)
nd, ndLnk, err := resolver.NewBasicResolver(s.Ipfs.UnixFSFetcherFactory).ResolvePath(ctx, p)
if err != nil {
// todo: make this error more versatile.
return nil, fuse.ENOENT
}
switch nd := nd.(type) {
case *mdag.ProtoNode, *mdag.RawNode:
return &Node{Ipfs: s.Ipfs, Nd: nd}, nil
default:
log.Error("fuse node was not a protobuf node")
return nil, fuse.ENOTSUP
cidLnk, ok := ndLnk.(cidlink.Link)
if !ok {
log.Debugf("non-cidlink returned from ResolvePath: %v", ndLnk)
return nil, fuse.ENOENT
}
// convert ipld-prime node to universal node
blk, err := s.Ipfs.Blockstore.Get(cidLnk.Cid)
if err != nil {
log.Debugf("fuse failed to retrieve block: %v: %s", cidLnk, err)
return nil, fuse.ENOENT
}
var fnd ipld.Node
switch cidLnk.Cid.Prefix().Codec {
case cid.DagProtobuf:
fnd, err = mdag.ProtoNodeConverter(blk, nd)
case cid.Raw:
fnd, err = mdag.RawNodeConverter(blk, nd)
default:
log.Error("fuse node was not a supported type")
return nil, fuse.ENOTSUP
}
if err != nil {
log.Error("could not convert protobuf or raw node")
return nil, fuse.ENOENT
}
return &Node{Ipfs: s.Ipfs, Nd: fnd}, nil
}
// ReadDirAll reads a particular directory. Disallowed for root.

22
go.mod
View File

@ -14,7 +14,7 @@ require (
github.com/hashicorp/go-multierror v1.1.1
github.com/ipfs/go-bitswap v0.3.4
github.com/ipfs/go-block-format v0.0.3
github.com/ipfs/go-blockservice v0.1.4
github.com/ipfs/go-blockservice v0.1.7
github.com/ipfs/go-cid v0.0.7
github.com/ipfs/go-cidutil v0.0.2
github.com/ipfs/go-datastore v0.4.5
@ -23,6 +23,7 @@ require (
github.com/ipfs/go-ds-flatfs v0.4.5
github.com/ipfs/go-ds-leveldb v0.4.2
github.com/ipfs/go-ds-measure v0.1.0
github.com/ipfs/go-fetcher v1.5.0
github.com/ipfs/go-filestore v0.0.3
github.com/ipfs/go-fs-lock v0.0.7
github.com/ipfs/go-graphsync v0.8.0
@ -36,26 +37,30 @@ require (
github.com/ipfs/go-ipfs-keystore v0.0.2
github.com/ipfs/go-ipfs-pinner v0.1.2
github.com/ipfs/go-ipfs-posinfo v0.0.1
github.com/ipfs/go-ipfs-provider v0.5.1
github.com/ipfs/go-ipfs-provider v0.6.1
github.com/ipfs/go-ipfs-routing v0.1.0
github.com/ipfs/go-ipfs-util v0.0.2
github.com/ipfs/go-ipld-cbor v0.0.5
github.com/ipfs/go-ipld-format v0.2.0
github.com/ipfs/go-ipld-git v0.0.4
github.com/ipfs/go-ipld-git v0.1.0
github.com/ipfs/go-ipld-legacy v0.1.0
github.com/ipfs/go-ipns v0.1.0
github.com/ipfs/go-log v1.0.5
github.com/ipfs/go-merkledag v0.3.2
github.com/ipfs/go-merkledag v0.4.0
github.com/ipfs/go-metrics-interface v0.0.1
github.com/ipfs/go-metrics-prometheus v0.0.2
github.com/ipfs/go-mfs v0.1.2
github.com/ipfs/go-namesys v0.3.0
github.com/ipfs/go-path v0.0.9
github.com/ipfs/go-path v0.1.1
github.com/ipfs/go-pinning-service-http-client v0.1.0
github.com/ipfs/go-unixfs v0.2.5
github.com/ipfs/go-unixfsnode v1.1.2
github.com/ipfs/go-verifcid v0.0.1
github.com/ipfs/interface-go-ipfs-core v0.4.0
github.com/ipfs/interface-go-ipfs-core v0.5.1
github.com/ipfs/tar-utils v0.0.1
github.com/ipld/go-car v0.3.1
github.com/ipld/go-codec-dagpb v1.3.0
github.com/ipld/go-ipld-prime v0.11.0
github.com/jbenet/go-random v0.0.0-20190219211222-123a90aedc0c
github.com/jbenet/go-temp-err-catcher v0.1.0
github.com/jbenet/goprocess v0.1.4
@ -90,6 +95,7 @@ require (
github.com/multiformats/go-multiaddr v0.3.3
github.com/multiformats/go-multiaddr-dns v0.3.1
github.com/multiformats/go-multibase v0.0.3
github.com/multiformats/go-multicodec v0.3.0
github.com/multiformats/go-multihash v0.0.15
github.com/opentracing/opentracing-go v1.2.0
github.com/pkg/errors v0.9.1
@ -101,10 +107,10 @@ require (
go.opencensus.io v0.23.0
go.uber.org/fx v1.13.1
go.uber.org/zap v1.19.0
golang.org/x/crypto v0.0.0-20210506145944-38f3c27a63bf
golang.org/x/crypto v0.0.0-20210513164829-c07d793c2f9a
golang.org/x/lint v0.0.0-20201208152925-83fdc39ff7b5 // indirect
golang.org/x/sync v0.0.0-20210220032951-036812b2e83c
golang.org/x/sys v0.0.0-20210511113859-b0526f3d8744
golang.org/x/sys v0.0.0-20210514084401-e8d321eab015
)
go 1.15

61
go.sum
View File

@ -356,10 +356,10 @@ github.com/ipfs/go-block-format v0.0.3/go.mod h1:4LmD4ZUw0mhO+JSKdpWwrzATiEfM7WW
github.com/ipfs/go-blockservice v0.0.7/go.mod h1:EOfb9k/Y878ZTRY/CH0x5+ATtaipfbRhbvNSdgc/7So=
github.com/ipfs/go-blockservice v0.1.0/go.mod h1:hzmMScl1kXHg3M2BjTymbVPjv627N7sYcvYaKbop39M=
github.com/ipfs/go-blockservice v0.1.1/go.mod h1:t+411r7psEUhLueM8C7aPA7cxCclv4O3VsUVxt9kz2I=
github.com/ipfs/go-blockservice v0.1.2/go.mod h1:t+411r7psEUhLueM8C7aPA7cxCclv4O3VsUVxt9kz2I=
github.com/ipfs/go-blockservice v0.1.3/go.mod h1:OTZhFpkgY48kNzbgyvcexW9cHrpjBYIjSR0KoDOFOLU=
github.com/ipfs/go-blockservice v0.1.4 h1:Vq+MlsH8000KbbUciRyYMEw/NNP8UAGmcqKi4uWmFGA=
github.com/ipfs/go-blockservice v0.1.4/go.mod h1:OTZhFpkgY48kNzbgyvcexW9cHrpjBYIjSR0KoDOFOLU=
github.com/ipfs/go-blockservice v0.1.7 h1:yVe9te0M7ow8i+PPkx03YFSpxqzXx594d6h+34D6qMg=
github.com/ipfs/go-blockservice v0.1.7/go.mod h1:GmS+BAt4hrwBKkzE11AFDQUrnvqjwFatGS2MY7wOjEM=
github.com/ipfs/go-cid v0.0.1/go.mod h1:GHWU/WuQdMPmIosc4Yn1bcCT7dSeX4lBafM7iqUPQvM=
github.com/ipfs/go-cid v0.0.2/go.mod h1:GHWU/WuQdMPmIosc4Yn1bcCT7dSeX4lBafM7iqUPQvM=
github.com/ipfs/go-cid v0.0.3/go.mod h1:GHWU/WuQdMPmIosc4Yn1bcCT7dSeX4lBafM7iqUPQvM=
@ -400,6 +400,8 @@ github.com/ipfs/go-ds-leveldb v0.4.2 h1:QmQoAJ9WkPMUfBLnu1sBVy0xWWlJPg0m4kRAiJL9
github.com/ipfs/go-ds-leveldb v0.4.2/go.mod h1:jpbku/YqBSsBc1qgME8BkWS4AxzF2cEu1Ii2r79Hh9s=
github.com/ipfs/go-ds-measure v0.1.0 h1:vE4TyY4aeLeVgnnPBC5QzKIjKrqzha0NCujTfgvVbVQ=
github.com/ipfs/go-ds-measure v0.1.0/go.mod h1:1nDiFrhLlwArTME1Ees2XaBOl49OoCgd2A3f8EchMSY=
github.com/ipfs/go-fetcher v1.5.0 h1:oreKTKBzja3S09rSmoZlA3KGVlRiUbJ1pQjtB4K6y3w=
github.com/ipfs/go-fetcher v1.5.0/go.mod h1:5pDZ0393oRF/fHiLmtFZtpMNBQfHOYNPtryWedVuSWE=
github.com/ipfs/go-filestore v0.0.3 h1:MhZ1jT5K3NewZwim6rS/akcJLm1xM+r6nz6foeB9EwE=
github.com/ipfs/go-filestore v0.0.3/go.mod h1:dvXRykFzyyXN2CdNlRGzDAkXMDPyI+D7JE066SiKLSE=
github.com/ipfs/go-fs-lock v0.0.7 h1:6BR3dajORFrFTkb5EpCUFIAypsoxpGpDSVUdFwzgL9U=
@ -442,8 +444,8 @@ github.com/ipfs/go-ipfs-posinfo v0.0.1/go.mod h1:SwyeVP+jCwiDu0C313l/8jg6ZxM0qqt
github.com/ipfs/go-ipfs-pq v0.0.1/go.mod h1:LWIqQpqfRG3fNc5XsnIhz/wQ2XXGyugQwls7BgUmUfY=
github.com/ipfs/go-ipfs-pq v0.0.2 h1:e1vOOW6MuOwG2lqxcLA+wEn93i/9laCY8sXAw76jFOY=
github.com/ipfs/go-ipfs-pq v0.0.2/go.mod h1:LWIqQpqfRG3fNc5XsnIhz/wQ2XXGyugQwls7BgUmUfY=
github.com/ipfs/go-ipfs-provider v0.5.1 h1:kZj72jzWLtGcorlwnMvBL6y6KJk6klO2Kb8QSeqEB0o=
github.com/ipfs/go-ipfs-provider v0.5.1/go.mod h1:fem6HKSru7n35Ljap6kowWdJrUzvcWJW01uhAkqNnzo=
github.com/ipfs/go-ipfs-provider v0.6.1 h1:4VenAH1J5XH+/mgM2Y7p+QN3wlk7CInMDG8rsT2CGW4=
github.com/ipfs/go-ipfs-provider v0.6.1/go.mod h1:I4Cig3InhftbRJohph76Qy/P2uKEZILNGiKvDJmmC28=
github.com/ipfs/go-ipfs-routing v0.0.1/go.mod h1:k76lf20iKFxQTjcJokbPM9iBXVXVZhcOwc360N4nuKs=
github.com/ipfs/go-ipfs-routing v0.1.0 h1:gAJTT1cEeeLj6/DlLX6t+NxD9fQe2ymTO6qWRDI/HQQ=
github.com/ipfs/go-ipfs-routing v0.1.0/go.mod h1:hYoUkJLyAUKhF58tysKpids8RNDPO42BVMgK5dNsoqY=
@ -459,8 +461,10 @@ github.com/ipfs/go-ipld-format v0.0.1/go.mod h1:kyJtbkDALmFHv3QR6et67i35QzO3S0dC
github.com/ipfs/go-ipld-format v0.0.2/go.mod h1:4B6+FM2u9OJ9zCV+kSbgFAZlOrv1Hqbf0INGQgiKf9k=
github.com/ipfs/go-ipld-format v0.2.0 h1:xGlJKkArkmBvowr+GMCX0FEZtkro71K1AwiKnL37mwA=
github.com/ipfs/go-ipld-format v0.2.0/go.mod h1:3l3C1uKoadTPbeNfrDi+xMInYKlx2Cvg1BuydPSdzQs=
github.com/ipfs/go-ipld-git v0.0.4 h1:fQv2Alq72g6mH+heDWQ9Awu5FQYc3hcCUVtzuWj/Mno=
github.com/ipfs/go-ipld-git v0.0.4/go.mod h1:RuvMXa9qtJpDbqngyICCU/d+cmLFXxLsbIclmD0Lcr0=
github.com/ipfs/go-ipld-git v0.1.0 h1:+HUPKbvOPicyF5J4gdRS3nvQRBgvYVGyvuAigiISkzc=
github.com/ipfs/go-ipld-git v0.1.0/go.mod h1:pfIxVWClpLmyDHDWI7qTva4/cdOQyVzC+0jzl5/n1Sk=
github.com/ipfs/go-ipld-legacy v0.1.0 h1:wxkkc4k8cnvIGIjPO0waJCe7SHEyFgl+yQdafdjGrpA=
github.com/ipfs/go-ipld-legacy v0.1.0/go.mod h1:86f5P/srAmh9GcIcWQR9lfFLZPrIyyXQeVlOWeeWEuI=
github.com/ipfs/go-ipns v0.0.2/go.mod h1:WChil4e0/m9cIINWLxZe1Jtf77oz5L05rO2ei/uKJ5U=
github.com/ipfs/go-ipns v0.1.0 h1:jk03sneWwh+/bSHrFjRfE38xHDnTzqCsx2wsJ8ipukM=
github.com/ipfs/go-ipns v0.1.0/go.mod h1:3IbsuPkR6eAGcnx+E7j6HpOSbSQJPZ6zlRj+NK3jPxQ=
@ -484,8 +488,9 @@ github.com/ipfs/go-merkledag v0.1.0/go.mod h1:SQiXrtSts3KGNmgOzMICy5c0POOpUNQLvB
github.com/ipfs/go-merkledag v0.2.3/go.mod h1:SQiXrtSts3KGNmgOzMICy5c0POOpUNQLvB3ClKnBAlk=
github.com/ipfs/go-merkledag v0.3.0/go.mod h1:4pymaZLhSLNVuiCITYrpViD6vmfZ/Ws4n/L9tfNv3S4=
github.com/ipfs/go-merkledag v0.3.1/go.mod h1:fvkZNNZixVW6cKSZ/JfLlON5OlgTXNdRLz0p6QG/I2M=
github.com/ipfs/go-merkledag v0.3.2 h1:MRqj40QkrWkvPswXs4EfSslhZ4RVPRbxwX11js0t1xY=
github.com/ipfs/go-merkledag v0.3.2/go.mod h1:fvkZNNZixVW6cKSZ/JfLlON5OlgTXNdRLz0p6QG/I2M=
github.com/ipfs/go-merkledag v0.4.0 h1:ixNu/5MJSaT/Qs073T0/HsWKwnOoBgqSq1g+GaJIen0=
github.com/ipfs/go-merkledag v0.4.0/go.mod h1:XshXBkhyeS63YNGisLL1uDSfuTyrQIxVUOg3ojR5MOE=
github.com/ipfs/go-metrics-interface v0.0.1 h1:j+cpbjYvu4R8zbleSs36gvB7jR+wsL2fGD6n0jO4kdg=
github.com/ipfs/go-metrics-interface v0.0.1/go.mod h1:6s6euYU4zowdslK0GKHmqaIZ3j/b/tL7HTWtJ4VPgWY=
github.com/ipfs/go-metrics-prometheus v0.0.2 h1:9i2iljLg12S78OhC6UAiXi176xvQGiZaGVF1CUVdE+s=
@ -495,8 +500,9 @@ github.com/ipfs/go-mfs v0.1.2/go.mod h1:T1QBiZPEpkPLzDqEJLNnbK55BVKVlNi2a+gVm4di
github.com/ipfs/go-namesys v0.3.0 h1:6lytKWj1rG0Ot6J0nTHvFw+06q1a6n7DLA2CbSGmZco=
github.com/ipfs/go-namesys v0.3.0/go.mod h1:/BL4xk8LP5Lq82AmaRKyxZv/eYRlumNiU9SZUe1Hlps=
github.com/ipfs/go-path v0.0.7/go.mod h1:6KTKmeRnBXgqrTvzFrPV3CamxcgvXX/4z79tfAd2Sno=
github.com/ipfs/go-path v0.0.9 h1:BIi831cNED8YnIlIKo9y1SI3u+E+FwQQD+rIIw8PwFA=
github.com/ipfs/go-path v0.0.9/go.mod h1:VpDkSBKQ9EFQOUgi54Tq/O/tGi8n1RfYNks13M3DEs8=
github.com/ipfs/go-path v0.1.1 h1:0rfiI0IoNTYUyQN0ifz2zQBR6mZhOKv7qW5Jjx/4fG8=
github.com/ipfs/go-path v0.1.1/go.mod h1:vC8q4AKOtrjJz2NnllIrmr2ZbGlF5fW2OKKyhV9ggb0=
github.com/ipfs/go-peertaskqueue v0.0.4/go.mod h1:03H8fhyeMfKNFWqzYEVyMbcPUeYrqP1MX6Kd+aN+rMQ=
github.com/ipfs/go-peertaskqueue v0.1.0/go.mod h1:Jmk3IyCcfl1W3jTW3YpghSwSEC6IJ3Vzz/jUmWw8Z0U=
github.com/ipfs/go-peertaskqueue v0.1.1/go.mod h1:Jmk3IyCcfl1W3jTW3YpghSwSEC6IJ3Vzz/jUmWw8Z0U=
@ -508,19 +514,24 @@ github.com/ipfs/go-unixfs v0.1.0/go.mod h1:lysk5ELhOso8+Fed9U1QTGey2ocsfaZ18h0NC
github.com/ipfs/go-unixfs v0.2.4/go.mod h1:SUdisfUjNoSDzzhGVxvCL9QO/nKdwXdr+gbMUdqcbYw=
github.com/ipfs/go-unixfs v0.2.5 h1:irj/WzIcgTBay48mSMUYDbKlIzIocXWcuUUsi5qOMOE=
github.com/ipfs/go-unixfs v0.2.5/go.mod h1:SUdisfUjNoSDzzhGVxvCL9QO/nKdwXdr+gbMUdqcbYw=
github.com/ipfs/go-unixfsnode v1.1.2 h1:aTsCdhwU0F4dMShMwYGroAj4v4EzSONLdoENebvTRb0=
github.com/ipfs/go-unixfsnode v1.1.2/go.mod h1:5dcE2x03pyjHk4JjamXmunTMzz+VUtqvPwZjIEkfV6s=
github.com/ipfs/go-verifcid v0.0.1 h1:m2HI7zIuR5TFyQ1b79Da5N9dnnCP1vcu2QqawmWlK2E=
github.com/ipfs/go-verifcid v0.0.1/go.mod h1:5Hrva5KBeIog4A+UpqlaIU+DEstipcJYQQZc0g37pY0=
github.com/ipfs/interface-go-ipfs-core v0.4.0 h1:+mUiamyHIwedqP8ZgbCIwpy40oX7QcXUbo4CZOeJVJg=
github.com/ipfs/interface-go-ipfs-core v0.4.0/go.mod h1:UJBcU6iNennuI05amq3FQ7g0JHUkibHFAfhfUIy927o=
github.com/ipfs/interface-go-ipfs-core v0.5.1 h1:1KMM7RkjUD8W5fSoRsa9xR6ZMzeL8fLHOUM1UEW9Y4M=
github.com/ipfs/interface-go-ipfs-core v0.5.1/go.mod h1:lNBJrdXHtWS46evMPBdWtDQMDsrKcGbxCOGoKLkztOE=
github.com/ipfs/tar-utils v0.0.1 h1:8Na0KBD6GddGyXwU4rXNtVTE24iuZws8mENJQPLG7W4=
github.com/ipfs/tar-utils v0.0.1/go.mod h1:ACflm9wXvV9w0eMJt6yYXxS2zuIV+yXGNwbuq1bhLeE=
github.com/ipld/go-car v0.3.1 h1:WT+3cdmXlvmWOlGxk9webhj4auGO5QvgqC2vCCkFRXs=
github.com/ipld/go-car v0.3.1/go.mod h1:dPkEWeAK8KaVvH5TahaCs6Mncpd4lDMpkbs0/SPzuVs=
github.com/ipld/go-codec-dagpb v1.2.0 h1:2umV7ud8HBMkRuJgd8gXw95cLhwmcYrihS3cQEy9zpI=
github.com/ipld/go-codec-dagpb v1.2.0/go.mod h1:6nBN7X7h8EOsEejZGqC7tej5drsdBAXbMHyBT+Fne5s=
github.com/ipld/go-codec-dagpb v1.3.0 h1:czTcaoAuNNyIYWs6Qe01DJ+sEX7B+1Z0LcXjSatMGe8=
github.com/ipld/go-codec-dagpb v1.3.0/go.mod h1:ga4JTU3abYApDC3pZ00BC2RSvC3qfBb9MSJkMLSwnhA=
github.com/ipld/go-ipld-prime v0.9.0/go.mod h1:KvBLMr4PX1gWptgkzRjVZCrLmSGcZCb/jioOQwCqZN8=
github.com/ipld/go-ipld-prime v0.9.1-0.20210324083106-dc342a9917db h1:kFwGn8rXa/Z31ev1OFNQsYeNKNCdifnTPl/NvPy5L38=
github.com/ipld/go-ipld-prime v0.9.1-0.20210324083106-dc342a9917db/go.mod h1:KvBLMr4PX1gWptgkzRjVZCrLmSGcZCb/jioOQwCqZN8=
github.com/ipld/go-ipld-prime v0.11.0 h1:jD/b/22R7CSL+F9xNffcexs+wO0Ji/TfwXO/TWck+70=
github.com/ipld/go-ipld-prime v0.11.0/go.mod h1:+WIAkokurHmZ/KwzDOMUuoeJgaRQktHtEaLglS3ZeV8=
github.com/jackpal/gateway v1.0.5/go.mod h1:lTpwd4ACLXmpyiCTRtfiNyVnUmqT9RivzCDQetPfnjA=
github.com/jackpal/go-nat-pmp v1.0.1/go.mod h1:QPH045xvCAeXUZOxsnwmrtiCoxIr9eob+4orBN1SBKc=
github.com/jackpal/go-nat-pmp v1.0.2 h1:KzKSgb7qkJvOUTqYl9/Hg/me3pWgBmERKrTGD7BdWus=
@ -565,8 +576,9 @@ github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+o
github.com/kkdai/bstream v0.0.0-20161212061736-f391b8402d23/go.mod h1:J+Gs4SYgM6CZQHDETBtE9HaSEkGmuNXF86RwHhHUvq4=
github.com/klauspost/compress v1.11.7 h1:0hzRabrMN4tSTvMfnL3SCv1ZGeAP23ynzodBgaHeMeg=
github.com/klauspost/compress v1.11.7/go.mod h1:aoV0uJVorq1K+umq18yTdKaF57EivdYsUV+/s2qKfXs=
github.com/klauspost/cpuid/v2 v2.0.4 h1:g0I61F2K2DjRHz1cnxlkNSBIaePVoJIjjnHui8QHbiw=
github.com/klauspost/cpuid/v2 v2.0.4/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
github.com/klauspost/cpuid/v2 v2.0.6 h1:dQ5ueTiftKxp0gyjKSx5+8BtPWkyQbd95m8Gys/RarI=
github.com/klauspost/cpuid/v2 v2.0.6/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
github.com/konsorten/go-windows-terminal-sequences v1.0.3 h1:CE8S1cTafDpPvMhIxNJKvHsGVBgn1xWYf1NbHQhywc8=
github.com/konsorten/go-windows-terminal-sequences v1.0.3/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
@ -595,8 +607,6 @@ github.com/libp2p/go-conn-security-multistream v0.0.2/go.mod h1:nc9vud7inQ+d6SO0
github.com/libp2p/go-conn-security-multistream v0.1.0/go.mod h1:aw6eD7LOsHEX7+2hJkDxw1MteijaVcI+/eP2/x3J1xc=
github.com/libp2p/go-conn-security-multistream v0.2.0/go.mod h1:hZN4MjlNetKD3Rq5Jb/P5ohUnFLNzEAR4DLSzpn2QLU=
github.com/libp2p/go-conn-security-multistream v0.2.1 h1:ft6/POSK7F+vl/2qzegnHDaXFU0iWB4yVTYrioC6Zy0=
github.com/libp2p/go-conn-security-multistream v0.2.1 h1:ft6/POSK7F+vl/2qzegnHDaXFU0iWB4yVTYrioC6Zy0=
github.com/libp2p/go-conn-security-multistream v0.2.1/go.mod h1:cR1d8gA0Hr59Fj6NhaTpFhJZrjSYuNmhpT2r25zYR70=
github.com/libp2p/go-conn-security-multistream v0.2.1/go.mod h1:cR1d8gA0Hr59Fj6NhaTpFhJZrjSYuNmhpT2r25zYR70=
github.com/libp2p/go-doh-resolver v0.3.1 h1:1wbVGkB4Tdj4WEvjAuYknOPyt4vSSDn9thnj13pKPaY=
github.com/libp2p/go-doh-resolver v0.3.1/go.mod h1:y5go1ZppAq9N2eppbX0xON01CyPBeUg2yS6BTssssog=
@ -616,6 +626,7 @@ github.com/libp2p/go-libp2p v0.7.4/go.mod h1:oXsBlTLF1q7pxr+9w6lqzS1ILpyHsaBPniV
github.com/libp2p/go-libp2p v0.8.1/go.mod h1:QRNH9pwdbEBpx5DTJYg+qxcVaDMAz3Ee/qDKwXujH5o=
github.com/libp2p/go-libp2p v0.12.0/go.mod h1:FpHZrfC1q7nA8jitvdjKBDF31hguaC676g/nT9PgQM0=
github.com/libp2p/go-libp2p v0.13.0/go.mod h1:pM0beYdACRfHO1WcJlp65WXyG2A6NqYM+t2DTVAJxMo=
github.com/libp2p/go-libp2p v0.14.0/go.mod h1:dsQrWLAoIn+GkHPN/U+yypizkHiB9tnv79Os+kSgQ4Q=
github.com/libp2p/go-libp2p v0.14.4 h1:QCJE+jGyqxWdrSPuS4jByXCzosgaIg4SJTLCRplJ53w=
github.com/libp2p/go-libp2p v0.14.4/go.mod h1:EIRU0Of4J5S8rkockZM7eJp2S0UrCyi55m2kJVru3rM=
github.com/libp2p/go-libp2p-asn-util v0.0.0-20200825225859-85005c6cf052 h1:BM7aaOF7RpmNn9+9g6uTjGJ0cTzWr5j9i9IKeun2M8U=
@ -805,6 +816,7 @@ github.com/libp2p/go-libp2p-yamux v0.2.8/go.mod h1:/t6tDqeuZf0INZMTgd0WxIRbtK2Ez
github.com/libp2p/go-libp2p-yamux v0.4.0/go.mod h1:+DWDjtFMzoAwYLVkNZftoucn7PelNoy5nm3tZ3/Zw30=
github.com/libp2p/go-libp2p-yamux v0.5.0/go.mod h1:AyR8k5EzyM2QN9Bbdg6X1SkVVuqLwTGf0L4DFq9g6po=
github.com/libp2p/go-libp2p-yamux v0.5.1/go.mod h1:dowuvDu8CRWmr0iqySMiSxK+W0iL5cMVO9S94Y6gkv4=
github.com/libp2p/go-libp2p-yamux v0.5.3/go.mod h1:Vy3TMonBAfTMXHWopsMc8iX/XGRYrRlpUaMzaeuHV/s=
github.com/libp2p/go-libp2p-yamux v0.5.4 h1:/UOPtT/6DHPtr3TtKXBHa6g0Le0szYuI33Xc/Xpd7fQ=
github.com/libp2p/go-libp2p-yamux v0.5.4/go.mod h1:tfrXbyaTqqSU654GTvK3ocnSZL3BuHoeTSqhcel1wsE=
github.com/libp2p/go-maddr-filter v0.0.1/go.mod h1:6eT12kSQMA9x2pvFQa+xesMKUBlj9VImZbj3B9FBH/Q=
@ -886,6 +898,7 @@ github.com/libp2p/go-yamux v1.4.0/go.mod h1:fr7aVgmdNGJK+N1g+b6DW6VxzbRCjCOejR/h
github.com/libp2p/go-yamux v1.4.1 h1:P1Fe9vF4th5JOxxgQvfbOHkrGqIZniTLf+ddhZp8YTI=
github.com/libp2p/go-yamux v1.4.1/go.mod h1:fr7aVgmdNGJK+N1g+b6DW6VxzbRCjCOejR/hkmpooHE=
github.com/libp2p/go-yamux/v2 v2.0.0/go.mod h1:NVWira5+sVUIU6tu1JWvaRn1dRnG+cawOJiflsAM+7U=
github.com/libp2p/go-yamux/v2 v2.1.1/go.mod h1:3So6P6TV6r75R9jiBpiIKgU/66lOarCZjqROGxzPpPQ=
github.com/libp2p/go-yamux/v2 v2.2.0 h1:RwtpYZ2/wVviZ5+3pjC8qdQ4TKnrak0/E01N1UWoAFU=
github.com/libp2p/go-yamux/v2 v2.2.0/go.mod h1:3So6P6TV6r75R9jiBpiIKgU/66lOarCZjqROGxzPpPQ=
github.com/lightstep/lightstep-tracer-common/golang/gogo v0.0.0-20190605223551-bc2310a04743/go.mod h1:qklhhLq1aX+mtWk9cPHPzaBjWImj5ULL6C7HFJtXQMM=
@ -991,8 +1004,6 @@ github.com/multiformats/go-multiaddr-dns v0.0.2/go.mod h1:9kWcqw/Pj6FwxAwW38n/94
github.com/multiformats/go-multiaddr-dns v0.2.0/go.mod h1:TJ5pr5bBO7Y1B18djPuRsVkduhQH2YqYSbxWJzYGdK0=
github.com/multiformats/go-multiaddr-dns v0.3.0/go.mod h1:mNzQ4eTGDg0ll1N9jKPOUogZPoJ30W8a7zk66FQPpdQ=
github.com/multiformats/go-multiaddr-dns v0.3.1 h1:QgQgR+LQVt3NPTjbrLLpsaT2ufAA2y0Mkk+QRVJbW3A=
github.com/multiformats/go-multiaddr-dns v0.3.1 h1:QgQgR+LQVt3NPTjbrLLpsaT2ufAA2y0Mkk+QRVJbW3A=
github.com/multiformats/go-multiaddr-dns v0.3.1/go.mod h1:G/245BRQ6FJGmryJCrOuTdB37AMA5AMOVuO6NY3JwTk=
github.com/multiformats/go-multiaddr-dns v0.3.1/go.mod h1:G/245BRQ6FJGmryJCrOuTdB37AMA5AMOVuO6NY3JwTk=
github.com/multiformats/go-multiaddr-fmt v0.0.1/go.mod h1:aBYjqL4T/7j4Qx+R73XSv/8JsgnRFlf0w2KGLCmXl3Q=
github.com/multiformats/go-multiaddr-fmt v0.1.0 h1:WLEFClPycPkp4fnIzoFoV9FVd49/eQsuaL3/CWe167E=
@ -1009,8 +1020,9 @@ github.com/multiformats/go-multiaddr-net v0.2.0/go.mod h1:gGdH3UXny6U3cKKYCvpXI5
github.com/multiformats/go-multibase v0.0.1/go.mod h1:bja2MqRZ3ggyXtZSEDKpl0uO/gviWFaSteVbWT51qgs=
github.com/multiformats/go-multibase v0.0.3 h1:l/B6bJDQjvQ5G52jw4QGSYeOTZoAwIO77RblWplfIqk=
github.com/multiformats/go-multibase v0.0.3/go.mod h1:5+1R4eQrT3PkYZ24C3W2Ue2tPwIdYQD509ZjSb5y9Oc=
github.com/multiformats/go-multicodec v0.2.0 h1:MUzKZWxOFagwLLtlx96pub9zwDQAbMAf1k9fXOdc3so=
github.com/multiformats/go-multicodec v0.2.0/go.mod h1:/y4YVwkfMyry5kFbMTbLJKErhycTIftytRV+llXdyS4=
github.com/multiformats/go-multicodec v0.3.0 h1:tstDwfIjiHbnIjeM5Lp+pMrSeN+LCMsEwOrkPmWm03A=
github.com/multiformats/go-multicodec v0.3.0/go.mod h1:qGGaQmioCDh+TeFOnxrbU0DaIPw8yFgAZgFG0V7p1qQ=
github.com/multiformats/go-multihash v0.0.1/go.mod h1:w/5tugSrLEbWqlcgJabL3oHFKTwfvkofsjW2Qa1ct4U=
github.com/multiformats/go-multihash v0.0.5/go.mod h1:lt/HCbqlQwlPBz7lv0sQCdtfcMtlJvakRUn/0Ual8po=
github.com/multiformats/go-multihash v0.0.8/go.mod h1:YSLudS+Pi8NHE7o6tb3D8vrpKa63epEDmG8nTduyAew=
@ -1026,7 +1038,6 @@ github.com/multiformats/go-multistream v0.1.0/go.mod h1:fJTiDfXJVmItycydCnNx4+wS
github.com/multiformats/go-multistream v0.1.1/go.mod h1:KmHZ40hzVxiaiwlj3MEbYgK9JFk2/9UktWZAF54Du38=
github.com/multiformats/go-multistream v0.2.0/go.mod h1:5GZPQZbkWOLOn3J2y4Y99vVW7vOfsAflxARk3x14o6k=
github.com/multiformats/go-multistream v0.2.1/go.mod h1:5GZPQZbkWOLOn3J2y4Y99vVW7vOfsAflxARk3x14o6k=
github.com/multiformats/go-multistream v0.2.1/go.mod h1:5GZPQZbkWOLOn3J2y4Y99vVW7vOfsAflxARk3x14o6k=
github.com/multiformats/go-multistream v0.2.2 h1:TCYu1BHTDr1F/Qm75qwYISQdzGcRdC21nFgQW7l7GBo=
github.com/multiformats/go-multistream v0.2.2/go.mod h1:UIcnm7Zuo8HKG+HkWgfQsGL+/MIEhyTqbODbIUwSXKs=
github.com/multiformats/go-varint v0.0.1/go.mod h1:3Ls8CIEsrijN6+B7PbrXRPxHRPuXSrVKRY101jdMZYE=
@ -1299,8 +1310,9 @@ go.uber.org/multierr v1.1.0/go.mod h1:wR5kodmAFQ0UK8QlbwjlSNy0Z68gJhDJUG5sjR94q/
go.uber.org/multierr v1.3.0/go.mod h1:VgVr7evmIr6uPjLBxg28wmKNXyqE9akIJ5XnfpiKl+4=
go.uber.org/multierr v1.4.0/go.mod h1:VgVr7evmIr6uPjLBxg28wmKNXyqE9akIJ5XnfpiKl+4=
go.uber.org/multierr v1.5.0/go.mod h1:FeouvMocqHpRaaGuG9EjoKcStLC43Zu/fmqdUMPcKYU=
go.uber.org/multierr v1.6.0 h1:y6IPFStTAIT5Ytl7/XYmHvzXQ7S3g/IeZW9hyZ5thw4=
go.uber.org/multierr v1.6.0/go.mod h1:cdWPpRnG4AhwMwsgIHip0KRBQjJy5kYEpYjJxpXp9iU=
go.uber.org/multierr v1.7.0 h1:zaiO/rmgFjbmCXdSYJWQcdvOCsthmdaHfr3Gm2Kx4Ec=
go.uber.org/multierr v1.7.0/go.mod h1:7EAYxJLBy9rStEaz58O2t4Uvip6FSURkq8/ppBp95ak=
go.uber.org/tools v0.0.0-20190618225709-2cfd321de3ee/go.mod h1:vJERXedbb3MVM5f9Ejo0C68/HhF8uaILCdgjnY+goOA=
go.uber.org/zap v1.10.0/go.mod h1:vwi/ZaCAaUcBkycHslxD9B2zi4UTXhF60s6SWpuDF0Q=
go.uber.org/zap v1.13.0/go.mod h1:zwrFLgMcdUuIBviXEYEH1YKNaOBnKXsx2IPda5bBwHM=
@ -1340,10 +1352,10 @@ golang.org/x/crypto v0.0.0-20200602180216-279210d13fed/go.mod h1:LzIPMQfyMNhhGPh
golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.0.0-20200820211705-5c72a883971a/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.0.0-20210220033148-5ea612d1eb83/go.mod h1:jdWPYTVW3xRLrWPugEBEK3UY2ZEsg3UU495nc5E+M+I=
golang.org/x/crypto v0.0.0-20210220033148-5ea612d1eb83/go.mod h1:jdWPYTVW3xRLrWPugEBEK3UY2ZEsg3UU495nc5E+M+I=
golang.org/x/crypto v0.0.0-20210322153248-0c34fe9e7dc2/go.mod h1:T9bdIzuCu7OtxOm1hfPfRQxPLYneinmdGuTeoZ9dtd4=
golang.org/x/crypto v0.0.0-20210506145944-38f3c27a63bf h1:B2n+Zi5QeYRDAEodEu72OS36gmTWjgpXr2+cWcBW90o=
golang.org/x/crypto v0.0.0-20210506145944-38f3c27a63bf/go.mod h1:P+XmwS30IXTQdn5tA2iutPOUgjI07+tq3H3K9MVA1s8=
golang.org/x/crypto v0.0.0-20210513164829-c07d793c2f9a h1:kr2P4QFmQr29mSLA43kwrOcgcReGTfbE9N577tCTuBc=
golang.org/x/crypto v0.0.0-20210513164829-c07d793c2f9a/go.mod h1:P+XmwS30IXTQdn5tA2iutPOUgjI07+tq3H3K9MVA1s8=
golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20190510132918-efd6b22b2522/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8=
@ -1499,7 +1511,6 @@ golang.org/x/sys v0.0.0-20201214210602-f9fddec55a1e/go.mod h1:h1NjWce9XRLGQEsW7w
golang.org/x/sys v0.0.0-20210112080510-489259a85091/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210124154548-22da62e12c0c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210303074136-134d130e1a04/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210303074136-134d130e1a04/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210309074719-68d13333faf2/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210315160823-c6e025ad8005/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210317225723-c4fcb01b228e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
@ -1507,8 +1518,9 @@ golang.org/x/sys v0.0.0-20210330210617-4fbd30eecc44/go.mod h1:h1NjWce9XRLGQEsW7w
golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210426080607-c94f62235c83/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210510120138-977fb7262007/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210511113859-b0526f3d8744 h1:yhBbb4IRs2HS9PPlAg6DMC6mUOKexJBNsLf4Z+6En1Q=
golang.org/x/sys v0.0.0-20210511113859-b0526f3d8744/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210514084401-e8d321eab015 h1:hZR0X1kPW+nwyJ9xRxqZk1vx5RUObAPBdKVvXPDUH/E=
golang.org/x/sys v0.0.0-20210514084401-e8d321eab015/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/term v0.0.0-20201117132131-f5c789dd3221/go.mod h1:Nr5EML6q2oocZ2LXRh80K7BxOlk5/8JxuGnuhpl+muw=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1 h1:v+OssWQX+hTHEmOBgwxdZxK4zHq3yOs8F9J7mk0PY8E=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
@ -1641,8 +1653,9 @@ google.golang.org/protobuf v1.23.1-0.20200526195155-81db48ad09cc/go.mod h1:EGpAD
google.golang.org/protobuf v1.24.0/go.mod h1:r/3tXBNzIEhYS9I1OUVjXDlt8tc493IdKGjtUeSXeh4=
google.golang.org/protobuf v1.25.0/go.mod h1:9JNX74DMeImyA3h4bdi1ymwjUzf21/xIlbajtzgsN7c=
google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw=
google.golang.org/protobuf v1.26.0 h1:bxAC2xTBsZGibn2RTntX0oH50xLsqy1OxA9tTL3p/lk=
google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc=
google.golang.org/protobuf v1.27.1 h1:SnqbnDw1V7RiZcXPx5MEeqPv2s79L9i7BJUlG/+RurQ=
google.golang.org/protobuf v1.27.1/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc=
gopkg.in/alecthomas/kingpin.v2 v2.2.6 h1:jMFz6MfLP0/4fUyZle81rXUoxOBFi19VUFKVDOQfozc=
gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=

View File

@ -1,16 +1,13 @@
package plugin
import (
"github.com/ipfs/go-ipfs/core/coredag"
ipld "github.com/ipfs/go-ipld-format"
multicodec "github.com/ipld/go-ipld-prime/multicodec"
)
// PluginIPLD is an interface that can be implemented to add handlers for
// for different IPLD formats
// for different IPLD codecs
type PluginIPLD interface {
Plugin
RegisterBlockDecoders(dec ipld.BlockDecoder) error
RegisterInputEncParsers(iec coredag.InputEncParsers) error
Register(multicodec.Registry) error
}

View File

@ -10,14 +10,13 @@ import (
config "github.com/ipfs/go-ipfs-config"
cserialize "github.com/ipfs/go-ipfs-config/serialize"
"github.com/ipld/go-ipld-prime/multicodec"
"github.com/ipfs/go-ipfs/core"
"github.com/ipfs/go-ipfs/core/coreapi"
coredag "github.com/ipfs/go-ipfs/core/coredag"
plugin "github.com/ipfs/go-ipfs/plugin"
fsrepo "github.com/ipfs/go-ipfs/repo/fsrepo"
ipld "github.com/ipfs/go-ipld-format"
logging "github.com/ipfs/go-log"
opentracing "github.com/opentracing/opentracing-go"
)
@ -335,11 +334,7 @@ func injectDatastorePlugin(pl plugin.PluginDatastore) error {
}
func injectIPLDPlugin(pl plugin.PluginIPLD) error {
err := pl.RegisterBlockDecoders(ipld.DefaultBlockDecoder)
if err != nil {
return err
}
return pl.RegisterInputEncParsers(coredag.DefaultInputEncParsers)
return pl.Register(multicodec.DefaultRegistry)
}
func injectTracerPlugin(pl plugin.PluginTracer) error {

View File

@ -2,17 +2,15 @@ package git
import (
"compress/zlib"
"fmt"
"io"
"math"
"github.com/ipfs/go-ipfs/core/coredag"
"github.com/ipfs/go-ipfs/plugin"
"github.com/ipfs/go-cid"
"github.com/ipfs/go-ipld-format"
// Note that depending on this package registers it's multicodec encoder and decoder.
git "github.com/ipfs/go-ipld-git"
mh "github.com/multiformats/go-multihash"
"github.com/ipld/go-ipld-prime"
"github.com/ipld/go-ipld-prime/multicodec"
mc "github.com/multiformats/go-multicodec"
)
// Plugins is exported list of plugins that will be loaded
@ -36,40 +34,21 @@ func (*gitPlugin) Init(_ *plugin.Environment) error {
return nil
}
func (*gitPlugin) RegisterBlockDecoders(dec format.BlockDecoder) error {
dec.Register(cid.GitRaw, git.DecodeBlock)
func (*gitPlugin) Register(reg multicodec.Registry) error {
// register a custom identifier in the reserved range for import of "zlib-encoded git objects."
reg.RegisterDecoder(uint64(mc.ReservedStart+mc.GitRaw), decodeZlibGit)
reg.RegisterEncoder(uint64(mc.GitRaw), git.Encode)
reg.RegisterDecoder(uint64(mc.GitRaw), git.Decode)
return nil
}
func (*gitPlugin) RegisterInputEncParsers(iec coredag.InputEncParsers) error {
iec.AddParser("raw", "git", parseRawGit)
iec.AddParser("zlib", "git", parseZlibGit)
return nil
}
func parseRawGit(r io.Reader, mhType uint64, mhLen int) ([]format.Node, error) {
if mhType != math.MaxUint64 && mhType != mh.SHA1 {
return nil, fmt.Errorf("unsupported mhType %d", mhType)
}
if mhLen != -1 && mhLen != mh.DefaultLengths[mh.SHA1] {
return nil, fmt.Errorf("invalid mhLen %d", mhLen)
}
nd, err := git.ParseObject(r)
if err != nil {
return nil, err
}
return []format.Node{nd}, nil
}
func parseZlibGit(r io.Reader, mhType uint64, mhLen int) ([]format.Node, error) {
func decodeZlibGit(na ipld.NodeAssembler, r io.Reader) error {
rc, err := zlib.NewReader(r)
if err != nil {
return nil, err
return err
}
defer rc.Close()
return parseRawGit(rc, mhType, mhLen)
return git.Decode(na, rc)
}

View File

@ -21,45 +21,158 @@ test_expect_success "make a few test files" '
HASH4=$(ipfs add --pin=false -q file4)
'
test_expect_success "make an ipld object in json" '
test_expect_success "make an ipld object in dag-json" '
printf "{\"hello\":\"world\",\"cats\":[{\"/\":\"%s\"},{\"water\":{\"/\":\"%s\"}}],\"magic\":{\"/\":\"%s\"},\"sub\":{\"dict\":\"ionary\",\"beep\":[0,\"bop\"]}}" $HASH1 $HASH2 $HASH3 > ipld_object
'
# This data is in https://github.com/ipld/codec-fixtures/tree/master/fixtures/dagpb_Data_some
test_expect_success "make the same ipld object in dag-cbor, dag-json and dag-pb" '
echo "omREYXRhRQABAgMEZUxpbmtzgA==" | base64 -d > ipld_object_dagcbor
echo "CgUAAQIDBA==" | base64 -d > ipld_object_dagpb
echo "{\"Data\":{\"/\":{\"bytes\":\"AAECAwQ\"}},\"Links\":[]}" > ipld_object_dagjson
'
test_dag_cmd() {
test_expect_success "can add an ipld object using protobuf" '
IPLDHASH=$(cat ipld_object | ipfs dag put -f protobuf)
'
# Working with a plain IPLD hello-world object that's dag-json and dag-cbor compatible
test_expect_success "output looks correct" '
EXPHASH="QmdfTbBqBPQ7VNxZEYEj14VmRuZBkqFbiwReogJgS1zR1n"
test $EXPHASH = $IPLDHASH
'
test_expect_success "can add an ipld object using protobuf and --cid=base=base32" '
IPLDHASHb32=$(cat ipld_object | ipfs dag put -f protobuf --cid-base=base32)
'
test_expect_success "output looks correct (does not upgrade to CIDv1)" '
test $EXPHASH = $IPLDHASHb32
'
test_expect_success "can add an ipld object" '
test_expect_success "can add an ipld object using defaults (dag-json to dag-cbor)" '
IPLDHASH=$(cat ipld_object | ipfs dag put)
'
test_expect_success "output looks correct" '
EXPHASH="bafyreidjtjfmavdk7epvztob2m5vlm3pxp3gjmpyewro4qlbw5n4f4iz64"
test_expect_success "CID looks correct" '
EXPHASH="bafyreiblwimnjbqcdoeafiobk6q27jcw64ew7n2fmmhdpldd63edmjecde"
test $EXPHASH = $IPLDHASH
'
test_expect_success "can add an ipld object using --cid-base=base32" '
IPLDHASHb32=$(cat ipld_object | ipfs dag put --cid-base=base32)
test_expect_success "can add an ipld object using dag-json to dag-json" '
IPLDHASH=$(cat ipld_object | ipfs dag put --input-enc=dag-json -f dag-json)
'
test_expect_success "output looks correct" '
test $(ipfs cid base32 $EXPHASH) = $IPLDHASHb32
test_expect_success "CID looks correct" '
EXPHASH="baguqeera6gviseelmbzn2ugoddo5vulxlshqs3kw5ymgsb6w4cabnoh4ldpa"
test $EXPHASH = $IPLDHASH
'
test_expect_success "can add an ipld object using dag-json to dag-cbor" '
IPLDHASH=$(cat ipld_object | ipfs dag put --input-enc=dag-json -f dag-cbor)
'
test_expect_success "CID looks correct" '
EXPHASH="bafyreiblwimnjbqcdoeafiobk6q27jcw64ew7n2fmmhdpldd63edmjecde"
test $EXPHASH = $IPLDHASH
'
test_expect_success "can add an ipld object using cid-base=base58btc" '
IPLDb58HASH=$(cat ipld_object | ipfs dag put -cid-base=base58btc)
'
test_expect_success "CID looks correct" '
EXPHASH="zdpuAoN1XJ3GsrxEzMuCbRKZzRUVJekJUCbPVgCgE4D9yYqVi"
test $EXPHASH = $IPLDb58HASH
'
# Same object, different forms
# (1) dag-cbor input
test_expect_success "can add a dag-cbor input block stored as dag-cbor" '
IPLDCBORHASH=$(cat ipld_object_dagcbor | ipfs dag put --input-enc=dag-cbor -f dag-cbor)
'
test_expect_success "dag-cbor CID looks correct" '
EXPHASH="bafyreieculsmrexh3ty5jentbvuku452o27mst4h2tq2rb2zntqhgcstji"
test $EXPHASH = $IPLDCBORHASH
'
test_expect_success "can add a dag-cbor input block stored as dag-pb" '
IPLDPBHASH=$(cat ipld_object_dagcbor | ipfs dag put --input-enc=dag-cbor -f dag-pb)
'
test_expect_success "dag-pb CID looks correct" '
EXPHASH="bafybeibazl2z4vqp2tmwcfag6wirmtpnomxknqcgrauj7m2yisrz3qjbom"
test $EXPHASH = $IPLDPBHASH
'
test_expect_success "can add a dag-cbor input block stored as dag-json" '
IPLDJSONHASH=$(cat ipld_object_dagcbor | ipfs dag put --input-enc=dag-cbor -f dag-json)
'
test_expect_success "dag-json CID looks correct" '
EXPHASH="baguqeerajwksxu3lxpomdwxvosl542zl3xknhjgxtq3277gafrhl6vdw5tcq"
test $EXPHASH = $IPLDJSONHASH
'
# (2) dag-json input
test_expect_success "can add a dag-json input block stored as dag-cbor" '
IPLDCBORHASH=$(cat ipld_object_dagjson | ipfs dag put --input-enc=dag-json -f dag-cbor)
'
test_expect_success "dag-cbor CID looks correct" '
EXPHASH="bafyreieculsmrexh3ty5jentbvuku452o27mst4h2tq2rb2zntqhgcstji"
test $EXPHASH = $IPLDCBORHASH
'
test_expect_success "can add a dag-json input block stored as dag-pb" '
IPLDPBHASH=$(cat ipld_object_dagjson | ipfs dag put --input-enc=dag-json -f dag-pb)
'
test_expect_success "dag-pb CID looks correct" '
EXPHASH="bafybeibazl2z4vqp2tmwcfag6wirmtpnomxknqcgrauj7m2yisrz3qjbom"
test $EXPHASH = $IPLDPBHASH
'
test_expect_success "can add a dag-json input block stored as dag-json" '
IPLDJSONHASH=$(cat ipld_object_dagjson | ipfs dag put --input-enc=dag-json -f dag-json)
'
test_expect_success "dag-json CID looks correct" '
EXPHASH="baguqeerajwksxu3lxpomdwxvosl542zl3xknhjgxtq3277gafrhl6vdw5tcq"
test $EXPHASH = $IPLDJSONHASH
'
# (3) dag-pb input
test_expect_success "can add a dag-pb input block stored as dag-cbor" '
IPLDCBORHASH=$(cat ipld_object_dagpb | ipfs dag put --input-enc=dag-pb -f dag-cbor)
'
test_expect_success "dag-cbor CID looks correct" '
EXPHASH="bafyreieculsmrexh3ty5jentbvuku452o27mst4h2tq2rb2zntqhgcstji"
test $EXPHASH = $IPLDCBORHASH
'
test_expect_success "can add a dag-pb input block stored as dag-pb" '
IPLDPBHASH=$(cat ipld_object_dagpb | ipfs dag put --input-enc=dag-pb -f dag-pb)
'
test_expect_success "dag-pb CID looks correct" '
EXPHASH="bafybeibazl2z4vqp2tmwcfag6wirmtpnomxknqcgrauj7m2yisrz3qjbom"
test $EXPHASH = $IPLDPBHASH
'
test_expect_success "can add a dag-pb input block stored as dag-json" '
IPLDJSONHASH=$(cat ipld_object_dagpb | ipfs dag put --input-enc=dag-pb -f dag-json)
'
test_expect_success "dag-json CID looks correct" '
EXPHASH="baguqeerajwksxu3lxpomdwxvosl542zl3xknhjgxtq3277gafrhl6vdw5tcq"
test $EXPHASH = $IPLDJSONHASH
'
test_expect_success "can get dag-cbor, dag-json, dag-pb blocks as dag-json" '
ipfs dag get $IPLDCBORHASH >& dag-get-cbor &&
ipfs dag get $IPLDJSONHASH >& dag-get-json &&
ipfs dag get $IPLDPBHASH >& dag-get-pb
'
# this doesn't tell us if they are correct, we test that better below
test_expect_success "outputs are the same" '
test_cmp dag-get-cbor dag-get-json &&
test_cmp dag-get-cbor dag-get-pb
'
# Traversals using the original hello-world object
test_expect_success "various path traversals work" '
ipfs cat $IPLDHASH/cats/0 > out1 &&
ipfs cat $IPLDHASH/cats/1/water > out2 &&
@ -81,25 +194,46 @@ test_dag_cmd() {
'
test_expect_success "sub-objects look right" '
echo "\"world\"" > sub1_exp &&
echo -n "\"world\"" > sub1_exp &&
test_cmp sub1_exp sub1 &&
echo "{\"beep\":[0,\"bop\"],\"dict\":\"ionary\"}" > sub2_exp &&
echo -n "{\"beep\":[0,\"bop\"],\"dict\":\"ionary\"}" > sub2_exp &&
test_cmp sub2_exp sub2 &&
echo "[0,\"bop\"]" > sub3_exp &&
echo -n "[0,\"bop\"]" > sub3_exp &&
test_cmp sub3_exp sub3 &&
echo "0" > sub4_exp &&
echo -n "0" > sub4_exp &&
test_cmp sub4_exp sub4 &&
echo "\"bop\"" > sub5_exp &&
echo -n "\"bop\"" > sub5_exp &&
test_cmp sub5_exp sub5
'
test_expect_success "can pin cbor object" '
ipfs pin add $EXPHASH
test_expect_success "traversals using /ipld/ work" '
ipfs dag get /ipld/$IPLDPBHASH/Data > ipld_path_Data_actual
'
test_expect_success "retrieved node looks right" '
echo -n "{\"/\":{\"bytes\":\"AAECAwQ\"}}" > ipld_path_Data_expected
test_cmp ipld_path_Data_actual ipld_path_Data_expected
'
test_expect_success "can pin ipld object" '
ipfs pin add $IPLDHASH
'
test_expect_success "can pin dag-pb object" '
ipfs pin add $IPLDPBHASH
'
test_expect_success "can pin dag-cbor object" '
ipfs pin add $IPLDCBORHASH
'
test_expect_success "can pin dag-json object" '
ipfs pin add $IPLDJSONHASH
'
test_expect_success "after gc, objects still accessible" '
ipfs repo gc > /dev/null &&
ipfs refs -r --timeout=2s $EXPHASH > /dev/null
ipfs refs -r --timeout=2s $IPLDJSONHASH > /dev/null
'
test_expect_success "can get object" '
@ -111,7 +245,7 @@ test_dag_cmd() {
'
test_expect_success "retrieved object hashes back correctly" '
IPLDHASH2=$(cat ipld_obj_out | ipfs dag put) &&
IPLDHASH2=$(cat ipld_obj_out | ipfs dag put --input-enc=dag-json -f dag-cbor) &&
test "$IPLDHASH" = "$IPLDHASH2"
'
@ -124,7 +258,7 @@ test_dag_cmd() {
'
test_expect_success "output looks correct" '
echo "{\"data\":\"CAISB2Zvb2JhcgoYBw==\",\"links\":[]}" > dag_get_pb_exp &&
echo -n "{\"Data\":{\"/\":{\"bytes\":\"CAISB2Zvb2JhcgoYBw\"}},\"Links\":[]}" > dag_get_pb_exp &&
test_cmp dag_get_pb_exp dag_get_pb_out
'
@ -133,24 +267,23 @@ test_dag_cmd() {
'
test_expect_success "output looks correct" '
echo "{\"data\":\"CAISBGZvbwoYBA==\",\"links\":[]}" > cat_exp &&
echo -n "{\"Data\":{\"/\":{\"bytes\":\"CAISBGZvbwoYBA\"}},\"Links\":[]}" > cat_exp &&
test_cmp cat_exp cat_out
'
test_expect_success "non-canonical cbor input is normalized" '
HASH=$(cat ../t0053-dag-data/non-canon.cbor | ipfs dag put --format=cbor --input-enc=raw) &&
test_expect_success "non-canonical dag-cbor input is normalized" '
HASH=$(cat ../t0053-dag-data/non-canon.cbor | ipfs dag put --format=dag-cbor --input-enc=dag-cbor) &&
test $HASH = "bafyreiawx7ona7oa2ptcoh6vwq4q6bmd7x2ibtkykld327bgb7t73ayrqm" ||
test_fsh echo $HASH
'
test_expect_success "non-canonical cbor input is normalized with input-enc cbor" '
HASH=$(cat ../t0053-dag-data/non-canon.cbor | ipfs dag put --format=cbor --input-enc=cbor) &&
test $HASH = "bafyreiawx7ona7oa2ptcoh6vwq4q6bmd7x2ibtkykld327bgb7t73ayrqm" ||
test_fsh echo $HASH
test_expect_success "cbor input can be fetched" '
EXPARR=$(ipfs dag get $HASH/arr)
test $EXPARR = "[]"
'
test_expect_success "add an ipld with pin" '
PINHASH=$(printf {\"foo\":\"bar\"} | ipfs dag put --pin=true)
PINHASH=$(printf {\"foo\":\"bar\"} | ipfs dag put --input-enc=dag-json --pin=true)
'
test_expect_success "after gc, objects still accessible" '
@ -158,23 +291,23 @@ test_dag_cmd() {
ipfs refs -r --timeout=2s $PINHASH > /dev/null
'
test_expect_success "can add an ipld object with sha3 hash" '
IPLDHASH=$(cat ipld_object | ipfs dag put --hash sha3)
test_expect_success "can add an ipld object with sha3-512 hash" '
IPLDHASH=$(cat ipld_object | ipfs dag put --hash sha3-512)
'
test_expect_success "output looks correct" '
EXPHASH="bafyriqgae54zjl3bjebmbat2rjem4ewj6vni6jxohandmvk3bibfgv3sioyeidppsghvulryxats43br3b7afa6jy77x6gqzqaicer6ljicck"
EXPHASH="bafyriqforjma7y7akqz7nhuu73r6liggj5zhkbfiqgicywe3fgkna2ijlhod2af3ue7doj56tlzt5hh6iu5esafc4msr3sd53jol5m2o25ucy"
test $EXPHASH = $IPLDHASH
'
test_expect_success "prepare dag-pb object" '
echo foo > test_file &&
HASH=$(ipfs add -wq test_file | tail -n1)
HASH=$(ipfs add -wq test_file | tail -n1 | ipfs cid base32)
'
test_expect_success "dag put with json dag-pb works" '
ipfs dag get $HASH > pbjson &&
cat pbjson | ipfs dag put --format=dag-pb --input-enc=json > dag_put_out
cat pbjson | ipfs dag put --format=dag-pb --input-enc=dag-json > dag_put_out
'
test_expect_success "dag put with dag-pb works output looks good" '
@ -184,7 +317,7 @@ test_dag_cmd() {
test_expect_success "dag put with raw dag-pb works" '
ipfs block get $HASH > pbraw &&
cat pbraw | ipfs dag put --format=dag-pb --input-enc=raw > dag_put_out
cat pbraw | ipfs dag put --format=dag-pb --input-enc=dag-pb > dag_put_out
'
test_expect_success "dag put with dag-pb works output looks good" '
@ -271,10 +404,10 @@ test_dag_cmd() {
test_expect_success "dag stat of simple IPLD object" '
ipfs dag stat $NESTED_HASH > actual_stat_inner_ipld_obj &&
echo "Size: 15, NumBlocks: 1" > exp_stat_inner_ipld_obj &&
echo "Size: 8, NumBlocks: 1" > exp_stat_inner_ipld_obj &&
test_cmp exp_stat_inner_ipld_obj actual_stat_inner_ipld_obj &&
ipfs dag stat $HASH > actual_stat_ipld_obj &&
echo "Size: 61, NumBlocks: 2" > exp_stat_ipld_obj &&
echo "Size: 54, NumBlocks: 2" > exp_stat_ipld_obj &&
test_cmp exp_stat_ipld_obj actual_stat_ipld_obj
'

View File

@ -8,14 +8,14 @@ test_init_ipfs
test_expect_success 'create test JSON files' '
WANT_JSON="{\"data\":1234}"
WANT_HASH="bafyreidcqah6v3py3ujdc3ris22rjlfntaw7ajrus2f2477kpnizulaoea"
WANT_HASH="bafyreidbm2zncsc3j25zn7lofgd4woeh6eygdy73thfosuni2rwr3bhcvu"
printf "${WANT_JSON}\n" > with_newline.json &&
printf "${WANT_JSON}" > without_newline.json
'
test_expect_success 'puts as CBOR work' '
GOT_HASH_WITHOUT_NEWLINE="$(cat without_newline.json | ipfs dag put -f cbor)"
GOT_HASH_WITH_NEWLINE="$(cat with_newline.json | ipfs dag put -f cbor)"
GOT_HASH_WITHOUT_NEWLINE="$(cat without_newline.json | ipfs dag put -f dag-cbor)"
GOT_HASH_WITH_NEWLINE="$(cat with_newline.json | ipfs dag put -f dag-cbor)"
'
test_expect_success 'put hashes with or without newline are equal' '
@ -27,12 +27,7 @@ test_expect_success 'hashes are of expected value' '
test "${WANT_HASH}" = "${GOT_HASH_WITHOUT_NEWLINE}"
'
# Retrieval must not contain a new-line regardless of input JSON, because
# objects are put using the stable CBOR format.
# despite this, dag retrieval returns JSON with new-line.
# Expect failure until fixed, as per:
# - https://github.com/ipfs/go-ipfs/issues/3503#issuecomment-877295280
test_expect_failure "retrieval by hash does not have new line" '
test_expect_success "retrieval by hash does not have new line" '
ipfs dag get "${WANT_HASH}" > got.json
test_cmp without_newline.json got.json
'

View File

@ -17,25 +17,40 @@ test_expect_success "prepare test data" '
test_dag_git() {
test_expect_success "add objects via dag put" '
find objects -type f -exec ipfs dag put --format=git --input-enc=zlib {} \; -exec echo \; > hashes
find objects -type f -exec ipfs dag put --format=git-raw --input-enc=0x300078 --hash=sha1 {} \; -exec echo -n \; > hashes
'
test_expect_success "successfully get added objects" '
cat hashes | xargs -I {} ipfs dag get -- {} > /dev/null
'
test_expect_success "path traversals work" '
echo \"YmxvYiA3ACcsLnB5Zgo=\" > file1 &&
ipfs dag get baf4bcfhzi72pcj5cc4ocz7igcduubuu7aa3cddi/object/parents/0/tree/dir2/hash/f3/hash > out1
test_expect_success "dag get works" '
echo -n "{\"message\":\"Some version\n\",\"object\":{\"/\":\"baf4bcfeq6c2mspupcvftgevza56h7rmozose6wi\"},\"tag\":\"v1\",\"tagger\":{\"date\":\"1497302532\",\"email\":\"johndoe@example.com\",\"name\":\"John Doe\",\"timezone\":\"+0200\"},\"type\":\"commit\"}" > tag_expected &&
ipfs dag get baf4bcfhzi72pcj5cc4ocz7igcduubuu7aa3cddi > tag_actual
'
test_expect_success "outputs look correct" '
test_cmp file1 out1
test_cmp tag_expected tag_actual
'
test_expect_success "path traversals work" '
echo -n "{\"date\":\"1497302532\",\"email\":\"johndoe@example.com\",\"name\":\"John Doe\",\"timezone\":\"+0200\"}" > author_expected &&
echo -n "{\"/\":{\"bytes\":\"YmxvYiAxMgBIZWxsbyB3b3JsZAo\"}}" > file1_expected &&
echo -n "{\"/\":{\"bytes\":\"YmxvYiA3ACcsLnB5Zgo\"}}" > file2_expected &&
ipfs dag get baf4bcfhzi72pcj5cc4ocz7igcduubuu7aa3cddi/object/author > author_actual &&
ipfs dag get baf4bcfhzi72pcj5cc4ocz7igcduubuu7aa3cddi/object/tree/file/hash > file1_actual &&
ipfs dag get baf4bcfhzi72pcj5cc4ocz7igcduubuu7aa3cddi/object/parents/0/tree/dir2/hash/f3/hash > file2_actual
'
test_expect_success "outputs look correct" '
test_cmp author_expected author_actual &&
test_cmp file1_expected file1_actual &&
test_cmp file2_expected file2_actual
'
}
# should work offline
#test_dag_git
test_dag_git
# should work online
test_launch_ipfs_daemon