This implements #5133 introducing an option to limit how deep we fetch and store
the DAG associated to a recursive pin ("--max-depth"). This feature
comes motivated by the need to fetch and pin partial DAGs in order to do
DAG sharding with IPFS Cluster.
This means that, when pinning something to --max-depth, the DAG will be
fetched only to that depth and not more.
In order to get this, the PR introduces new recursive pin types: "recursive1"
means: the given CID is pinned along with its direct children (maxDepth=1)
"recursive2" means: the given CID is pinned along with its direct children
and its grandchildren.
And so on...
This required introducing "maxDepth" limits to all the functions walking down
DAGs (in merkledag, pin, core/commands, core/coreapi, exchange/reprovide modules).
maxDepth == -1 effectively acts as no-limit, and all these functions behave like
they did before.
In order to facilitate the task, a new CID Set type has been added:
thirdparty/recpinset. This set carries the MaxDepth associated to every Cid.
This allows to shortcut exploring already explored branches just like the original
cid.Set does. It also allows to store the Recursive pinset (and replaces cid.Set).
recpinset should be moved outside to a different repo eventually.
TODO: tests
TODO: refs -r with --max-depth
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
Remove ~50 lines of code, some casting, and a superfluous map (when go starts
looking like python, something's wrong).
License: MIT
Signed-off-by: Steven Allen <steven@stebalien.com>
This uses a working libp2p-kad-dht and libp2p-record libraries,
reverts the changes that were introduced to support the newer versions
License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
Currently the "path" module does two very different things:
* Defines how ipfs paths look like and provides tools to parse/split etc.
* Provides a resolver to resolve paths.
This moves the resolver stuff to `path/resolver` and leaves the
path utilities in `path`.
The result is that now the IPFS `path` package just defines what a path
looks like and becomes a module that can be exported/re-used without problems.
Currently there are circular dependency cycles (resolve_test -> merkledag/utils,
merkledag->path), which the prevent the export of merkledag itself.
License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
This extracts the blocks/blockstore package and renames the
blocks/blockstore/util package to /blocks/blockstoreutil
(because util depends on Pin and I don't plan to extract
Pin and its depedencies).
The history of blocks/blockstore has been preserved. It has
been gx'ed and imported. Imports have been rewritten accordingly
and re-ordered.
License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
This extracts the routing package to its own repository
(https://github.com/ipfs/go-ipfs-routing). History has
been preserved. The new module has been gx'ed and published.
Imports have been rewritten and re-ordered accordingly.
An internal dependency to go-ipfs/repo has been removed
by substituting it with the go-datastore.Batching interface.
License: MIT
Signed-off-by: Hector Sanjuan <hector@protocol.ai>
We'll dedup the code later, for now:
1. Don't drop errors on the floor.
2. Don't modify anything returned by `Links()` directly...
License: MIT
Signed-off-by: Steven Allen <steven@stebalien.com>
