InternalPins() is a pinset composed by:
- Recursive pins CIDs
- Direct pins CIDs
- The empty node CID
- A root CID pointing to all above (and any of the subbuckets that may have been created)
It is only set during Flush/Load operations for the pinner.
Thus recursively exploring internal pins in order to decide which CIDs are safe
from GC only re-explores the recursive DAGs and should not be necessary.
Mind that, previously, the CidSet will correctly prune any already explored branches so
it did not have pernicious effects. But now it does.
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
This implements #5133 introducing an option to limit how deep we fetch and store
the DAG associated to a recursive pin ("--max-depth"). This feature
comes motivated by the need to fetch and pin partial DAGs in order to do
DAG sharding with IPFS Cluster.
This means that, when pinning something to --max-depth, the DAG will be
fetched only to that depth and not more.
In order to get this, the PR introduces new recursive pin types: "recursive1"
means: the given CID is pinned along with its direct children (maxDepth=1)
"recursive2" means: the given CID is pinned along with its direct children
and its grandchildren.
And so on...
This required introducing "maxDepth" limits to all the functions walking down
DAGs (in merkledag, pin, core/commands, core/coreapi, exchange/reprovide modules).
maxDepth == -1 effectively acts as no-limit, and all these functions behave like
they did before.
In order to facilitate the task, a new CID Set type has been added:
thirdparty/recpinset. This set carries the MaxDepth associated to every Cid.
This allows to shortcut exploring already explored branches just like the original
cid.Set does. It also allows to store the Recursive pinset (and replaces cid.Set).
recpinset should be moved outside to a different repo eventually.
TODO: tests
TODO: refs -r with --max-depth
License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
retry publishing with a longer EOL if the first attempt fails due to a timeout.
fixes#5099
License: MIT
Signed-off-by: Steven Allen <steven@stebalien.com>
They've never really been enabled but they:
1. Are causing the tests to fail for various reasons (e.g., out of space).
2. Take time.
License: MIT
Signed-off-by: Steven Allen <steven@stebalien.com>
To avoid duplicating fields and making the code easier to follow.
Remove all of `FSNode` previous fields in favor on a single `pb.Data` structure
that is not exported. Accessor methods are added only for the necessary internal
fields. This takes up more memory, `pb.Data` is always created inside `FSNode`
and it stays there instead of just being created and destroyed during the
(un)marshal operations.
The removed fields `Data`, `blocksizes` and `Type` had a direct counterpart in
the embedded `pb.Data` structure, in contrast (only) the `subtotal` field
doesn't have one, it was used as a temporary accumulator to track the
`Filesize`, which is now being kept updated on every modification (to ensure the
entire `FSNode` is always at a valid state), so `subtotal` could just be removed
without the addition of any other field (this temporary accumulator was
obscuring how `Filesize` was computed).
To keep `Filesize` up to date a method was added (`UpdateFilesize()`) to adjust
its value in the two places where the file size could be modified, when changing
its data (in `SetData()`, accessor method added) and when adding or removing
child nodes (in `AddBlockSize()` and `RemoveBlockSize()`).
A constructor method was added (`NewFSNode()`) to initialize the required
fields, like `Type` which is explicitly set, this deprecates the previous
methodology of just calling `new(FSNode)` and relying in the default value of
`pb.Data_DataType` (`Data_Raw`) to avoid an explicit assignment. Also,
`Filesize` is initialized to avoid being left with a `nil` value before
marshaling empty nodes, which would result in a different hash from previous
versions, to be backwards compatible. Previous versions of `GetBytes()` always
set the `Filesize` value, even though it is reflected as an `optional` field in
the `.proto` file (this may be an inaccurate field rule).
Without the duplicated fields the functions `GetBytes()` and `FSNodeFromBytes()`
are now reduced to simple `Marshal()` and `Unmarshal()` operations respectively.
License: MIT
Signed-off-by: Lucas Molas <schomatis@gmail.com>
This is almost never an error, it just means we don't have any connections. We
could leave this at Warning but we'd like to be able to turn those on by
default at some point.
fixes#5029
License: MIT
Signed-off-by: Steven Allen <steven@stebalien.com>
