coosld/kubo
1
0
Fork 0
mirror of https://github.com/ipfs/kubo.git synced 2026-02-21 10:27:46 +08:00
Code Issues Actions 3 Packages Projects Releases Wiki Activity

docs: clarify LevelDB compaction limitations and StorageMax scope

Browse Source 
set expectations for behaviors like https://github.com/ipfs/kubo/issues/11096
This commit is contained in:
Marcin Rataj 2026-02-09 21:31:42 +01:00
parent 0fed2c35c3
commit b64d3309d4
2 changed files with 31 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

8
docs/config.md
View File

@ -910,6 +910,14 @@ storage system.
A soft upper limit for the size of the ipfs repository's datastore. With `StorageGCWatermark`,
is used to calculate whether to trigger a gc run (only if `--enable-gc` flag is set).
> [!NOTE]
> This only controls when automatic GC of raw blocks is triggered. It is not a
> hard limit on total disk usage. The metadata stored alongside blocks (pins,
> MFS, provider system state, pubsub message ID tracking, and other internal
> data) is not counted against this limit. Always include extra headroom to
> account for metadata overhead. See [datastores.md](datastores.md) for details
> on how different datastore backends handle disk space reclamation.
Default: `"10GB"`
Type: `string` (size)

24
docs/datastores.md
View File

@ -34,7 +34,9 @@ The shardFunc is prefixed with `/repo/flatfs/shard/v1` then followed by a descri
NOTE: flatfs must only be used as a block store (mounted at `/blocks`) as it only partially implements the datastore interface. You can mount flatfs for /blocks only using the mount datastore (described below).
## levelds
Uses a leveldb database to store key-value pairs.
Uses a [leveldb](https://github.com/syndtr/goleveldb) database to store key-value
pairs via [go-ds-leveldb](https://github.com/ipfs/go-ds-leveldb).
```json
{
@ -44,6 +46,26 @@ Uses a leveldb database to store key-value pairs.
}
```
> [!NOTE]
> LevelDB uses a log-structured merge-tree (LSM) storage engine. When keys are
> deleted, the data is not removed immediately. Instead, a tombstone marker is
> written, and the actual data is removed later by background compaction.
>
> LevelDB's compaction decides what to compact based on file counts (L0) and
> total level size (L1+), without considering how many tombstones a file
> contains. This means that after bulk deletions (such as pin removals or the
> periodic provider keystore sync), disk space may not be reclaimed promptly.
> The `datastore/` directory can grow significantly larger than the live data it
> holds, especially on long-running nodes with many CIDs.
>
> Unlike flatfs (which deletes files immediately) or pebble (which has
> tombstone-aware compaction), LevelDB has no way to prioritize reclaiming
> space from deleted keys. Restarting the daemon may trigger some compaction,
> but this is not guaranteed.
>
> If slow compaction is a problem, consider using the `pebbleds` datastore
> instead (see below), which handles this workload more efficiently.
## pebbleds
Uses [pebble](https://github.com/cockroachdb/pebble) as a key-value store.