kubo/core
Hector Sanjuan ba22102a64 provider: buffer pin providers.
Fixes #10596.

The reproviding process can take long. Currently, each CID to be provided is
obtained by making a query to the pinner and reading one by one as the CIDs
get provided.

While this query is ongoing, the pinner holds a Read mutex on the pinset.

If a pin-add-request arrives, a goroutine will start waiting for a Write mutex
on the pinset. From that point, no new Read mutexes can be taken until the writer
can proceed and finishes.

However, no one can proceed because the read mutex is still held while the
reproviding is ongoing.

The fix is mostly in Boxo, where we add a "buffered" provider which reads the
cids onto memory so that they can be provided at its own pace without making
everyone wait.

The consequence is we will need more RAM memory. Rule of thumb is 1GiB extra per 20M cids to be reprovided.
2025-03-07 08:43:18 +01:00
..
commands fix(mfs): basic UnixFS sanity checks in files cp (#10701) 2025-03-05 20:40:33 +01:00
coreapi feat(AutoTLS): enabled by default with 1h RegistrationDelay (#10724) 2025-02-25 22:25:58 +01:00
corehttp feat: ipfs-webui v4.5.0 (#10735) 2025-02-25 23:31:10 +01:00
coreiface Upgrade to Boxo v0.28.0 (#10725) 2025-02-24 16:54:14 -08:00
corerepo chore: migrate go-libipfs to boxo 2023-03-28 22:05:25 -04:00
coreunix feat: Support storing UnixFS 1.5 Mode and ModTime (#10478) 2024-08-21 02:02:46 +02:00
mock fix: deadlock on retrieving WebTransport addresses (#9857) 2023-05-08 16:11:03 +02:00
node provider: buffer pin providers. 2025-03-07 08:43:18 +01:00
.gitignore ignore testdb 2014-07-05 15:04:04 -07:00
builder.go chore: migrate bootstrap to ipfs/boxo (#10158) 2023-10-31 18:25:14 +01:00
core_test.go chore: update boxo, go-libp2p, and internalize mplex (#10095) 2023-08-24 06:11:02 +00:00
core.go Update to boxo with refactored providerQueryManager. (#10595) 2024-11-26 03:34:06 -08:00