GH Actions recently changed their Docker build implementation and it
has a different output than previously, causing the tests that parse
its output to fail.
This switches the test to not parse Docker build output. The parsing
was used to extract the image ID while still showing logs. A better
way to show logs and still know the image ID is to tag it, which is
what this now does.
This also renames the Docker tests so that they run earlier. This
takes better advantage of the fact that the sharness tests are run in
parallel. Since the Docker test are quite long, and are at the end of
the list, the test runner is not running other tests in parallel while
the Docker tests are running.
The multinode test is effectively the same as the twonode test. There
are some problems with it too: it *looks* like it's testing the
Websocket transport with the "listentype,ws" IPTB attribute, but that
attribute doesn't actually exist in ipfs/iptb-plugins, so it does
nothing, so that test actually just runs the same test twice (Yamux
disabled). Furthermore, this is just the same test as in the mplex
twonode test. So this just removes the useless multinode test
entirely.
Also, this removes the part of the twonode test that checks the amount
of data transferred over Bitswap. This is an implementation detail of
Bitswap, it's not appropriate to test this in an end-to-end test as it
depends on algorithmic details of how Bitswap works, and has nothing
to do with transports. This is probably more appropriate as a perf or
benchmark test of Bitswap.
This also moves equivalent functionality from jbenet/go-random-files
into the testutils package. This just copies the code and modifies it
slightly for better ergonomics.
This adds the ability to enable "optimistic provide" to the default
DHT client, which enables faster provides and reprovides.
For more information about optimistic provide, see:
https://protocollabs.notion.site/Optimistic-Provide-2c79745820fa45649d48de038516b814
Note that this feature only works when using non-custom router
types. This does not include the ability to enable optimistic provide
on custom routers for now, to minimize the footprint of this
experimental feature. We intend on continuing to test this and improve
the UX, which may or may not involve adding configuration for it to
custom routers. We also plan on refactoring/redesigning custom routers
more broadly so I don't want this to add more effort for maintainers
and confusion for users.
trying two things:
- speed up arm build by allowing cross-comp from go instead of slow QEMU
- take a stab at caching buildx layers, without infinite growth
This fix was laready applied in
14cfa48bed
and reduced build time from 20m to 3m.
* chore: update version
* chore: update go-libp2p to v0.26.4
* fix: remove timeout on default DHT operations (#9783)
* fix: remove timeout on default DHT operations
This removes the timeout by default for DHT operations. In particular
this causes issues with ProvideMany requests which can take an
indeterminate amount of time, but really these should just respect
context timeouts by default. Users can still specify timeouts here if
they want, but by default they will be set to "0" which means "no
timeout".
This is unlikely to break existing users of custom routing, because
there was previously no utility in configuring a router with timeout=0
because that would cause the router to immediately fail, so it is
unlikely (and incorrect) if anybody was using timeout=0.
* fix: remove 5m timeout on ProvideManyRouter
For context see
5fda291b66
---------
Co-authored-by: Marcin Rataj <lidel@lidel.org>
* chore: bump go-blockservice to v0.5.1
* chore: update version
* chore: update changelog for v0.19
---------
Co-authored-by: Jorropo <jorropo.pgm@gmail.com>
Co-authored-by: Gus Eggert <gus@gus.dev>
Co-authored-by: Marcin Rataj <lidel@lidel.org>
In order to make it possible to easily-overwrite the path Resolvers (i.e. via
plugins), this creates resolvers as part of the Node rather than creating them
ad-hoc.
* feat: add changelog entry for router timeouts for v0.19.1
* Apply suggestions from code review
Co-authored-by: Steve Loeppky <biglep@protocol.ai>
---------
Co-authored-by: Steve Loeppky <biglep@protocol.ai>
* fix: remove timeout on default DHT operations
This removes the timeout by default for DHT operations. In particular
this causes issues with ProvideMany requests which can take an
indeterminate amount of time, but really these should just respect
context timeouts by default. Users can still specify timeouts here if
they want, but by default they will be set to "0" which means "no
timeout".
This is unlikely to break existing users of custom routing, because
there was previously no utility in configuring a router with timeout=0
because that would cause the router to immediately fail, so it is
unlikely (and incorrect) if anybody was using timeout=0.
* fix: remove 5m timeout on ProvideManyRouter
For context see
5fda291b66
---------
Co-authored-by: Marcin Rataj <lidel@lidel.org>
This is pretty common when working through PRs and ends up causing
tons of in-flight GitHub Actions workflows running because they aren't
currently canceled when a new commit is added. This will cancel
previous runs if a new commit is added on a branch (which is the
behavior we had on CircleCI).
This also means that rb-pinning-service-api is no longer required for
running remote pinning tests. This alone saves at least 3 minutes in
test runtime in CI because we don't need to checkout the repo, build
the Docker image, run it, etc.
Instead this implements a simple pinning service in Go that the test
runs in-process, with a callback that can be used to control the async
behavior of the pinning service (e.g. simulate work happening
asynchronously like transitioning from "queued" -> "pinning" ->
"pinned").
This also adds an environment variable to Kubo to control the MFS
remote pin polling interval, so that we don't have to wait 30 seconds
in the test for MFS changes to be repinned. This is purely for tests
so I don't think we should document this.
This entire test suite runs in around 2.5 sec on my laptop, compared to
the existing 3+ minutes in CI.
The test trims all whitespace bytes from the output of 'ipfs cat' but
if the random bytes end in a whitespace char then it trims that too,
resulting in random test failure.
Instead this updates the test harness to only trim a single trailing
newline char, so that it doesn't end up chomping legitimate output.
This fixes a deadlock introduced in 1457b4fd4a.
We can't use the coreapi here because it will try to take the PinLock (RLock) again, so revert this small part of 1457b4fd4a.
This used cause a deadlock when concurrently running `ipfs dag import` concurrently with the GC.
The bug is that `ipfs dag import` takes an RLock with the PinLock.
*the cars are imported, leaving a wide window of time*
Then GC Takes a Lock on that same RWMutex while taking the GC Lock (it blocks because it waits for the RLock to be released).
Then the car imports are finished and `ipfs dag import` tries to aqcuire the PinLock (doing an RLock) again in `Api().Pin`.
However at this point the RWMutex is starved, the runtime put a fence in front of RLocks if a Lock has been waiting for too lock (else you could have an endless stream of RLock / RUnlock forever delaying a Lock to ever go through).
The issue is that `ipfs dag import`'s original RLock which is blocking everyone will be released once it returns, which only happens when `Api().Pin` completes.
So we have a deadlock (ABA kind ?), because `ipfs dag import` waits on the GC Lock, which waits on `ipfs dag import`.
Calling the Pinner directly does not acquire the PinLock again, and thus does not have this issue.
- don't bypass the CoreApi
- don't use a goroutine and return channel for `importWorker`, when what's happening is really just a synchronous call
- only `PinLock()` when we are going to pin
- use `cid.Set` instead of an explicit map
- fail the request early if any pinning fail, no need to try to pin more if the request failed already
