Commit Graph

257 Commits

Author SHA1 Message Date
petricadaipegsp
b798de5871
Trigger sync on ahead peer (#366) 2024-11-20 17:12:57 -06:00
petricadaipegsp
4d3ac60e2b
Increase default sync timeout to 4 seconds (#369) 2024-11-20 16:59:24 -06:00
petricadaipegsp
bc05a4d7b9
Adaptive reserved cores (#363)
* Add adaptive data worker count

* Use runtime worker count for validation workers

* Reserve cores for networking during transition application

* Automatically set GOGC and GOMEMLIMIT
2024-11-19 16:51:14 -06:00
Cassandra Heart
91e8a16124
don't apply pruning until after seniority cutoff 2024-11-18 02:17:42 -06:00
Cassandra Heart
61179c13a0
node info should show worker counts 2024-11-18 02:03:37 -06:00
Cassandra Heart
4b69a40e56
more adjustment to validators 2024-11-17 16:22:32 -06:00
Cassandra Heart
5d1b572035
switch validators to sync 2024-11-17 16:00:51 -06:00
Cassandra Heart
063102eab2
adjust validators 2024-11-17 15:58:26 -06:00
Cassandra Heart
29a129d543
allow choice of indexed or non-indexed batches 2024-11-17 07:58:15 -06:00
Cassandra Heart
f511149b36
frame pruning 2024-11-17 04:37:55 -06:00
petricadaipegsp
0b499c649c
Move time triggered synchronization to dedicated goroutine (#359) 2024-11-17 02:18:55 -06:00
Cassandra Heart
a798347ccf
re-add #358 2024-11-16 23:20:09 -06:00
Cassandra Heart
6e2c7f4028
Revert "Use buffered channels in the engines (#357)"
This reverts commit d32d79f58d.
2024-11-16 23:11:25 -06:00
Cassandra Heart
d97a093c18
remove logging in validators due to locking, allow 2.0.3 minversion 2024-11-16 23:11:19 -06:00
Cassandra Heart
706ec2997d
adjust runonce 2024-11-16 18:29:41 -06:00
Cassandra Heart
d63f587d40
reduce bandwidth on workers 2024-11-16 17:58:57 -06:00
petricadaipegsp
d32d79f58d
Use buffered channels in the engines (#357)
* Make time data reel sending consistent

* Use buffered channels to avoid drops
2024-11-16 17:58:32 -06:00
petricadaipegsp
49566c2280
Add additional P2P configuration (#352)
* Add peer discovery configuration

* Add peer monitor configuration

* Add message validation configuration

---------

Co-authored-by: Cassandra Heart <7929478+CassOnMars@users.noreply.github.com>
2024-11-16 17:54:34 -06:00
petricadaipegsp
80c7ec2889
Add initial Prometheus support (#353)
* Add Prometheus server

* Add Prometheus gRPC metrics

* Add BlossomSub metrics

---------

Co-authored-by: Cassandra Heart <7929478+CassOnMars@users.noreply.github.com>
2024-11-16 17:53:19 -06:00
petricadaipegsp
d57d76e627
Add transaction timestamps (#354) 2024-11-16 17:52:27 -06:00
petricadaipegsp
da5b7a6126
Add sync timeout and make it async (#356)
* Add sync timeout

* Make sync async
2024-11-16 17:50:25 -06:00
Cassandra Heart
8b18568bae
the mystery of 6% 2024-11-13 14:51:41 -06:00
petricadaipegsp
37761ebd83
Log frame age (#351) 2024-11-13 11:36:49 -06:00
Cassandra Heart
07a27d7dc6
set it under either case 2024-11-13 10:43:57 -06:00
petricadaipegsp
5333b4a8cc
Sample peers which are ahead during sync (#337) 2024-11-13 10:43:28 -06:00
petricadaipegsp
f06d2c0ff2
Ensure latest frame monotonicity (#350) 2024-11-13 10:41:23 -06:00
Cassandra Heart
e58280b7e4
small adjustments 2024-11-13 10:00:37 -06:00
Cassandra Heart
22c0b55ab8
nil check 2024-11-13 06:36:08 -06:00
Cassandra Heart
b0a87b2fe4
wrapping up 2024-11-13 06:03:14 -06:00
Cassandra Heart
c2b9b1d460
resolve:
- critbit trie bug
- proof messages performing incorrect repeat
2024-11-13 00:10:22 -06:00
Cassandra Heart
71b13c5490
v2.0.3-p3 2024-11-12 03:33:20 -06:00
Cassandra Heart
d4a5e30ba2
v2.0.3-p3 2024-11-12 01:45:18 -06:00
Cassandra Heart
7a1ba529b0
Merge branch 'develop' into v2.0.3-p2 2024-11-11 15:48:50 -06:00
petricadaipegsp
db28f1b81e
Remove vendored gostream (#347)
* Remove vendored go-libp2p-gostream

* Remove error wrapping
2024-11-11 15:05:45 -06:00
petricadaipegsp
3dbe0723bd
Add message validators (#346) 2024-11-11 14:10:00 -06:00
Cassandra Heart
26cbb2092e
rate limiter logic 2024-11-11 13:30:04 -06:00
Cassandra Heart
b9694ddc99
fix rate limit 2024-11-11 11:32:11 -06:00
petricadaipegsp
de87f3f81b
Change frame delivery filter (#345) 2024-11-11 11:11:28 -06:00
petricadaipegsp
4e108bff07
Safe peer ID lookup from gostream (#344) 2024-11-11 11:10:53 -06:00
Cassandra Heart
2ce7eb26d4
change order of operations, don't commit an empty tree? 2024-11-11 04:39:52 -06:00
Cassandra Heart
1d262794cb
nil check 2024-11-11 03:50:13 -06:00
Cassandra Heart
4238b3ff5a
initial testnet v2.0.3-p2 2024-11-11 03:34:28 -06:00
Cassandra Heart
77c036b94b
v2.0.3-p0 2024-11-11 03:33:35 -06:00
Cassandra Heart
e178153593
initial testnet v2.0.3-p2 2024-11-11 03:32:14 -06:00
petricadaipegsp
05f1bf94fe
Remove additional signatures from data peer list announcements (#335)
* Do not verify data peer announcement signatures

* Do not sign data peer announcements explicitly

* Implement single peer data peer list announcement
2024-11-10 06:03:38 -06:00
Cassandra Heart
cf89b1c4ee
v2.0.3-p0 2024-11-10 01:15:15 -06:00
Cassandra Heart
042848e7cb
minor nice-to-haves 2024-11-09 16:04:12 -06:00
Cassandra Heart
1c0d707f7d
add bounds check for proofs 2024-11-09 15:54:49 -06:00
Cassandra Heart
67d454acb9
add light prover support 2024-11-09 14:46:53 -06:00
Cassandra Heart
812939a97a
v2.0.3-b7 2024-11-08 20:29:30 -06:00
Cassandra Heart
71a3e70129
log to error 2024-11-08 04:47:58 -06:00
Cassandra Heart
bc851de1bb
add testnet variant of seniority and penalties 2024-11-08 03:54:32 -06:00
Cassandra Heart
3a84cc65c8
ctrl-c = stop, damn it 2024-11-08 00:48:26 -06:00
Cassandra Heart
9a6a8a7a8e
v2.0.3-b5 2024-11-07 20:06:59 -06:00
Cassandra Heart
7ac7fc2b67
v2.0.3-b4 2024-11-07 18:03:50 -06:00
Cassandra Heart
5c924eab68
add simple reapplication logic 2024-11-07 13:14:12 -06:00
Cassandra Heart
9bc8a84f1a
fix dangling mutex lock 2024-11-07 03:05:37 -06:00
Cassandra Heart
80be91d067
v2.0.3-b3 2024-11-06 21:43:55 -06:00
Cassandra Heart
84279f2b5c
handle mints with more failure proofing 2024-11-06 20:34:08 -06:00
Cassandra Heart
5851db6433
move this back 2024-11-06 18:04:54 -06:00
Cassandra Heart
cb0918212e
make proving check the prior landed 2024-11-06 16:35:50 -06:00
Cassandra Heart
5679b19fdb
use up to date info 2024-11-06 14:46:04 -06:00
petricadaipegsp
b3cc442efd
Stop master clock spin (#334) 2024-11-06 04:04:06 -06:00
Cassandra Heart
f3e502a2d1
v2.0.3-b2 2024-11-06 03:51:30 -06:00
Cassandra Heart
e469fad46a
don't spin workers 2024-11-05 22:00:40 -06:00
Cassandra Heart
d2c5b42cdf
fix iterator bug 2024-11-05 21:24:27 -06:00
Cassandra Heart
6578040774
also 2024-11-05 17:41:42 -06:00
Cassandra Heart
3279666948
final adjustments 2024-11-05 17:37:17 -06:00
petricadaipegsp
ceb53b39c9
Use peer address while checking if date time proofs should be created (#333)
* Check PubSub key in trie, not proving key

* Use gRPC errors
2024-11-05 17:04:14 -06:00
Cassandra Heart
cdc1618964
get tries in sequence of events 2024-11-05 02:31:14 -06:00
Cassandra Heart
ee7ff329d7
fix: don't loop, reset genesis, shut off handlers on sigint 2024-11-04 23:49:15 -06:00
Cassandra Heart
8f955523d3
use correct signature path 2024-11-04 20:05:07 -06:00
petricadaipegsp
e23ad7869c
Trigger automatic peer discovery on frame stall (#328) 2024-11-04 19:25:30 -06:00
Cassandra Heart
d7501a142e
don't announce unless we're up to head 2024-11-04 18:24:12 -06:00
Cassandra Heart
a9225b2508
2.0.3-b1 2024-11-04 18:09:35 -06:00
Cassandra Heart
4b61a00095
restore prover rings 2024-10-31 23:44:23 -05:00
Cassandra Heart
3dd9a0c5f3
get develop caught up (#322)
* Update qcommander.sh bootrap (#304)

* v2.0.1 (#308)

* roll up v2.0.1-b2 to develop

* b2-fixed

* adjust return data of fast sync so it doesn't return the earliest frame

* -b3

* fix: announce peer based on leading frame, not initial frame; fix: looping bug

* fix: last batch fails due to underflow; qol: make logging chattier

* -b4

* resolve frame cache issue

* fix: mint loop + re-migrate

* fix: register execution panic

* fix: mint loop, other side

* fix: handle unexpected return of nil status

* final -b4

* handle subtle change to migration

* qol: add heuristic to handle corruption scenario

* bump genesis

* qol: use separate channel for worker

* final parameterization, parallelize streams

* deprecate signers 10, 11, 14, 17

* adjust signatory check size to match rotated out signers

* V2.0.2.3 (#321)

* roll up v2.0.1-b2 to develop

* b2-fixed

* adjust return data of fast sync so it doesn't return the earliest frame

* -b3

* fix: announce peer based on leading frame, not initial frame; fix: looping bug

* fix: last batch fails due to underflow; qol: make logging chattier

* -b4

* resolve frame cache issue

* fix: mint loop + re-migrate

* fix: register execution panic

* fix: mint loop, other side

* fix: handle unexpected return of nil status

* final -b4

* handle subtle change to migration

* qol: add heuristic to handle corruption scenario

* bump genesis

* qol: use separate channel for worker

* final parameterization, parallelize streams

* Add direct peers to blossomsub (#309)

Co-authored-by: Tyler Sturos <tyler.john@qcommander.sh>

* chore(docker): add ca-certificates to fix x509 error. (#307)

* Update qcommander.sh bootrap (#304)

* chore(docker): add ca-certificates to fix x509 error.

---------

Co-authored-by: Tyler Sturos <55340199+tjsturos@users.noreply.github.com>

* deprecate signers 10, 11, 14, 17

* adjust signatory check size to match rotated out signers

* qol: sync by rebroadcast

* upgrade version

* more small adjustments

* wait a little longer

* fix: don't use iterator for frame directly until iterator is fixed

* change iterator, genesis for testnet

* adjust to previous sync handling

* adjust: don't grab the very latest while it's already being broadcasted

* ok, ready for testnet

* handle rebroadcast quirks

* more adjustments from testing

* faster

* temporarily bulk process on frame candidates

* resolve separate frames

* don't loop

* make worker reset resume to check where it should continue

* move window

* reduce signature count now that supermajority signed last

* resolve bottlenecks

* remove GOMAXPROCS limit for now

* revisions for v2.0.2.1

* bump version

* bulk import

* reintroduce sync

* small adustments to make life better

* check bitmask for peers and keep alive

* adjust reconnect

* ensure peer doesn't fall off address list

* adjust blossomsub to background discovery

* bump version

* remove dev check

* remove debug log line

* further adjustments

* a little more logic around connection management

* v2.0.2.3

* Fix peer discovery (#319)

* Fix peer discovery

* Make peer discovery connections parallel

* Monitor peers via pings (#317)

* Support QUILIBRIUM_SIGNATURE_CHECK in client (#314)

* Ensure direct peers are not pruned by resource limits (#315)

* Support pprof profiling via HTTP (#313)

* Fix CPU profiling

* Add pprof server support

* Additional peering connection improvements (#320)

* Lookup peers if not enough external peers are available

* Make bootstrap peer discovery sensitive to a lack of bootstrappers

---------

Co-authored-by: Tyler Sturos <55340199+tjsturos@users.noreply.github.com>
Co-authored-by: Tyler Sturos <tyler.john@qcommander.sh>
Co-authored-by: linquanisaac <33619994+linquanisaac@users.noreply.github.com>
Co-authored-by: petricadaipegsp <155911522+petricadaipegsp@users.noreply.github.com>

---------

Co-authored-by: Tyler Sturos <55340199+tjsturos@users.noreply.github.com>
Co-authored-by: Tyler Sturos <tyler.john@qcommander.sh>
Co-authored-by: linquanisaac <33619994+linquanisaac@users.noreply.github.com>
Co-authored-by: petricadaipegsp <155911522+petricadaipegsp@users.noreply.github.com>
2024-10-31 16:46:58 -05:00
Cassandra Heart
c8e8d1b889
small adustments to make life better 2024-10-27 00:39:25 -05:00
Cassandra Heart
730e0b9f68
reintroduce sync 2024-10-26 23:54:23 -05:00
Cassandra Heart
5e5c421ce9
bulk import 2024-10-26 23:22:28 -05:00
Cassandra Heart
c0396f57a9
revisions for v2.0.2.1 2024-10-26 03:32:35 -05:00
Cassandra Heart
3a04a9c263
resolve bottlenecks 2024-10-25 23:56:41 -05:00
Cassandra Heart
b36bab321f
move window 2024-10-25 14:44:23 -05:00
Cassandra Heart
fe5ed13588
make worker reset resume to check where it should continue 2024-10-25 14:39:55 -05:00
Cassandra Heart
ca98e96aea
don't loop 2024-10-25 13:32:34 -05:00
Cassandra Heart
4fd6955c5a
resolve separate frames 2024-10-25 05:40:28 -05:00
Cassandra Heart
7ad50ec33b
temporarily bulk process on frame candidates 2024-10-25 04:45:46 -05:00
Cassandra Heart
eccf5ebf59
faster 2024-10-25 02:39:59 -05:00
Cassandra Heart
7296cb0ea2
more adjustments from testing 2024-10-25 00:46:38 -05:00
Cassandra Heart
ec94872ca7
handle rebroadcast quirks 2024-10-25 00:10:58 -05:00
Cassandra Heart
65c2585cc8
ok, ready for testnet 2024-10-24 23:23:10 -05:00
Cassandra Heart
7396ed97e8
adjust: don't grab the very latest while it's already being broadcasted 2024-10-24 22:59:01 -05:00
Cassandra Heart
5e521ca50f
adjust to previous sync handling 2024-10-24 22:51:35 -05:00
Cassandra Heart
95b0c13a4e
change iterator, genesis for testnet 2024-10-24 22:39:40 -05:00
Cassandra Heart
550ce3ec0c
fix: don't use iterator for frame directly until iterator is fixed 2024-10-24 22:31:17 -05:00
Cassandra Heart
6b3f42cc4b
wait a little longer 2024-10-24 22:17:53 -05:00
Cassandra Heart
33015adc53
more small adjustments 2024-10-24 22:15:36 -05:00
Cassandra Heart
1e590ad159
qol: sync by rebroadcast 2024-10-24 21:54:51 -05:00
Cassandra Heart
b7d2e17c63
final parameterization, parallelize streams 2024-10-23 14:26:01 -05:00
Cassandra Heart
b3b3afad4b
qol: use separate channel for worker 2024-10-23 01:34:39 -05:00