Commit Graph

280 Commits

Author SHA1 Message Date
petricadaipegsp
e0993a94ea
Avoid further copies of the tries (#393)
* Revert logging change

* Avoid further copies of tries

* Avoid sync on beacon

* Avoid recreating the address for each transaction
2024-12-01 15:24:38 -06:00
petricadaipegsp
1b78d758f5
Prefer connected peers for sync (#395)
* Add externally reachable data peer flag

* Announce node reachability

* Go through candidates based on reachability
2024-12-01 15:07:08 -06:00
petricadaipegsp
4be1888496
Separate dialing from retrieval (#398) 2024-12-01 15:02:07 -06:00
petricadaipegsp
dd030560bc
Do not clone full tries (#392) 2024-11-28 06:21:04 -06:00
petricadaipegsp
46e88bfc95
Deprecate zero timestamp transactions (#391) 2024-11-28 05:04:18 -06:00
petricadaipegsp
8b61918d43
Add transaction request validation (#390) 2024-11-27 18:13:57 -06:00
Cassandra Heart
87f4d7014c
only ignore, and use peer id of sender, not hop 2024-11-27 00:52:55 -06:00
Cassandra Heart
0242eafa3e
add decay, make validation check a little smarter 2024-11-26 23:45:20 -06:00
Cassandra Heart
28e8331ca1
refined to work across the board 2024-11-26 23:24:05 -06:00
Cassandra Heart
8a26318e1a
remove minversion check, no need 2024-11-26 23:06:44 -06:00
Cassandra Heart
70b30ddde1
don't set within validator 2024-11-26 22:52:22 -06:00
Cassandra Heart
7b2de084fd
no, really, get out 2024-11-26 21:05:01 -06:00
Cassandra Heart
6fba1a3cd9
apply to set early 2024-11-26 21:02:08 -06:00
Cassandra Heart
75716be4fa
さよなら 2024-11-26 18:50:25 -06:00
Cassandra Heart
3d58907e1c
bump patch version, support filtering on patch version 2024-11-24 18:44:14 -06:00
petricadaipegsp
7fd4d32521
Always start sync from head (#386)
* Always start sync from head

* Allow skips during sync using data time reel head
2024-11-24 18:09:17 -06:00
petricadaipegsp
ec37cb34d7
Add structured stop procedure (#379) 2024-11-24 17:07:15 -06:00
petricadaipegsp
d8321bf812
Deduplicate mint requests by public key (#381) 2024-11-24 17:06:16 -06:00
Black Swan
849feddf84
enhance testnet frame pruning (#377)
* reduce testnet frame pruning delay to 1 minute

* set seniority repair cutoff frame for testnet to 25745

* add log message to confirm frame pruning is enabled
2024-11-22 10:29:48 -06:00
Cassandra Heart
f2a3bd2d8e
don't try to make workers run if they aren't in ready state 2024-11-22 01:52:36 -06:00
petricadaipegsp
44ccd14871
Use buffered channels when applicable (#373)
* Use buffered channels when applicable

* Do not start additional goroutines for processing

* Use context to stop ongoing loops
2024-11-21 19:32:04 -06:00
petricadaipegsp
db66b3f118
Optimise token request duplicate handling (#375)
* Optimise token request duplicate handling

* Do not hold lock while applying transitions
2024-11-21 18:00:29 -06:00
Cassandra Heart
df77d408dc
make it wait until one after 2024-11-20 20:31:46 -06:00
petricadaipegsp
b798de5871
Trigger sync on ahead peer (#366) 2024-11-20 17:12:57 -06:00
petricadaipegsp
4d3ac60e2b
Increase default sync timeout to 4 seconds (#369) 2024-11-20 16:59:24 -06:00
petricadaipegsp
bc05a4d7b9
Adaptive reserved cores (#363)
* Add adaptive data worker count

* Use runtime worker count for validation workers

* Reserve cores for networking during transition application

* Automatically set GOGC and GOMEMLIMIT
2024-11-19 16:51:14 -06:00
Cassandra Heart
91e8a16124
don't apply pruning until after seniority cutoff 2024-11-18 02:17:42 -06:00
Cassandra Heart
61179c13a0
node info should show worker counts 2024-11-18 02:03:37 -06:00
Cassandra Heart
4b69a40e56
more adjustment to validators 2024-11-17 16:22:32 -06:00
Cassandra Heart
5d1b572035
switch validators to sync 2024-11-17 16:00:51 -06:00
Cassandra Heart
063102eab2
adjust validators 2024-11-17 15:58:26 -06:00
Cassandra Heart
29a129d543
allow choice of indexed or non-indexed batches 2024-11-17 07:58:15 -06:00
Cassandra Heart
f511149b36
frame pruning 2024-11-17 04:37:55 -06:00
petricadaipegsp
0b499c649c
Move time triggered synchronization to dedicated goroutine (#359) 2024-11-17 02:18:55 -06:00
Cassandra Heart
a798347ccf
re-add #358 2024-11-16 23:20:09 -06:00
Cassandra Heart
6e2c7f4028
Revert "Use buffered channels in the engines (#357)"
This reverts commit d32d79f58d.
2024-11-16 23:11:25 -06:00
Cassandra Heart
d97a093c18
remove logging in validators due to locking, allow 2.0.3 minversion 2024-11-16 23:11:19 -06:00
Cassandra Heart
706ec2997d
adjust runonce 2024-11-16 18:29:41 -06:00
Cassandra Heart
d63f587d40
reduce bandwidth on workers 2024-11-16 17:58:57 -06:00
petricadaipegsp
d32d79f58d
Use buffered channels in the engines (#357)
* Make time data reel sending consistent

* Use buffered channels to avoid drops
2024-11-16 17:58:32 -06:00
petricadaipegsp
49566c2280
Add additional P2P configuration (#352)
* Add peer discovery configuration

* Add peer monitor configuration

* Add message validation configuration

---------

Co-authored-by: Cassandra Heart <7929478+CassOnMars@users.noreply.github.com>
2024-11-16 17:54:34 -06:00
petricadaipegsp
80c7ec2889
Add initial Prometheus support (#353)
* Add Prometheus server

* Add Prometheus gRPC metrics

* Add BlossomSub metrics

---------

Co-authored-by: Cassandra Heart <7929478+CassOnMars@users.noreply.github.com>
2024-11-16 17:53:19 -06:00
petricadaipegsp
d57d76e627
Add transaction timestamps (#354) 2024-11-16 17:52:27 -06:00
petricadaipegsp
da5b7a6126
Add sync timeout and make it async (#356)
* Add sync timeout

* Make sync async
2024-11-16 17:50:25 -06:00
Cassandra Heart
8b18568bae
the mystery of 6% 2024-11-13 14:51:41 -06:00
petricadaipegsp
37761ebd83
Log frame age (#351) 2024-11-13 11:36:49 -06:00
Cassandra Heart
07a27d7dc6
set it under either case 2024-11-13 10:43:57 -06:00
petricadaipegsp
5333b4a8cc
Sample peers which are ahead during sync (#337) 2024-11-13 10:43:28 -06:00
petricadaipegsp
f06d2c0ff2
Ensure latest frame monotonicity (#350) 2024-11-13 10:41:23 -06:00
Cassandra Heart
e58280b7e4
small adjustments 2024-11-13 10:00:37 -06:00
Cassandra Heart
22c0b55ab8
nil check 2024-11-13 06:36:08 -06:00
Cassandra Heart
b0a87b2fe4
wrapping up 2024-11-13 06:03:14 -06:00
Cassandra Heart
c2b9b1d460
resolve:
- critbit trie bug
- proof messages performing incorrect repeat
2024-11-13 00:10:22 -06:00
Cassandra Heart
71b13c5490
v2.0.3-p3 2024-11-12 03:33:20 -06:00
Cassandra Heart
d4a5e30ba2
v2.0.3-p3 2024-11-12 01:45:18 -06:00
Cassandra Heart
7a1ba529b0
Merge branch 'develop' into v2.0.3-p2 2024-11-11 15:48:50 -06:00
petricadaipegsp
db28f1b81e
Remove vendored gostream (#347)
* Remove vendored go-libp2p-gostream

* Remove error wrapping
2024-11-11 15:05:45 -06:00
petricadaipegsp
3dbe0723bd
Add message validators (#346) 2024-11-11 14:10:00 -06:00
Cassandra Heart
26cbb2092e
rate limiter logic 2024-11-11 13:30:04 -06:00
Cassandra Heart
b9694ddc99
fix rate limit 2024-11-11 11:32:11 -06:00
petricadaipegsp
de87f3f81b
Change frame delivery filter (#345) 2024-11-11 11:11:28 -06:00
petricadaipegsp
4e108bff07
Safe peer ID lookup from gostream (#344) 2024-11-11 11:10:53 -06:00
Cassandra Heart
2ce7eb26d4
change order of operations, don't commit an empty tree? 2024-11-11 04:39:52 -06:00
Cassandra Heart
1d262794cb
nil check 2024-11-11 03:50:13 -06:00
Cassandra Heart
4238b3ff5a
initial testnet v2.0.3-p2 2024-11-11 03:34:28 -06:00
Cassandra Heart
77c036b94b
v2.0.3-p0 2024-11-11 03:33:35 -06:00
Cassandra Heart
e178153593
initial testnet v2.0.3-p2 2024-11-11 03:32:14 -06:00
petricadaipegsp
05f1bf94fe
Remove additional signatures from data peer list announcements (#335)
* Do not verify data peer announcement signatures

* Do not sign data peer announcements explicitly

* Implement single peer data peer list announcement
2024-11-10 06:03:38 -06:00
Cassandra Heart
cf89b1c4ee
v2.0.3-p0 2024-11-10 01:15:15 -06:00
Cassandra Heart
042848e7cb
minor nice-to-haves 2024-11-09 16:04:12 -06:00
Cassandra Heart
1c0d707f7d
add bounds check for proofs 2024-11-09 15:54:49 -06:00
Cassandra Heart
67d454acb9
add light prover support 2024-11-09 14:46:53 -06:00
Cassandra Heart
812939a97a
v2.0.3-b7 2024-11-08 20:29:30 -06:00
Cassandra Heart
71a3e70129
log to error 2024-11-08 04:47:58 -06:00
Cassandra Heart
bc851de1bb
add testnet variant of seniority and penalties 2024-11-08 03:54:32 -06:00
Cassandra Heart
3a84cc65c8
ctrl-c = stop, damn it 2024-11-08 00:48:26 -06:00
Cassandra Heart
9a6a8a7a8e
v2.0.3-b5 2024-11-07 20:06:59 -06:00
Cassandra Heart
7ac7fc2b67
v2.0.3-b4 2024-11-07 18:03:50 -06:00
Cassandra Heart
5c924eab68
add simple reapplication logic 2024-11-07 13:14:12 -06:00
Cassandra Heart
9bc8a84f1a
fix dangling mutex lock 2024-11-07 03:05:37 -06:00
Cassandra Heart
80be91d067
v2.0.3-b3 2024-11-06 21:43:55 -06:00
Cassandra Heart
84279f2b5c
handle mints with more failure proofing 2024-11-06 20:34:08 -06:00
Cassandra Heart
5851db6433
move this back 2024-11-06 18:04:54 -06:00
Cassandra Heart
cb0918212e
make proving check the prior landed 2024-11-06 16:35:50 -06:00
Cassandra Heart
5679b19fdb
use up to date info 2024-11-06 14:46:04 -06:00
petricadaipegsp
b3cc442efd
Stop master clock spin (#334) 2024-11-06 04:04:06 -06:00
Cassandra Heart
f3e502a2d1
v2.0.3-b2 2024-11-06 03:51:30 -06:00
Cassandra Heart
e469fad46a
don't spin workers 2024-11-05 22:00:40 -06:00
Cassandra Heart
d2c5b42cdf
fix iterator bug 2024-11-05 21:24:27 -06:00
Cassandra Heart
6578040774
also 2024-11-05 17:41:42 -06:00
Cassandra Heart
3279666948
final adjustments 2024-11-05 17:37:17 -06:00
petricadaipegsp
ceb53b39c9
Use peer address while checking if date time proofs should be created (#333)
* Check PubSub key in trie, not proving key

* Use gRPC errors
2024-11-05 17:04:14 -06:00
Cassandra Heart
cdc1618964
get tries in sequence of events 2024-11-05 02:31:14 -06:00
Cassandra Heart
ee7ff329d7
fix: don't loop, reset genesis, shut off handlers on sigint 2024-11-04 23:49:15 -06:00
Cassandra Heart
8f955523d3
use correct signature path 2024-11-04 20:05:07 -06:00
petricadaipegsp
e23ad7869c
Trigger automatic peer discovery on frame stall (#328) 2024-11-04 19:25:30 -06:00
Cassandra Heart
d7501a142e
don't announce unless we're up to head 2024-11-04 18:24:12 -06:00
Cassandra Heart
a9225b2508
2.0.3-b1 2024-11-04 18:09:35 -06:00
Cassandra Heart
4b61a00095
restore prover rings 2024-10-31 23:44:23 -05:00
Cassandra Heart
3dd9a0c5f3
get develop caught up (#322)
* Update qcommander.sh bootrap (#304)

* v2.0.1 (#308)

* roll up v2.0.1-b2 to develop

* b2-fixed

* adjust return data of fast sync so it doesn't return the earliest frame

* -b3

* fix: announce peer based on leading frame, not initial frame; fix: looping bug

* fix: last batch fails due to underflow; qol: make logging chattier

* -b4

* resolve frame cache issue

* fix: mint loop + re-migrate

* fix: register execution panic

* fix: mint loop, other side

* fix: handle unexpected return of nil status

* final -b4

* handle subtle change to migration

* qol: add heuristic to handle corruption scenario

* bump genesis

* qol: use separate channel for worker

* final parameterization, parallelize streams

* deprecate signers 10, 11, 14, 17

* adjust signatory check size to match rotated out signers

* V2.0.2.3 (#321)

* roll up v2.0.1-b2 to develop

* b2-fixed

* adjust return data of fast sync so it doesn't return the earliest frame

* -b3

* fix: announce peer based on leading frame, not initial frame; fix: looping bug

* fix: last batch fails due to underflow; qol: make logging chattier

* -b4

* resolve frame cache issue

* fix: mint loop + re-migrate

* fix: register execution panic

* fix: mint loop, other side

* fix: handle unexpected return of nil status

* final -b4

* handle subtle change to migration

* qol: add heuristic to handle corruption scenario

* bump genesis

* qol: use separate channel for worker

* final parameterization, parallelize streams

* Add direct peers to blossomsub (#309)

Co-authored-by: Tyler Sturos <tyler.john@qcommander.sh>

* chore(docker): add ca-certificates to fix x509 error. (#307)

* Update qcommander.sh bootrap (#304)

* chore(docker): add ca-certificates to fix x509 error.

---------

Co-authored-by: Tyler Sturos <55340199+tjsturos@users.noreply.github.com>

* deprecate signers 10, 11, 14, 17

* adjust signatory check size to match rotated out signers

* qol: sync by rebroadcast

* upgrade version

* more small adjustments

* wait a little longer

* fix: don't use iterator for frame directly until iterator is fixed

* change iterator, genesis for testnet

* adjust to previous sync handling

* adjust: don't grab the very latest while it's already being broadcasted

* ok, ready for testnet

* handle rebroadcast quirks

* more adjustments from testing

* faster

* temporarily bulk process on frame candidates

* resolve separate frames

* don't loop

* make worker reset resume to check where it should continue

* move window

* reduce signature count now that supermajority signed last

* resolve bottlenecks

* remove GOMAXPROCS limit for now

* revisions for v2.0.2.1

* bump version

* bulk import

* reintroduce sync

* small adustments to make life better

* check bitmask for peers and keep alive

* adjust reconnect

* ensure peer doesn't fall off address list

* adjust blossomsub to background discovery

* bump version

* remove dev check

* remove debug log line

* further adjustments

* a little more logic around connection management

* v2.0.2.3

* Fix peer discovery (#319)

* Fix peer discovery

* Make peer discovery connections parallel

* Monitor peers via pings (#317)

* Support QUILIBRIUM_SIGNATURE_CHECK in client (#314)

* Ensure direct peers are not pruned by resource limits (#315)

* Support pprof profiling via HTTP (#313)

* Fix CPU profiling

* Add pprof server support

* Additional peering connection improvements (#320)

* Lookup peers if not enough external peers are available

* Make bootstrap peer discovery sensitive to a lack of bootstrappers

---------

Co-authored-by: Tyler Sturos <55340199+tjsturos@users.noreply.github.com>
Co-authored-by: Tyler Sturos <tyler.john@qcommander.sh>
Co-authored-by: linquanisaac <33619994+linquanisaac@users.noreply.github.com>
Co-authored-by: petricadaipegsp <155911522+petricadaipegsp@users.noreply.github.com>

---------

Co-authored-by: Tyler Sturos <55340199+tjsturos@users.noreply.github.com>
Co-authored-by: Tyler Sturos <tyler.john@qcommander.sh>
Co-authored-by: linquanisaac <33619994+linquanisaac@users.noreply.github.com>
Co-authored-by: petricadaipegsp <155911522+petricadaipegsp@users.noreply.github.com>
2024-10-31 16:46:58 -05:00