feat: ingest stage tx scheduling backpressure #39
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…sed-tx-fetching-capacity-adjustment
…sed-tx-fetching-capacity-adjustment
lancevincentsalera
changed the title
lancevincentsalera
force-pushed
the
feat/queued-length-based-tx-fetching-capacity-adjustment
branch
from
c7c8b4c to
5252394
Compare
mduthey
added a commit
that referenced
this pull request
Apr 8, 2025
commit 4624a6b Author: Lance Vincent A. Salera <88526526+lancevincentsalera@users.noreply.github.com> Date: Thu Apr 3 04:13:57 2025 +0800 feat: implement server-side signing via hashicorp vault service (#41) * feat: integrate HashiCorp Vault for mnemonic key management * feat: enhance VaultAdapter to support generic secret types * feat: update mnemonic generation to accept passphrase and refactor Secret struct * feat: introduce signing module with HashiCorp Vault integration for secret management * refactor: get key via config and remove unnecessary enums * refactor: reorganize module imports and improve formatting in main.rs * feat: update SecretAdapter to accept a key parameter and enhance Config with server_signing option * refactor: update transaction signing to use mnemonic directly and clean up imports * refactor: remove direct instantiation of HashiCorpVaultClient in main and move to pipeline run function * refactor: replace SecretAdapter with SigningAdapter and streamline transaction signing process * refactor: update SigningAdapter to accept owned data and improve transaction signing logic * refactor: replace KeyValuePairs with BTreeMap type * chore: update pallas dependency feature flag * refactor: update phase one and phase two validation functions to use new feature flag names commit 997504e Author: Lance Vincent A. Salera <88526526+lancevincentsalera@users.noreply.github.com> Date: Fri Mar 21 22:23:08 2025 +0800 feat: ingest stage tx scheduling backpressure (#39) * feat: adjust transaction fetching capacity based on queued length * fix: undequeued broadcast channel messages because of peer_manager receiver reference * refactor: rename start_recv_drain to start_recv_task for clarity * refactor: update PeerManager to use broadcast sender's subscribe() * refactor: update PeerManager to use receiver instead of sender for input connection commit ad43bce Author: Lance Vincent A. Salera <88526526+lancevincentsalera@users.noreply.github.com> Date: Fri Mar 21 22:22:56 2025 +0800 feat: refactor PeerManager to use broadcast sender for input connections (#40) commit 7657919 Author: Lance Vincent A. Salera <88526526+lancevincentsalera@users.noreply.github.com> Date: Tue Mar 18 19:39:46 2025 +0800 feat: tx validation module for ingest stage / grpc endpoint (#37) * feat: tx validator module for ingest stage / grpc endpoint * refactor: remove TxValidator struct and integrate validation logic directly * refactor: replace U5cDataAdapterImpl with a trait object for improved flexibility commit ba103c6 Author: Lance Vincent A. Salera <88526526+lancevincentsalera@users.noreply.github.com> Date: Mon Mar 17 21:49:11 2025 +0800 feat: handle potential memory leak (#38) commit 3a636b3 Author: Lance Vincent A. Salera <88526526+lancevincentsalera@users.noreply.github.com> Date: Thu Mar 13 21:57:39 2025 +0800 feat: initial boros tx validation implementation (#28) * Implement sorting algorithm (#25) * feat: updated priority to queue * feat: implemented priority config * feat: added priority module * chore: adjusted lint * feat: implemented priority quote * feat: implemented storage query to be used in priority module * test: implemented priority test * chore: removed dbg * chore: added validation when the state is empty * docs: added priority docs * chore: updated config structure * docs: updated docs * chore: changed workflow job name * chore: changed workflow job name * feat: boros tx validation initial proof of concept * feat: phase 1 and phase 2 validation code for ingest stage * feat: improve u5c method implementations * chore: update pallas dependency to use git source * refactor: simplify fetch_pparams function and update ingest tests * refactor: remove params mapping for other eras other than conway * refactor: update transaction evaluation in ingest stage * chore: update pallas dependency to use git source and add features * chore: remove phase one feature flag * chore: format import statements * refactor: simplify type annotations in transaction validation functions * refactor: expose fields in Config and PeerManagerConfig structs for easier access * feat: enhance transaction validation and update broadcast channel capacity * feat: simplify transaction processing by removing redundant status assignment * feat: streamline transaction validation by removing redundant processing * fix: ingest stage / tx validation unit tests * feat: reimplement peer sharing broken peer handling --------- Co-authored-by: Paulo Bressan <paulo.bressan@outlook.com> commit 903aad2 Author: Clark Alesna <clark_alesna@hotmail.com> Date: Wed Mar 12 02:41:01 2025 +0800 feat: boros gasket broadcast queue / backpressure (#36) * feat: add broadcast queue stage * refactor: improve stage initialization and transaction capacity handling * feat: integrate U5cDataAdapter into broadcast stage for enhanced transaction handling * refactor: remove unused PeerSharingClient from Peer struct * feat: introduce mempool module with basic FIFO functionality and transaction handling * fix: mempool locking contention that causes inconsistency in tx submission * feat: add peer discovery stage to the pipeline for improved transaction processing * refactor: update transaction handling in mempool and peer modules for improved clarity and efficiency * refactor: replace receiver with sender in PeerManager for improved message handling * fix: improve transaction reception handling in Peer by addressing lagged and closed channel scenarios * feat: broadcast implementation using gasket * refactor: follow gasket broadcast naming convention * refactor: improved handling of request txids blocking and nonblocking * feat: introduce network module with mempool and peer_manager implementations * feat: add MockOuroborosTxSubmitPeerServer implementation and update pipeline broadcast channel capacity * fix: handle lagged messages generically * fix: handle lagged broadcast messages generically * feat: add broadcast backpressure unit test simulation * feat: enhance MockOuroborosTxSubmitPeerServer with processing delay and improve broadcast tests * refactor: simplify transaction broadcasting logic in MockOuroborosTxSubmitPeerServer * refactor: extract transaction hashes for clarity in broadcast tests * fix: improve error handling in mempool methods * refactor: fix clippy warning about enum variant names * chore: update pallas dependency to use GitHub repo * fix: change transaction status from Validated to Pending in ingest pipeline --------- Co-authored-by: Lance Vincent Salera <salera.lancevincent@gmail.com> commit 6d2be22 Author: Paulo Bressan <paulo.bressan@outlook.com> Date: Mon Mar 10 11:07:01 2025 -0300 feat: implemented lock mechanism (#35) * feat: implemented lock mechanism * feat: improved code structure * docs: updated examples config * feat: implemented submit token validation * docs: updated tx-gen to accept chained queue * feat: implemented response with latest cbor * docs: updated config docs * chore: enabled fanout stage commit 7f7919e Author: Lance Vincent A. Salera <88526526+lancevincentsalera@users.noreply.github.com> Date: Mon Mar 10 17:55:26 2025 +0800 fix: peer sharing agency is theirs error (#33) * Implement sorting algorithm (#25) * feat: updated priority to queue * feat: implemented priority config * feat: added priority module * chore: adjusted lint * feat: implemented priority quote * feat: implemented storage query to be used in priority module * test: implemented priority test * chore: removed dbg * chore: added validation when the state is empty * docs: added priority docs * chore: updated config structure * docs: updated docs * chore: changed workflow job name * chore: changed workflow job name * feat: add timeout handling for peer sharing query and initialization * fix: handle peers with broken peer sharing by checking for agency * refactor: cleanup peer and peer_manager code --------- Co-authored-by: Paulo Bressan <paulo.bressan@outlook.com> commit 9c43d8a Author: Paulo Bressan <paulo.bressan@outlook.com> Date: Wed Mar 5 09:42:31 2025 -0300 feat: define proto spec for tx chaining lock mechanism (#32) * feat: added specs * feat: implemented custom spec to test lock mechanism * docs: updated examples with new spec commit 421e431 Author: Lance Vincent A. Salera <88526526+lancevincentsalera@users.noreply.github.com> Date: Mon Mar 3 18:44:52 2025 +0800 fix: fanout stage peer client locking contention (#34) * Implement sorting algorithm (#25) * feat: updated priority to queue * feat: implemented priority config * feat: added priority module * chore: adjusted lint * feat: implemented priority quote * feat: implemented storage query to be used in priority module * test: implemented priority test * chore: removed dbg * chore: added validation when the state is empty * docs: added priority docs * chore: updated config structure * docs: updated docs * chore: changed workflow job name * chore: changed workflow job name * fix: peer client locking contention * refactor: simplify client initialization --------- Co-authored-by: Paulo Bressan <paulo.bressan@outlook.com> commit 69847b5 Author: Clark Alesna <clark_alesna@hotmail.com> Date: Thu Feb 27 19:39:38 2025 +0800 refactor: update sorting algorithm (#31) * Implement sorting algorithm (#25) * feat: updated priority to queue * feat: implemented priority config * feat: added priority module * chore: adjusted lint * feat: implemented priority quote * feat: implemented storage query to be used in priority module * test: implemented priority test * chore: removed dbg * chore: added validation when the state is empty * docs: added priority docs * chore: updated config structure * docs: updated docs * chore: changed workflow job name * chore: changed workflow job name * chore: updated quota to cap * docs: updated priority docs --------- Co-authored-by: Paulo Bressan <paulo.bressan@outlook.com> commit da4181d Author: Clark Alesna <clark_alesna@hotmail.com> Date: Thu Feb 27 19:34:54 2025 +0800 feat: implement sorting algorithm (#30) * feat: updated priority to queue * feat: implemented priority config * feat: added priority module * chore: adjusted lint * feat: implemented priority quote * feat: implemented storage query to be used in priority module * test: implemented priority test * chore: removed dbg * chore: added validation when the state is empty * docs: added priority docs * chore: updated config structure * docs: updated docs --------- Co-authored-by: paulobressan <paulo.bressan@outlook.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
current_capusing gasket's exposed len() function