From 86d5e0a02b6d219ee784744b987a619570a81fcc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Andr=C3=A9=20Carneiro?= Date: Tue, 12 Dec 2023 15:13:58 -0500 Subject: [PATCH 1/5] docs: event queue connection class design --- .../0001-event-queue-connection-design.md | 195 ++++++++++++++++++ 1 file changed, 195 insertions(+) create mode 100644 projects/wallet-lib-client-for-event-queue/0001-event-queue-connection-design.md diff --git a/projects/wallet-lib-client-for-event-queue/0001-event-queue-connection-design.md b/projects/wallet-lib-client-for-event-queue/0001-event-queue-connection-design.md new file mode 100644 index 0000000..aa9d480 --- /dev/null +++ b/projects/wallet-lib-client-for-event-queue/0001-event-queue-connection-design.md @@ -0,0 +1,195 @@ +# Summary + +A new connection class that streams events from the fullnode event queue. +# Motivation + +The event queue is a way to sequentially stream events, making sure we do not miss any transaction or transaction update, we can also continue where we left in case of any connection issues, currently any connection issue makes it so we have to sync the entire wallet history from the beggining. +# Guide-level explanation + +The fullnode's event queue (a.k.a. reliable integration) has 2 ways to stream events, a websocket and a http api (not server-side events (SSE), just polling events with an event id based pagination). +We will focus on the websocket implementation but we can later create another class that uses the http api. + +## Events + +The events from the event queue are not filtered in any way, meaning we have to implement our own filter. +The events that represent new transactions or blocks is called `NEW_VERTEX_ACCEPTED` and any updates on the transaction data is called `VERTEX_METADATA_CHANGED` these 2 events come with the current transaction data, we can use this data to filter for transaction of our wallet. + +## Address subscription + +The current implementation uses the fullnode pubsub to listen only for addresses of our wallet, meaning that during startup we send a subscribe command for each address of our wallet. +Since the events will be filtered locally we can instead use the `subscribeAddresses` method to create a local list of addresses being listened to and use this list to filter the events. + +The "list of addresses" will be an object where the address is the key, since determining if an object has a key is O(1) we can ensure that this does not become a bottleneck for wallets with many addresses. +Alternatively, we could use the storage `isAddressMine` method which is already O(1). + +## Event streaming + +To get the full state of the wallet we would need to stream all events of the fullnode, but we can still use the address history api to fetch the balance and transactions of our addresses and start listening the newer events. + +The best way to achieve this is to use the event api to fetch a single event, this will come with the latest event id. +Example response of `GET ${FULLNODE_URL}/v1a/event?size=1` + +```json +{ + "events": [ + { + "peer_id": "ca084565aa4ac6f84c452cb0085c1bc03e64317b64e0761f119b389c34fcfede", + "id": 0, + "timestamp": 1686186579.306944, + "type": "LOAD_STARTED", + "data": {}, + "group_id": null + } + ], + "latest_event_id": 9038 +} +``` + +Then we save `latest_event_id` and sync the history with the address history api. +Once we have the wallet on the current state we can start streaming from the `latest_event_id`. +There can be transactions arriving during this process which would mean we add them during the history sync and during the event streaming, but this issue does not affect the balance or how we process the history. + +## Best block update + +The fullnode pubsub sends updates whenever the best chain height changes, this is so our wallets can unlock our block rewards, the event queue does not send an update like this but we receive all block transactions as events, meaning we can listen for any transaction with `version` 0 or 3 (block or merged mining block) and with the metadata `voided_by` as `null` (this is because if a block is not voided, it is on the main chain) and derive the best chain height. + +We will always expect the latest unvoided block to be the best chain newest block since during re-orgs (where the best chain changes) we will receive updates and the new best chain will be updated with `voided_by` as `null`. + +## EventQueueConnection class + +The `EventQueueConnection` class will manage a websocket instance to the event queue api and emit a `wallet-update` event, this is to keep compatibility with the existing `Connection` class. + +The `wallet-update` event will work with the schema: + +```ts +interface WalletUpdateEvent { + type: 'wallet:address_history', + history: IHistoryTx, +} + +// Where IHistoryTx is defined as: + +interface IHistoryTx { + tx_id: string; + signalBits: number; + version: number; + weight: number; + timestamp: number; + is_voided: boolean; + nonce: number, + inputs: IHistoryInput[]; + outputs: IHistoryOutput[]; + parents: string[]; + token_name?: string; + token_symbol?: string; + tokens: string[]; + height?: number; +} + +export interface IHistoryInput { + value: number; + token_data: number; + script: string; + decoded: IHistoryOutputDecoded; + token: string; + tx_id: string; + index: number; +} + +export interface IHistoryOutputDecoded { + type?: string; + address?: string; + timelock?: number | null; + data?: string; +} + +export interface IHistoryOutput { + value: number; + token_data: number; + script: string; + decoded: IHistoryOutputDecoded; + token: string; + spent_by: string | null; +} +``` + +The transaction data from the event queue is in a different format as described below: + +```ts +interface EventQueueTxData { + hash: string; + version: number; + weight: number; + timestamp: number; + nonce?: number; + inputs: EventQueueTxInput[]; + outputs: EventQueueTxOutput[]; + parents: string[]; + token_name?: string; + token_symbol?: string; + tokens: string[]; + metadata: EventQueueTxMetadata; + aux_pow?: string; +} + +interface EventQueueTxMetadata { + hash: string; + spent_outputs: EventQueueSpentOutput[]; + conflict_with: string[]; + voided_by: string[]; + received_by: string[]; + children: string[]; + twins: string[]; + accumulated_weight: number; + score: number; + first_block?: string; + height: number; + validation: string; +} + +interface EventQueueTxInput { + tx_id: string; + index: number; + data: string; +} + +interface EventQueueTxOutput { + value: number; + script: string; + token_data: number; +} + +interface EventQueueSpentOutput { + index: number; + tx_ids: string[]; +} +``` + + +### Data conversion process + +To keep compatibility with the current `Connection` class we need to convert the data with the following process: + +1. Convert `hash` to `tx_id` +2. Assert `nonce` is valid +3. Assign `signalBits` from `version` +4. Assign `is_voided` from metadata's `voided_by` +5. Assign `height` from metadata's height +6. Convert outputs +7. Convert inputs +8. remove `metadata` and `aux_pow` fields + +Process to convert outputs: + +1. Derive `token` from `token_data` and `tx.tokens` +2. Derive `spent_by` from the output index and `tx.metadata.spent_outputs` +3. Derive `decoded` from the script + +Process to convert inputs: + +1. Try to find the transaction with the input's `tx_id` in storage + 1. If not found we must fetch the tx from the fullnode api +2. Assign `value`, `token_data`, `script` and `token` from the spent output +3. Derive `decoded` from the script. + +Now that the data matches the current websocket transaction we can emit the data and all processes to manage history from the facade will work as intended. From 0e6162a6632a5f25e8f16eaaa280636540efe81f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Andr=C3=A9=20Carneiro?= Date: Wed, 13 Dec 2023 17:50:39 -0500 Subject: [PATCH 2/5] docs: connection manager design --- .../0001-event-queue-connection-design.md | 6 +- .../0002-connection-manager-design.md | 155 ++++++++++++++++++ 2 files changed, 160 insertions(+), 1 deletion(-) create mode 100644 projects/wallet-lib-client-for-event-queue/0002-connection-manager-design.md diff --git a/projects/wallet-lib-client-for-event-queue/0001-event-queue-connection-design.md b/projects/wallet-lib-client-for-event-queue/0001-event-queue-connection-design.md index aa9d480..08eff0c 100644 --- a/projects/wallet-lib-client-for-event-queue/0001-event-queue-connection-design.md +++ b/projects/wallet-lib-client-for-event-queue/0001-event-queue-connection-design.md @@ -1,10 +1,15 @@ # Summary +[summary]: #summary A new connection class that streams events from the fullnode event queue. + # Motivation +[motivation]: #motivation The event queue is a way to sequentially stream events, making sure we do not miss any transaction or transaction update, we can also continue where we left in case of any connection issues, currently any connection issue makes it so we have to sync the entire wallet history from the beggining. + # Guide-level explanation +[guide-level-explanation]: #guide-level-explanation The fullnode's event queue (a.k.a. reliable integration) has 2 ways to stream events, a websocket and a http api (not server-side events (SSE), just polling events with an event id based pagination). We will focus on the websocket implementation but we can later create another class that uses the http api. @@ -165,7 +170,6 @@ interface EventQueueSpentOutput { } ``` - ### Data conversion process To keep compatibility with the current `Connection` class we need to convert the data with the following process: diff --git a/projects/wallet-lib-client-for-event-queue/0002-connection-manager-design.md b/projects/wallet-lib-client-for-event-queue/0002-connection-manager-design.md new file mode 100644 index 0000000..9c20cbc --- /dev/null +++ b/projects/wallet-lib-client-for-event-queue/0002-connection-manager-design.md @@ -0,0 +1,155 @@ +# Summary +[summary]: #summary + +Create a connection manager class to organize connection classes and make the +wallet facade manage its connection to the source of transactions. + +# Motivation +[motivation]: #motivation + +The connection manager should be responsible for managing the state of the +connection and how the events from the connection are used, this way we can have +a multiple connection classes to handle the different behaviors but maintaining +a common interface. + +# Guide-level explanation +[guide-level-explanation]: #guide-level-explanation + + +## Current implementation + +The current `Connection` class implementation is a wrapper around a `WebSocket` +class, the `WebSocket` class is responsible for managing the connection and +actually sending and receiving data. +The `Connection` class will interpret the events from the websocket instance and +emit events that are used by the facade, it also subscribes to specific events +and handles them differently depending on the type of event. + +The layers of abstraction make the connection very easy to instantiate and use +but create a black box of events that are interpreted by the facade. + +## New connection classes + +### FullnodePubSubConnection and FullnodeEventQueueWSConnection + +Will connect to the fullnode websocket and subscribe to the apropriate +events, meaning the best block updates and the transaction events. + +Transaction events will be inserted into the storage and the history will be +processed (calculating metadata, balance, etc) by the connection instance. +Best block updates will be processed as usual. + +#### Events + +- `new-tx` and `update-tx` + - The event data will have the transaction id. // or data +- `best-block-update` + - The event data will have the new height +- `conn_state` + - Event data is a `ConnectionState`, so the wallet can know if it is receiving events in real + time or not. +- `sync-history-partial-update` + - This is so the wallet facade can update the UI when we are waiting for the + wallet to load. + - Will have 3 stages: `syncing`, `processing` and `fetching-tokens`. + - `syncing` means that the wallet is still fetching the history from the + fullnode. + - `processing` means that we are calculating the balance of the wallet. + - `fetching-tokens` means we are fetching the token data for our tokens. + +#### Methods + +- subscribeAddresses, unsubscribeAddresse(s) + - Start/stop listening for transactions with the provided addresses. +- start + - Start the websocket and add listeners +- stop + - Close the websocket and remove listeners +- getter methods for the configuration + - getNetwork, getServerURL + +The other methods will be internal. + + +#### pubsub vs event queue + +These classes have the same methods and the same events because they are meant +to be used by the same wallet facade (i.e. `HathorWallet`). +The difference between them is the underlying websocket class and how it is +managed. + +The events are also derived in a very different way, the pubsub will receive +only transactions from the wallet but the event queue will receive all events +from the fullnode and will have to filter them locally. + +Even with the different implementations since we will expose a common interface +the facade will be able to work with both of them interchangeably. + +### WalletServiceConnection + +The wallet-service connection does not have to handle transaction processing +since all of this is handled by the wallet-service. +The main concern of this connection is to re-emit events from the wallet-service +and to signal the wallet facade that it needs to update the "new addresses" list. + +#### Events + +- `new-tx` and `update-tx` + - The event data will have the transaction id. // or data +- `conn_state` + - Event data is a `ConnectionState`, so the wallet can know if it is receiving events in real + time or not. +- `reload-data` + - When the connection becomes available after it went offline this event will + be emitted to signal that the data needs to be reloaded. + +#### Methods + +- start +- stop +- setWalletId +- getter methods for the configuration + +The other methods will be internal. + +### FullnodeDataConnection + +This is the only connection class not meant to be used by a wallet facade but to +be used by an external client to get real-time events from the fullnode, an +example of how to use this is the explorer main page, to update with new +transactions and blocks. + +#### Events + +- `network-update` + - network events are related to peers of the connected node or when a new tx + is accepted by the network. +- `dashboard` + - Dashboard data from the fullnode. + +## Changes to the wallet facade + +The wallet facade will be updated to use the new connection classes. +This means that instead of receiving a connection instance the wallet facade +will receive the connection params and options (i.e. `connection_params`) and +will instantiate the appropriate connection class. + +The wallet-service facade can only use the `WalletServiceConnection`, but the +`HathorWallet` will need an aditional parameter to choose which connection class +to use. +The `connection_mode` argument will be introduced to resolve this issue, it will +be one of the following: + +- `pubsub` + - Uses `FullnodePubSubConnection` +- `event_queue` + - Uses `FullnodeEventQueueConnection` +- `auto` + +If not provided, the `pubsub` mode will be used, since this is the most common +connection used by our wallets. + +Ideally the `auto` mode would decide to use either `FullnodePubSubConnection` or +`FullnodeEventQueueConnection` based on the size of the wallet and other +parameters, but until we have a better strategy it will simply use +`FullnodePubSubConnection`. From 6da92d485e26a70b0ba694fdcb50571fb04ae7ea Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Andr=C3=A9=20Carneiro?= Date: Wed, 13 Dec 2023 18:03:09 -0500 Subject: [PATCH 3/5] docs: event queue stream from last acked event --- .../0002-connection-manager-design.md | 20 +++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/projects/wallet-lib-client-for-event-queue/0002-connection-manager-design.md b/projects/wallet-lib-client-for-event-queue/0002-connection-manager-design.md index 9c20cbc..54b5fc8 100644 --- a/projects/wallet-lib-client-for-event-queue/0002-connection-manager-design.md +++ b/projects/wallet-lib-client-for-event-queue/0002-connection-manager-design.md @@ -153,3 +153,23 @@ Ideally the `auto` mode would decide to use either `FullnodePubSubConnection` or `FullnodeEventQueueConnection` based on the size of the wallet and other parameters, but until we have a better strategy it will simply use `FullnodePubSubConnection`. + +## FullnodeEventQueueWSConnection connection managementstate + +Now that the connection class is responsible for syncing the history, we can +implement a special logic for the event queue. +When we are reestablishing the connection we can start from the last +acknowledged event. + +Usually when the connection is lost we will need to clean and sync the history +of transactions from the beginning since we may have lost some events, but the +event queue allows us to start streaming events from the moment we lost +connection. +This makes it so we don't have to clean the history and can just start the +stream from where we left off. +Although this is only applicable when we are connected to the same fullnode, so +we need to check if the fullnode we are connected to is the same and if it isn't +we will need to re-sync as usual. + +To make sure we are connected to the same fullnode we will use the peer-id, an +unique 32 byte identifier. From 3d94df68eb69c99eb99b4d0809ab80ec9f6908db Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Andr=C3=A9=20Carneiro?= Date: Fri, 22 Dec 2023 08:05:21 -0500 Subject: [PATCH 4/5] docs: phase 3 - individual tx processing --- .../0002-connection-manager-design.md | 3 +- .../0003-sequential-tx-processing.md | 137 ++++++++++++++++++ 2 files changed, 138 insertions(+), 2 deletions(-) create mode 100644 projects/wallet-lib-client-for-event-queue/0003-sequential-tx-processing.md diff --git a/projects/wallet-lib-client-for-event-queue/0002-connection-manager-design.md b/projects/wallet-lib-client-for-event-queue/0002-connection-manager-design.md index 54b5fc8..0ccdbe9 100644 --- a/projects/wallet-lib-client-for-event-queue/0002-connection-manager-design.md +++ b/projects/wallet-lib-client-for-event-queue/0002-connection-manager-design.md @@ -1,8 +1,7 @@ # Summary [summary]: #summary -Create a connection manager class to organize connection classes and make the -wallet facade manage its connection to the source of transactions. +Organize the connection manager classes and move responsabilities from the wallet facade to the connection manager. # Motivation [motivation]: #motivation diff --git a/projects/wallet-lib-client-for-event-queue/0003-sequential-tx-processing.md b/projects/wallet-lib-client-for-event-queue/0003-sequential-tx-processing.md new file mode 100644 index 0000000..6e22b23 --- /dev/null +++ b/projects/wallet-lib-client-for-event-queue/0003-sequential-tx-processing.md @@ -0,0 +1,137 @@ +# Summary +[summary]: #summary + +Process each transaction as they arrive making atomic changes to balance and +other metadata in storage. + +# Motivation +[motivation]: #motivation + +The current history processing strategy is not efficient and is not scalable to +large numbers of transactions. +With the new event queue connection we have a solution for the issues that +required the current aproach so we can now create a more scalable solution. + +# Guide-level explanation +[guide-level-explanation]: #guide-level-explanation + +When the wallet receives a new transaction or an existing transaction is +updated we need to update the calculated balances for the tokens on the +transaction since a token may have been spent or a transaction may have been +voided. +The current implementation simply erases all wallet metadata (balances, list of +utxos, etc) and calculates it by iterating over all transactions. While this +approach was needed in the past, due to how the event system was setup it would +take longer times with larger wallets. + +We will also have to consider the load of processing the history of transactions +on startup and when we are re-joining the network if the connection goes offline. + +## Processing a single transaction + +The core of the implementation will be a method to process a single transaction. +This method will first check if a transaction with the same id is already on +storage, this means we are updating an existing transaction. + +### New transactions + +If this is a new transaction but it is voided, we can ignore it and return null. +This should not happen but we will treat it as a no-op. + +Otherwise we need to iterate over the outputs of the transaction and for each one we need +to check if the address is from our wallet, if it is we save the: +- tokens involved in the transaction +- token balance +- addresses involved in the transaction (only the ones from our wallet) +- address balance (by token) +- utxos, for unspent outputs +- highest address index + +We only need to check the outputs since any change of the inputs should be +calculated by the update of their transaction. + +With the information above we can update the necessary data on the storage. + +### Update existing transactions + +We should first check if the transaction is being voided or unvoided. +Voiding means the `voided_by` field is `null` on storage and not `null` on the transaction. +Unvoiding means the `voided_by` field is `null` on the transaction and not `null` on storage. + +When voiding a transaction we need to undo any impact it has on the balance of +the wallet and the other metadata. +This means calculating the same data as a new transaction but we should make the +opposite changes (balances will be removed, etc). + +When unvoiding a transaction we can treat it as a new transaction since its +effect on the wallet was already removed when it was voided. + +If the `voided_by` field is the same we can iterate on the outputs and calculate +the changes the update will have on the wallet. +This is different from the new transaction because we will need to compare the +output from the transaction to the output from the storage. +Since a transaction cannot change any output data we can just check the metadata +`spent_by` which will indicate if the output is being spent or unspent. +With this information we can calculate the balance change, since the other +metadata will not change. + +### Output for a single transaction + +```ts +interface ITxMetadataEffect { + was_voided: boolean; + addresses: string[]; + tokens: string[]; + // A map of address to balance (indexed by token uid) + address_balance: Record>; + // A map of token uid to balance + token_balance: Record; + add_utxos: IUtxo[]; + del_utxos: IUtxo[]; + highest_address_index: number; +} +``` + +## Processing a stream of transactions + +To process a list or stream of transactions we will process each transaction, +make atomic changes on the storage then continue down the stream. + +We will calculate the `ITxMetadataEffect` for the transaction, then apply the +changes to the storage. +The process will be different depending on `was_voided`. + +### `was_voided === false` + +This means the transaction was not voided, so we will act on the `ITxMetadataEffect`. +This is the process for new transactions and updates. + +- `addresses`: We should add 1 to the `numTransactions` for each address (or set 1) +- `tokens`: We should add 1 to the `numTransactions` for each token (or set 1) +- `add_utxos`: Add the utxos to the storage +- `del_utxos`: Remove the utxos to the storage +- `highest_address_index`: Check if we have a new highest address index and set it. +- `token_balance`: for each token, add the balance to the existing balance. +- `address_balance`: for each address, add the balance to the existing balance. + +If `highest_address_index` is higher than the wallet's `lastUsedAddressIndex` we +will set it as the `lastUsedAddressIndex`. If it is higher than +`current_address_index` we will set the current address index to +`min(lastLoadedAddressIndex, highest_address_index + 1)`. + +If we had to set the `numTransactions` to 1 for any token, this means the token +just entered the wallet, this also means that we may have to fetch the token data +from the fullnode. + +### `was_voided === true` + +This means the transaction was voided, so we will remove the `ITxMetadataEffect`. + +- `addresses`: We should subtract 1 from `numTransactions` for each address + - If `numTransactions` is 1, this means that the address just became unused. +- `tokens`: We should subtract 1 from `numTransactions` for each token + - If `numTransactions` is 1, this means that the token was removed from the wallet. +- `add_utxos`: Add the utxos to the storage. +- `del_utxos`: Remove the utxos to the storage. +- `token_balance`: for each token, remove the balance from the existing balance. +- `address_balance`: for each address, remove the balance from the existing balance. From 805506644467e5c01996cf01da4bfde7bae8f343 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Andr=C3=A9=20Carneiro?= Date: Mon, 8 Jan 2024 14:11:57 -0300 Subject: [PATCH 5/5] chore: review changes --- .../0001-event-queue-connection-design.md | 1 - .../0002-connection-manager-design.md | 3 +-- .../0003-sequential-tx-processing.md | 2 +- 3 files changed, 2 insertions(+), 4 deletions(-) diff --git a/projects/wallet-lib-client-for-event-queue/0001-event-queue-connection-design.md b/projects/wallet-lib-client-for-event-queue/0001-event-queue-connection-design.md index 08eff0c..d8cc0c7 100644 --- a/projects/wallet-lib-client-for-event-queue/0001-event-queue-connection-design.md +++ b/projects/wallet-lib-client-for-event-queue/0001-event-queue-connection-design.md @@ -194,6 +194,5 @@ Process to convert inputs: 1. Try to find the transaction with the input's `tx_id` in storage 1. If not found we must fetch the tx from the fullnode api 2. Assign `value`, `token_data`, `script` and `token` from the spent output -3. Derive `decoded` from the script. Now that the data matches the current websocket transaction we can emit the data and all processes to manage history from the facade will work as intended. diff --git a/projects/wallet-lib-client-for-event-queue/0002-connection-manager-design.md b/projects/wallet-lib-client-for-event-queue/0002-connection-manager-design.md index 0ccdbe9..83e69b8 100644 --- a/projects/wallet-lib-client-for-event-queue/0002-connection-manager-design.md +++ b/projects/wallet-lib-client-for-event-queue/0002-connection-manager-design.md @@ -170,5 +170,4 @@ Although this is only applicable when we are connected to the same fullnode, so we need to check if the fullnode we are connected to is the same and if it isn't we will need to re-sync as usual. -To make sure we are connected to the same fullnode we will use the peer-id, an -unique 32 byte identifier. +To make sure we are connected to the same fullnode we will use the peer-id (32 byte identifier) and the `stream_id` (uuid4). diff --git a/projects/wallet-lib-client-for-event-queue/0003-sequential-tx-processing.md b/projects/wallet-lib-client-for-event-queue/0003-sequential-tx-processing.md index 6e22b23..73f2998 100644 --- a/projects/wallet-lib-client-for-event-queue/0003-sequential-tx-processing.md +++ b/projects/wallet-lib-client-for-event-queue/0003-sequential-tx-processing.md @@ -66,7 +66,7 @@ opposite changes (balances will be removed, etc). When unvoiding a transaction we can treat it as a new transaction since its effect on the wallet was already removed when it was voided. -If the `voided_by` field is the same we can iterate on the outputs and calculate +If the `voided_by` field is `null` we can iterate on the outputs and calculate the changes the update will have on the wallet. This is different from the new transaction because we will need to compare the output from the transaction to the output from the storage.