feat(core): multi-pubkey DB support #2093

onur-ozkan · 2024-04-01T09:30:57Z

This pull request enables support for managing multiple public keys in mm2 databases. Initially, a single database instance is generated using the startup public key rmd160. With this update, the mm_ctx will manage multiple database instances by leveraging the public key from MmCoin::account_db_id if defined, or defaulting to the mm2 startup public key.

Completed

Introduced sql_connection_pool for managing all instances of shared and sqlite connections from one place
Introduced async_sql_connection_pool for managing all instances of async sqlite connections from one place
Refactored indexeddb driver implementation to allow multi pubkey support
Refactored SwapContext, MySwapStorage, MakerSwapStorage, TakerSwapStorage NftCtx, UtxoBlockHeaderStorage, Zcoin storages(web only), lp_ordermatch and other storage impl to support multipubkey
Implement database migrations upon new database instance creation or account activation using a custom public key.

Complex Todos:

kick_start unfinished swap upon new database instance creation or account activation using a custom public key.

Checklists for next PR

discuss and enhance get_public_key, get_public_key_hash, show_priv_key rpc for multi pubkey dbs

Signed-off-by: onur-ozkan <work@onurozkan.dev>

onur-ozkan

I think tendermint_tx_history_v2 also needs some magic in order to keep the history data separated between inner-external accounts 🫤 I hope I am wrong.. A confirmation from GUI team or QA team could be nice.

mm2src/coins/hd_wallet/storage/wasm_storage.rs

mm2src/coins/lp_coins.rs

shamardy · 2024-05-28T13:16:35Z

@borngraced docker tests timeout after 90 minutes, my guess is that there is a test that uses the db that now runs forever after the current changes. can you please find the problem and try to fix it?

shamardy

Huge work! First review iteration where I cover most of the todo notes :)

shamardy · 2024-05-29T12:16:22Z

mm2src/coins/nft.rs

+    // TODO: db_id
+    let db_id: Option<String> = None;


This easy to fix I think, we should use the enabled platform coin db_id. Similar to here

komodo-defi-framework/mm2src/coins/nft.rs

Lines 1656 to 1657 in 24774a4

let db_id = coin.inner.account_db_id().unwrap_or_else(|| ctx.rmd160_hex());

let entry = active_id_chains.entry(db_id).or_insert_with(Vec::new);

I think you should start implementing account_db_id for all coins, this is needed for hd wallet (in utxo and eth) where the db_id will be the enabled_address one.

c.c. @laruh

This easy to fix I think, we should use the enabled platform coin db_id

Speaking in general (not about should we use platform coin db_id or id of enabled_address): if to fix this in clear_nft_db you need to require Nft activation, then please do it. Its not a problem, since most of NFT RPC already ask about it.

I think you should start implementing account_db_id for all coins, this is needed for hd wallet (in utxo and eth) where the db_id will be the enabled_address one.

need to do a deeper review of how current multi-pubkey feature work, to answer this question.
But now my answer is we should treat global NFT the same way as erc20 token (in general implementations), it is just not registered inerc20_tokens_infos platform coin field. So nft behavior should not interfere with the implementation of suggested idea.

I think you should start implementing account_db_id for all coins, this is needed for hd wallet (in utxo and eth) where the db_id will be the enabled_address one.

This was a comment for @borngraced so no need to do a deeper review, thanks for your input :)

I think you should start implementing account_db_id for all coins, this is needed for hd wallet (in utxo and eth) where the db_id will be the enabled_address one.

This is not done yet.

mm2src/mm2_main/src/lp_swap.rs

mm2src/mm2_main/src/lp_swap/pubkey_banning.rs

mm2src/mm2_main/src/lp_swap/swap_v2_common.rs

shamardy · 2024-05-29T12:28:11Z

mm2src/mm2_main/src/rpc/lp_commands/lp_commands.rs

+// TODO: Return public_key_hash for all available and active unique pubkeys
 pub async fn get_public_key_hash(ctx: MmArc, _req: Json) -> GetPublicKeyRpcResult<GetPublicKeyHashResponse> {


Same here, add this to a checklist to do in another PR. The enhancement can be to get a specific enabled coin pubkeyhash which is the enabled db_id.

mm2src/mm2_main/src/rpc/lp_commands/lp_commands.rs

mm2src/mm2_main/src/lp_ordermatch.rs

shamardy · 2024-05-29T12:32:43Z

mm2src/mm2_main/src/lp_ordermatch.rs

    let req: OrderStatusReq = try_s!(json::from_value(req));
-
+    let db_id: Option<String> = None; // TODO


This is tricky and not easy to solve, we need to search for a uuid across all databases. We need to check how this is used in GUIs if we want to simplify it.

smk762 · 2024-05-30T05:16:32Z

For QA reference, I'll list a few things here to be tested. Please add to the list (or tell me to remove) any items which are missing/not required.

With new HD wallet:

Perform swap as maker / taker for UTXO, ZHTLC, tendermint, polygon. Do with parent coin and token where applicable.
Perform withdraw / deposit of the UTXO, ZHTLC, tendermint, polygon. Do with parent coin and token where applicable.
Test failed swap recovery for UTXO, ZHTLC, tendermint, polygon. Do with parent coin and token where applicable.
Test all NFT related functions.

DB migration tests: with old HD wallet (use master branch to generate some activity, then update to this branch):

Perform swap as maker / taker for UTXO, ZHTLC, tendermint, polygon. Do with parent coin and token where applicable.
Perform withdraw / deposit of the UTXO, ZHTLC, tendermint, polygon. Do with parent coin and token where applicable.
Test failed swap recovery for UTXO, ZHTLC, tendermint, polygon. Do with parent coin and token where applicable.
Test all NFT related functions.

Regression testing

With a legacy (nonHD) wallet, repeat the above tests.

Stress testing:

Initiate as many simultaneous swaps as possible while also initiating as many other DB triggering methods at the same time (zhtlc activation, tx history, nft update etc).

With CLI:

inspect all database tables to identify updates required for https://komodoplatform.com/en/docs/komodo-defi-framework/tutorials/query-the-mm2-database/
Test seednode stats related methods

With Web:

Test the following case: if the same seed is used in both Keplr and mm2, and a swap is initiated using Keplr, then the app was closed midway through the swap. Upon restarting, if the Tendermint coin is activated using mm2 seed and the address enabled for swaps is the same address that was used before in Keplr, the swap will complete successfully. and its inverse.
Check tx history of tendermint assets to confirm tx history data is separated between inner and external accounts.

Outside the scope of testing

get_public_key, get_public_key_hash, show_priv_key

Related issues for context

shamardy

Only 2 comments in this review iteration!

shamardy · 2024-06-05T13:06:39Z

mm2src/coins/nft.rs

+    // TODO: db_id
+    let db_id: Option<String> = None;


I think you should start implementing account_db_id for all coins, this is needed for hd wallet (in utxo and eth) where the db_id will be the enabled_address one.

This is not done yet.

shamardy · 2024-06-05T13:31:30Z

mm2src/coins_activation/src/standalone_coin/init_standalone_coin.rs

            coin.start_history_background_fetching(
                self.ctx.metrics.clone(),
-                TxHistoryStorageBuilder::new(&self.ctx).build()?,
+                TxHistoryStorageBuilder::new(&self.ctx, coin_clone.into().account_db_id()).build()?,


Transaction history for HD wallet gets all transactions for all activated addresses, in my_tx_history_v2_impl the user specifies which address or addresses they want to see. We should used shared db for it but I see that this also create problems since activating the same address in non HD mode will lead to not knowing the shared db id. For now we can have redundancy for data where it will be fetched for both dbs, we use account_shared_db_id which should be called just shared_db_id

komodo-defi-framework/mm2src/coins/lp_coins.rs

Line 3212 in fbfe747

fn account_shared_db_id(&self) -> Option<H160> { None }

And in it's implementation we get either the shared_db_id of the hd wallet or the pubkeyhash (db_id) of other wallets depending on the activation mode.
P.S. I still need to examine this solution further to make sure it won't create any problems. Feel free to look at it also to make sure it will work fine.

onur-ozkan added 3 commits March 28, 2024 16:27

WIP: resolve db paths by account keys

aa0909f

Signed-off-by: onur-ozkan <work@onurozkan.dev>

WIP: handle db ids optionally

f051600

Signed-off-by: onur-ozkan <work@onurozkan.dev>

use dummy values to fix build errors for development

c8a434e

Signed-off-by: onur-ozkan <work@onurozkan.dev>

onur-ozkan added enhancement New feature or request in progress Changes will be made from the author labels Apr 1, 2024

borngraced added 2 commits April 9, 2024 14:49

fix conflicts after merge

3e0061f

more wip changes and start coin pubkey for indexeddb storage

c6146d0

borngraced self-assigned this Apr 9, 2024

borngraced added 5 commits April 9, 2024 23:50

Z coin or default pubkey for initializing blocksdb

eff374c

zcoin - allow db_id for walletdb

fef8a4e

wip — SwapsContext

6247766

changes to maker and taker swap db_id

fa0f5ab

complete zcoin dbs multi pubkey support for WASM target

947ab8d

borngraced linked an issue Apr 10, 2024 that may be closed by this pull request

Implementing Per-Coin Database Paths Based on PubkeyHash for Enhanced Modularization #2094

Open

borngraced added 3 commits April 10, 2024 17:13

fix clippy/fmt

ae95d4e

impl account_db_id for wasm target

c0d13e7

complete some todos in swap mod

71c4e9c

borngraced marked this pull request as ready for review April 10, 2024 20:01

borngraced added 12 commits April 10, 2024 21:03

minor changes to DbIdentifier constructor(new)

be3dc81

changes to swaps db

ffbc262

handle recv_swap_msg with db_id

07c1168

add todos for StateMachineStorage for TakerSwapStorage

1ec96bc

add todos for StateMachineStorage for TakerSwapStorage

3ccfa00

get_swap_repr

29763e2

mark account_db_id as todo for native target

cf749eb

impl find_unique_active_account_ids and use for swap_kick_starts

368441c

make swap_v2 multi_key

7095605

WIP - start nft multi key support

8823be2

WIP - my_swaps_storage

42a22b1

fix field assignment outside of initializer

23f0566

borngraced added 6 commits May 20, 2024 05:19

minor changes to DbMigrationWatcher

789ce8d

update coins_needed_for_kick_start for new db activation

ac5cc96

minor fix

d8caebd

comment out gui_storage_dispatcher and related

76efdf3

merge with dev and fix conflicts

24774a4

comment out gui_storage_accounts_functionality unit tests

6a0d0b6

onur-ozkan commented May 22, 2024

View reviewed changes

mm2src/coins/hd_wallet/storage/wasm_storage.rs Outdated Show resolved Hide resolved

mm2src/coins/hd_wallet/storage/wasm_storage.rs Outdated Show resolved Hide resolved

mm2src/coins/lp_coins.rs Show resolved Hide resolved

mm2src/coins/lp_coins.rs Outdated Show resolved Hide resolved

borngraced added 5 commits May 22, 2024 08:05

fix review notes

fbfe747

merge with dev and fix conflicts

097a63e

fix error handling

8a6d0cd

fix hd_wallet todos

f685f72

remove unneeded comments

df3cf1e

mark use_wacher as true

c028061

shamardy reviewed May 29, 2024

View reviewed changes

borngraced added 3 commits May 29, 2024 19:17

fix sqlite deadlock and improve api design

309a5ac

cargo clippy

3cfd616

remove todos after review

0b5275c

smk762 mentioned this pull request May 30, 2024

Update to KDF sqlite interaction docs. KomodoPlatform/komodo-docs-mdx#262

Open

borngraced added 2 commits May 30, 2024 10:47

fix build error

a53bd38

revert hd_wallet storage changes

ff42791

smk762 self-requested a review June 5, 2024 12:32

shamardy reviewed Jun 5, 2024

View reviewed changes

borngraced added 6 commits June 7, 2024 00:16

impl account_db_id for eth and make account_db_id async

4b99dba

impl account_db_id for utxo coins

2d427d9

impl account_db_id for utxo and eth

b516435

fix clippy

497a006

fix native unit tests and other fixes

df10264

fix clippy

d32f8f6

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(core): multi-pubkey DB support #2093

feat(core): multi-pubkey DB support #2093

onur-ozkan commented Apr 1, 2024 •

edited by borngraced

onur-ozkan left a comment

shamardy commented May 28, 2024

shamardy left a comment

shamardy May 29, 2024

shamardy May 29, 2024

laruh May 29, 2024

shamardy May 29, 2024

shamardy Jun 5, 2024

borngraced Jun 7, 2024

shamardy May 29, 2024

shamardy May 29, 2024

smk762 commented May 30, 2024

shamardy left a comment

shamardy Jun 5, 2024

shamardy Jun 5, 2024

	let db_id = coin.inner.account_db_id().unwrap_or_else(\|\| ctx.rmd160_hex());
	let entry = active_id_chains.entry(db_id).or_insert_with(Vec::new);

		// TODO: Return public_key_hash for all available and active unique pubkeys
		pub async fn get_public_key_hash(ctx: MmArc, _req: Json) -> GetPublicKeyRpcResult<GetPublicKeyHashResponse> {

		let req: OrderStatusReq = try_s!(json::from_value(req));

		let db_id: Option<String> = None; // TODO

feat(core): multi-pubkey DB support #2093

Are you sure you want to change the base?

feat(core): multi-pubkey DB support #2093

Conversation

onur-ozkan commented Apr 1, 2024 • edited by borngraced

Completed

Complex Todos:

Checklists for next PR

onur-ozkan left a comment

Choose a reason for hiding this comment

shamardy commented May 28, 2024

shamardy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

smk762 commented May 30, 2024

shamardy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

onur-ozkan commented Apr 1, 2024 •

edited by borngraced