tests: create CLI tool to help with mainnet export testing #3082
Conversation
|
Hi! Could you post the But the log shows "This node is not a validator" on start. |
|
We managed to get around the issue mentioned by @dasanchez --it turns out this build only works if the gaia home is the default However, we then hit this nice consensus failure: we figure something's checking the number of validators in the set and panicking when it goes from 50 to 1 from one block to the next |
Sure I can, but I've just pushed the version where the code for modifying app and consensus state is executed by the newly added unsafe-set-local-validator command and not in the app.go, which is what we want and you have flags to add the keys from your local validator. To run the binary including this command use this commit and build gaia with the unsafe_set_local_validator tag(we agreed not to include this command unconditionally) Note: the pr is still in draft state, the command is tested only on local machine by running gaia app for some time, then stopping it, then taking that state and setting new validator info and starting the node again with new validators. It is possible that command might change when we test it with the real mainnet data and check if something more needs to be covered and see if we need some additional flags to make some things configurable. After we confirm everything we will write detail description how to use the command. In the case you want to use my keys and the first commit, I attached the files used val-files.zip |
yeah I ran to similar consensus failure error and also few different and fixed everything I run to. I managed to run the node after I switched validator but note that i did it with local gaia app, so maybe something more will happen when I take the mainnet data. Tomorrow will focus on testing, because until now I was spent time fixing those numerous consensus issues, Thanks for posting the error, I'll see if this is fixed with the new code. Can you point me to the chain state you used for testing to try the same tomorrow? And yes, will double check for some additional flags to add to the command. Depends on the way how we want to use it, but i thought the use case is following(correct me if I'm wrong):
note that unsafe_set_local_validator command will not start the node, it will just update the state. in sdk v0.50 there is server command AddTestnetCreatorCommand that offers convenient way of providing app with updated state and it will run node as well but since gaia is on v0.47 we're going with the unsafe_set_local_validator command for now(its good enough for now and later we can switch to AddTestnetCreatorCommand after upgrade and use the code from unsafe_set_local_validator for upgrading the app state). I'll do the tests tomorrow with the more representative state(can use the same you did) to check if everything works ok as it does now with my local gaia app state. |
this looks perfect to me! I do think it'd be potentially useful to have a different --home just to not have to wreck the existing I'm testing out the new command on a node export right now. You can download it here. This is the key I'm using: {
"address": "973C48DF8B3356C45E44494723A6E0D45DEB8131",
"pub_key": {
"type": "tendermint/PubKeyEd25519",
"value": "xAqzjs6UkEg8YvoQy60bxytIocODxoDTNRz4+H81tTc="
},
"priv_key": {
"type": "tendermint/PrivKeyEd25519",
"value": "V9E2OFJ8ghMu/M15KALuNh0ZafFBDt7aUrGcSPOfP9rECrOOzpSQSDxi+hDLrRvHK0ihw4PGgNM1HPj4fzW1Nw=="
}
} |
|
hmm i got an error from i'm using |
|
Actually, nevermind my above comment --realized I needed to pass the valoper address to the --validator-operator flag. I'm still getting the consensus failure when using the state I posted, though |
Hey, I've just pushed the changes that fixes mentioned consensus failure. Now the node works after switching validator, I had to add some updates to consensus state and staking module(staking was the another error after the cons err was fixed). I've tested this with the data you provided, thanks for that! Also, you can use --home flag it will be taken into consideration. I've also changed the command a bit, I has to add the flag for validator's private key as well because I had to add the validators vote signature(this key is in the priv_validator_key file where the pubkey is also). Here is example of the command: and I've noticed that indeed some addresses cannot be parsed to betch32 for some reason, I've tried some from the localMerlinAccounts and for some reason some of them give an error even when called with ./gaiad keys parse , which is weird but will take a look tomorrow why is that. You can use some addresses that can be parsed when running ./gaiad keys parse e.g. the one i used is cosmos1ju6tlfclulxumtt2kglvnxduj5d93a64r5czge. running the command takes ~10min on my machine for the data you provided, and after that node runs successfully |
|
Yes!! This latest build works. I'm able to fork the chain I linked and produce blocks 💃 |
stana-miric
requested review from
alexanderbez,
zmanian,
crodriguezvega,
jackzampolin and
a team
as code owners
|
I gave this a shot with v15 --you can see the branch I used here-- and a v15 mainnet snapshot from nodestake. The tool ran successfully as far as I can tell, but then the v15 chain refuses to start: I'm guessing my very naive v15 port was missing something or other |
Same error on my machine, but as it looks like it is because this state is significantly larger (~18gb app store) and it took 1.5 hours for the command to be executed, and it finishes without saving changes to the app store. I tried to find other way to save the changes without calling revert to previous height because that call is the one that lasts, otherwise the command would be very quick. But unfortunately did not have success. We would not have this problem once we switch to a command that runs the node immediately after changing the state (this is enabled in sdk v0.50). I can try a few more things when I return from holiday next Tuesday, in order to avoid calling rollback that makes a problem with large state. |
@stana-miric Stopping after swapping the valset is not a hard requirement, just a nice to have. Users can always stop the chain themselves. |
The command has been modified so that now, after it updates the state in the store, it also starts a node. The command name (as well as the build tag name) has been changed, so the command is now "unsafe-set-local-validator" instead of "unsafe-start-local-validator." Since the command also starts a node, in addition to the flags related to the new validator, we will specify the desired flags for starting the validator as well.
I've tested this with the new changes and everything works ok. Execution is also much faster compared to the old command. |
|
i think i got it to work, but i do see this sort of thing spammed in logs: not urgent, but if there's a way we could stop those, that'd be nice --they make it tough to read much of anything in logs |
Those error msgs dont have effect on node startup as you said, but you can remove them by simply deleting files from the cs.wal folder which is located in data folder (this is Write-Ahead Log but the node will start even if it is deleted) before running the command and starting the node |
MSalopek
changed the title
* replace validators * added unsafe-set-local-validator testnet command * fixed blockstate and lastcommit * update comments and descriptions * unsafe-set-local-validator changed to unsafe-start-local-validator * delete block from store if app and cons state are not at that version * add docs on testnet extensions * fix typo; add extra docs --------- Co-authored-by: MSalopek <matija.salopek994@gmail.com> (cherry picked from commit 14e5927)
* replace validators * added unsafe-set-local-validator testnet command * fixed blockstate and lastcommit * update comments and descriptions * unsafe-set-local-validator changed to unsafe-start-local-validator * delete block from store if app and cons state are not at that version * add docs on testnet extensions * fix typo; add extra docs --------- Co-authored-by: MSalopek <matija.salopek994@gmail.com> (cherry picked from commit 14e5927)
* replace validators * added unsafe-set-local-validator testnet command * fixed blockstate and lastcommit * update comments and descriptions * unsafe-set-local-validator changed to unsafe-start-local-validator * delete block from store if app and cons state are not at that version * add docs on testnet extensions * fix typo; add extra docs --------- Co-authored-by: MSalopek <matija.salopek994@gmail.com> (cherry picked from commit 14e5927)
…3095) * replace validators * added unsafe-set-local-validator testnet command * fixed blockstate and lastcommit * update comments and descriptions * unsafe-set-local-validator changed to unsafe-start-local-validator * delete block from store if app and cons state are not at that version * add docs on testnet extensions * fix typo; add extra docs --------- Co-authored-by: MSalopek <matija.salopek994@gmail.com> (cherry picked from commit 14e5927) Co-authored-by: Stana Miric <stana.miric@ethernal.tech>
…3096) * replace validators * added unsafe-set-local-validator testnet command * fixed blockstate and lastcommit * update comments and descriptions * unsafe-set-local-validator changed to unsafe-start-local-validator * delete block from store if app and cons state are not at that version * add docs on testnet extensions * fix typo; add extra docs --------- Co-authored-by: MSalopek <matija.salopek994@gmail.com> (cherry picked from commit 14e5927) Co-authored-by: Stana Miric <stana.miric@ethernal.tech>
…3097) * replace validators * added unsafe-set-local-validator testnet command * fixed blockstate and lastcommit * update comments and descriptions * unsafe-set-local-validator changed to unsafe-start-local-validator * delete block from store if app and cons state are not at that version * add docs on testnet extensions * fix typo; add extra docs --------- Co-authored-by: MSalopek <matija.salopek994@gmail.com> (cherry picked from commit 14e5927) Co-authored-by: Stana Miric <stana.miric@ethernal.tech>
This PR introduces a new
unsafe-start-local-validatorcommand that should be used only for testing.The command makes changes to the local mainnet node to make it suitable for local testing. After changing the data, command will also start the node. The changes include modification of consensus and application states by removing old validator data and injecting the new one, and funding the addresses to be used in testing without affecting existing addresses.
The command uses the approach described here and adds necessary changes to consensus state and staking module.
The command is added as a sub-command of the
gaiad testnetcommand. It is included in the gaia binary only if the tagunsafe_start_local_validatoris used during building it (make build BUILD_TAGS="-tag unsafe_start_local_validator").Example of running the command:
The use case is following:
The changes added in this command have some additional modifications compering to sdk guide:
Future improvements
AddTestnetCreatorCommandreceives the newTestnetApp app initer so we could easily switch to using that command when gaia is upgraded to v0.50.x, leveraging the current code since both commands do the same thing which is changing the state and running the node.