Merge QGB feature branch into main #742

rach-id · 2022-09-20T11:32:42Z

This merge will add the QGB state machine to Celestia-app.

* removes dummy field from data commitment * adds TODOs * adds comments * linter fixes * linter fixes * linter fixes * linter fixes * linter fixes * CI fixes * query rename

* adds valset confirm logic and queries * adds valset confirm logic * removes unnecessary parameter from qgb keeper * removes unnecessary cli for valset confirm * updates comments and removes unnecessary ones * remove query command from module

* add_qgb_orchestrator_address_logic * updates comment * don't return giant struct for linting reason Co-authored-by: CHAMI Rachid <chamirachid1@gmail.com> * update imports Co-authored-by: Evan Forbes <42654277+evan-forbes@users.noreply.github.com>

* adds msgs + query proto * adds data commitment basic functions * adds data commitment query skeleton + server emitting events * linter changes * adds data commitment keeper + basic queries * persisting data commitment confirm on submission * adds data commitment confirms by validator query * adds verification for validator and ethereum address when handling data commitments confirm messages * adds data commitment by range keeper + query * adds comments * adds staking keeper to qgb and GetOrchestratorValidator implementation * adds delegate key query * adds qgb module address config * cosmetics * removes unnecessary comment * uses restricted version of staking keeper for QGB * cosmetics * removes config file

* add valset and params * create new val set request if the voting power has changed enough at the end of the block * comment out broken query method we don't strictly need yet * delete some comments

* fixes test common foundation * fixes valset confirm query + query by nonce test * go mod * remove unnecessary comment * cosmetics * adds query current valset test * adds prefix range test * adds current valset normalized power test * adds GetDelegateKeys test * adds NewMsgSetOrchestratorAddress test * adds ethereum signer test * adds genesis test * adds valset power diff test * adds valset sort test * adds Valset Creation Upon Unbonding test * adds abci tests * adds valset handler tests * adds no lint * adds no lint

* fixes data commitment keeper iterator * adds data commitment by commitment query * cosmetics * adds data commitment queries tests * cosmetics * adds data commitment message validation tests * adds commented msg data commitment confirm test

* adds handler test * adds events check for handler test

* use index prefix for commitments query * adds test for indexed query * linter

* adds overview and valset init ADRs * cleanup * adds Data commitment ADR init * improves the valset adr * cosmetics * adds more details and removes unnecessary stuff * cosmetics * proto format + ADR updates * updates proto files + ADR * adds endblocker doc to adr * typo

* initial orchestrator client * update go mod and change the makefile so it stops breaking the ci * copy and paste the needed code from the keystore lib

…reum/Orchestrator addresses to cosmos-sdk.staking.Validator (#276) * initial orchestrator client * format proto and add necessary queries * add orchestrator logic * and encoding of data commitments and validator set changes * add orchestrator command * add relayer * add the orchestrator command * proto formatting * use correct name Co-authored-by: CHAMI Rachid <chamirachid1@gmail.com> * use correct path to endpoint Co-authored-by: CHAMI Rachid <chamirachid1@gmail.com> * get rid of GravityNonces struct * use correct name Co-authored-by: CHAMI Rachid <chamirachid1@gmail.com> * Adds QGB related ADRs (#250) * adds overview and valset init ADRs * cleanup * adds Data commitment ADR init * improves the valset adr * cosmetics * adds more details and removes unnecessary stuff * cosmetics * proto format + ADR updates * updates proto files + ADR * adds endblocker doc to adr * typo * regenerate proto * go mod tidy * use correct name Co-authored-by: CHAMI Rachid <chamirachid1@gmail.com> * handle error * begin refactor * adds data commitments and valset orchestrator processors * add the app client that queries for validator sets and data commitments * add two new methods * add query methods to appClient * WIP: update orchestrator and relayer to use the appClient * WIP: update orchestrator and relayer to use the appClient * WIP: update orchestrator and relayer to use the appClient * adds evm_client implementation * add the nil clients for the evm and mockappclient * add the new methods to the mock app client * flesh out the mock app client * flesh out app orchestrator test * adds QueryLatestValset to appClient * wip: adds orchestrator test * adds orchestrator valset test * moves orchestrator test utils to a separate file * adds orchestrator valset processEvents error check in test * rename ValidatorSignature field in DataCommitmentConfirm to ValidatorAddress * fix valset orchestrator signature test * fix last valset request in orchestrator * fix last valset request in orchestrator common test * remove unnecessary nonce increment for data commitment signing in orchestrator * fix orchestrator data commitment test * closes the mocked app client in orchestrator test * add the mock evm client * fix the command to orchestrate valsets properly * get rid of zerologger * use tmlog instead of zerolog * remove extra err handling * add comment to question nonce usage in evm client submit dataroot * adds data commitment orchestrator test * adds deploy command * adds evmRpc flag to qgb config * deubgging the qgb install * add set orchestrator command * removes set Ethereum/Orch address + fixes tests + general cleanup (#296) * removing setting the address separatly and relying on the eth/orch addresses to be defined in sdk.staking.Validator * add tx command for qgb module * added proto generated files * fixes tests * adds todo * todo * pass the keyring backend and path to app client * adds error check the newKeyedTransactorWithChainId * rename QueryLastValset1 to QueryLastValsetRequests * rename evmChainIDFlag typo * rename QueryLastValsets * adds todo * update deploy command to use the right latest valset * adds todo * updated go.mod and go.sum * update the single-node.sh script to initialize three accounts and create one validator with an orchestrator and ethereum address * imports * adds events testing main.go * remove unused variable * use sweexordious fork for cosmos sdk in the app * update go.sum * update .gitignore * update lint.yml * linter fixes * new line * linter * todo * linter checks * linter checks * linter checks * adds mutexes for race conditions Co-authored-by: CHAMI Rachid <chamirachid1@gmail.com>

…ayer for testing (#314) * temporary disables listening for valsets and query for latest valset instead * app client listening for arbitrary events for valsets * temporary disable signatures for testing * temporarily disable threshold checks for relayer * update count in end blocker * fixes orchestrator listening to valset events * fix orchestrator signatures + enables checks on msg_server + makes bridgeId a const * fixes signature tests + adds new checks * adds todo * remove unnecessary comment * adds todo * adds todo * moves the signer outside of the NewAppClient * cosmetics * updates singe node script to only create one validator * uncomments the threshold for querying two third data commitment confirms * linter checks + cosmetics fixes * cosmetics * removes todo

* adds valset request by nonce * trigger CI * revert: trigger CI * update go.mod * Revert "update go.mod" This reverts commit 0007050.

* adds valset request by nonce * add querier, broadcaster and updates orchestrator to use them * partially fix the tests for the new design * update the deployer for the new design * update orchestrator to new design * Adds querier and evm client update * update relayer to new design * fix query.proto * update deploy_command to use new Querier * formatting * formatting * go.sum * revert go.mod change

* go mod tidy * go mod tidy

* adds valset request by nonce * add querier, broadcaster and updates orchestrator to use them * partially fix the tests for the new design * update the deployer for the new design * update orchestrator to new design * Adds querier and evm client update * update relayer to new design * fix query.proto * update deploy_command to use new Querier * formatting * formatting * go.sum * adds query last unbonding height

* uses the uint64 instead of int64 in right places * typo * cosmetics

…r) (#359)

* adds last valset request before height query * regenerating correct query.pb.go * adds check before querying next valset * Update x/qgb/keeper/query_valset.go Co-authored-by: John Adler <adlerjohn@users.noreply.github.com> * adds comment Co-authored-by: John Adler <adlerjohn@users.noreply.github.com>

* adds valset request by nonce * add querier, broadcaster and updates orchestrator to use them * partially fix the tests for the new design * update the deployer for the new design * update orchestrator to new design * Adds querier and evm client update * update relayer to new design * fix query.proto * update deploy_command to use new Querier * formatting * formatting * go.sum * adds query last unbonding height * adds orchestrator valset replay * adds orchestrator data commitment replay * cosmetics * puts valset signature catchup in a separate function * puts data commitment signature catchup in a separate function * Querier cosmetics * go.sum * rename querier and catchup functions * adds genesis case to querylastvalset * format + todo * initialized tm logger in test * update orchestrator code to use uint64 instead of int64 * better logging * Update x/qgb/orchestrator/orchestrator_client.go Co-authored-by: Evan Forbes <42654277+evan-forbes@users.noreply.github.com> * formats import * rename addOldValsetAttestations and addOldDataCommitmentAttestations * defering logging instead of repeating it on every return * adds comment * Update x/qgb/orchestrator/relayer.go Co-authored-by: John Adler <adlerjohn@users.noreply.github.com> * Update x/qgb/orchestrator/querier.go Co-authored-by: John Adler <adlerjohn@users.noreply.github.com> * remove unnecessary error Co-authored-by: Evan Forbes <42654277+evan-forbes@users.noreply.github.com> Co-authored-by: John Adler <adlerjohn@users.noreply.github.com>

* adds valset request by nonce * add querier, broadcaster and updates orchestrator to use them * partially fix the tests for the new design * update the deployer for the new design * update orchestrator to new design * Adds querier and evm client update * update relayer to new design * fix query.proto * update deploy_command to use new Querier * formatting * formatting * go.sum * adds query last unbonding height * adds orchestrator valset replay * adds orchestrator data commitment replay * cosmetics * puts valset signature catchup in a separate function * puts data commitment signature catchup in a separate function * Querier cosmetics * go.sum * rename querier and catchup functions * adds genesis case to querylastvalset * format + todo * initialized tm logger in test * update orchestrator code to use uint64 instead of int64 * better logging * adds last valset request before height query * regenerating correct query.pb.go * adds logs * adds querier QueryLastValsetBeforeHeight and QueryDataCommitmentConfirmsByExactRange * fixes relayer to correctly send commitments * Update x/qgb/orchestrator/orchestrator_client.go Co-authored-by: Evan Forbes <42654277+evan-forbes@users.noreply.github.com> * formats import * rename addOldValsetAttestations and addOldDataCommitmentAttestations * defering logging instead of repeating it on every return * adds comment * nit * use the right function for data commitment to tm.hash * better logging * missing from merge * add comments Co-authored-by: Evan Forbes <42654277+evan-forbes@users.noreply.github.com>

# Conflicts: # .gitignore # Makefile # app/app.go # go.mod # go.sum

codecov-commenter · 2022-09-20T11:56:25Z

Codecov Report

Merging #742 (13a9db9) into main (e58c8a5) will decrease coverage by 23.13%.
The diff coverage is 5.58%.

@@             Coverage Diff             @@
##             main     #742       +/-   ##
===========================================
- Coverage   46.38%   23.24%   -23.14%     
===========================================
  Files          33       71       +38     
  Lines        3251     8796     +5545     
===========================================
+ Hits         1508     2045      +537     
- Misses       1625     6578     +4953     
- Partials      118      173       +55

Impacted Files	Coverage Δ
app/app.go	`6.49% <0.00%> (ø)`
x/payment/types/builder.go	`40.10% <ø> (ø)`
x/qgb/genesis.go	`0.00% <0.00%> (ø)`
x/qgb/handler.go	`0.00% <ø> (ø)`
x/qgb/keeper/keeper_data_commitment.go	`0.00% <0.00%> (ø)`
x/qgb/keeper/msg_server.go	`0.00% <0.00%> (ø)`
x/qgb/keeper/query_attestation.go	`0.00% <0.00%> (ø)`
x/qgb/keeper/query_general.go	`0.00% <0.00%> (ø)`
x/qgb/keeper/query_valset.go	`0.00% <0.00%> (ø)`
x/qgb/module.go	`7.14% <0.00%> (ø)`
... and 49 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

evan-forbes · 2022-09-20T13:40:13Z

I will update the QGB universal nonce ADR to make the review easier

side note, we need to straighten out all of the ADR numbers

evan-forbes · 2022-09-20T13:45:15Z

imo, let's make as few changes to this PR as possible, since reviewing any of those changes would be difficult and not mesh with the git history. Instead if we have any gripes, we should create (a) follow up issue(s)

evan-forbes

super super super dope. glad we're finally getting this merged, and I'm really proud of us (almost 100% @sweexordious lol) for iterating and shipping so fast. Not only are we shipping a working version of the qgb, but one that has altered the original design dramatically to produce something significantly better.

Really big hats off, @sweexordious. Top shelf work.

docs/architecture/ADR-003-QGB-DataCommitment.md

docs/architecture/ADR-002-QGB-ValSet.md

go.mod

proto/qgb/genesis.proto

scripts/protocgen.sh

evan-forbes · 2022-09-20T20:05:53Z

scripts/single-node.sh

-celestia-appd gentx validator 5000000000utia --keyring-backend="test" --chain-id $CHAINID
+celestia-appd gentx validator 5000000000utia \
+  --keyring-backend="test" \
+  --chain-id $CHAINID \
+  --orchestrator-address $(celestia-appd keys show validator -a --keyring-backend="test") \
+  #da6ed55cb2894ac2c9c10209c09de8e8b9d109b910338d5bf3d747a7e1fc9eb9
+  --ethereum-address 0x966e6f22781EF6a6A82BBB4DB3df8E225DfD9488
+


@YazzyYaz just an fyi that this change requires validators to add two extra fields when they create a validator. This differs from the normal cosmos-sdk, so if we have docs for creating a validator, we will need to update them.

also @sweexordious we also want to make a note of this somewhere to make sure that we document this in our own docs.

We should decide on which docs do we want to write for now. Do we want to wait until the orchestrator/relayer are ready and document everything? or we want to update existing validator docs?

@evan-forbes does this apply to Mamaki users or the new Arabica testnet?

does this apply to Mamaki users or the new Arabica testnet?

this will apply to whichever chains will be running using v0.7.0 of celestia-app.

OK cc @sysrex @Bidon15 just a heads up for when version 0.7.0 of celestia-app is released, we will need to update the configs on running Arabica.

celestiaorg/docs#222

Bumping our discussion,
What exact rc candidate of 0.7.0 do we want to deploy in arabica for celestia-app?

depends on celestiaorg/celestia-node#1147, but if we can't merge that, then I would recommend sticking to v0.7.0-rc-1

testutil/common.go

x/qgb/handler.go

x/qgb/types/validator.go

rootulp

I did a first pass review prior to Thursday's synchronous review. A lot of my feedback pertains to code comments / consistency / naming. They can be addressed in follow-up PRs if we want to minimize the changes in this PR.

docs/architecture/ADR-002-QGB-ValSet.md

go.mod

proto/qgb/query.proto

x/qgb/keeper/keeper_valset.go

x/qgb/keeper/query_valset.go

x/qgb/types/genesis.go

x/qgb/types/valset.go

x/qgb/types/valset_confirm.go

evan-forbes · 2022-09-20T21:13:03Z

if we don't want to create a cleanup issue @sweexordious with all the feedback here (sounds kinda tedious), we could create a different PR with all the feed back and squash it into this PR

rach-id · 2022-09-20T21:14:42Z

@evan-forbes Yes, you're right. I will apply the changes on a separate PR and merge it here.

…#749)

Co-authored-by: Rootul P <rootulp@gmail.com>

evan-forbes · 2022-09-23T00:57:00Z

if we can't figure out this silly CI bug tmrw morning, I'm calling it as this thing is stupid and weirdly frustrating. We have more important things to do atm lol

we can fix this CI after we ship incentivized testnet

…of cosmos-proto/protoc-gen-gocosmos during proto gen

rach-id · 2022-09-23T09:01:46Z

@evan-forbes this commit seems to fix it: 13a9db9
It updates tendermintdev/sdk-proto-gen to v0.7. For some reason, the latest is still pointing towards v0.6 which doesn't have go1.18 installed.

Also, it removes this

celestia-app/scripts/protocgen.sh

Line 11 in d53f66b

    
           go get github.com/regen-network/cosmos-proto/protoc-gen-gocosmos@latest 2>/dev/null

from protocgen.sh. Is there a reason why we have? it seems that it's already imported in the go.mod file.

evan-forbes · 2022-09-23T10:33:07Z

Is there a reason why we have? it seems that it's already imported in the go.mod file.

it does seem redundant, so I'm good with removing it.

great work debugging!

friendly reminder to double check that we merge and not squash 🙂

rach-id · 2022-09-23T11:11:35Z

Finally mergiiiiing 🎉 🎉
Thanks a lot for the reviews and for all the help provided 👍 👍

rach-id and others added 30 commits February 16, 2022 11:03

Adds remaining messages and queries to QGB skeleton scaffolding (#212)

7c0e453

* removes dummy field from data commitment * adds TODOs * adds comments * linter fixes * linter fixes * linter fixes * linter fixes * linter fixes * CI fixes * query rename

adds remaining valset confirm handler (#219)

e20e44d

Record validator set during EndBlock for QGB orchestrator (#232)

0171fff

* add valset and params * create new val set request if the voting power has changed enough at the end of the block * comment out broken query method we don't strictly need yet * delete some comments

Add qgb data commitment tests (#240)

eaed5db

* fixes data commitment keeper iterator * adds data commitment by commitment query * cosmetics * adds data commitment queries tests * cosmetics * adds data commitment message validation tests * adds commented msg data commitment confirm test

Adds msg server data commitment test for QGB module (#242)

8209e23

* adds handler test * adds events check for handler test

Adds simple indexing to qgb module (#246)

df46d12

* use index prefix for commitments query * adds test for indexed query * linter

Orchestrator and relayer client (#255)

2f3686c

* initial orchestrator client * update go mod and change the makefile so it stops breaking the ci * copy and paste the needed code from the keystore lib

Add valset request by nonce support for QGB (#342)

95f5208

* adds valset request by nonce * trigger CI * revert: trigger CI * update go.mod * Revert "update go.mod" This reverts commit 0007050.

QGB go mod tidy (#350)

e310505

* go mod tidy * go mod tidy

adds new relayer method (#355)

d8dd2e7

adds evm_client wait for transaction confirmation (#354)

1534631

Use uint64 instead of int64 in QGB types (#358)

d79b858

* uses the uint64 instead of int64 in right places * typo * cosmetics

indexes data commitment confirms by (end_block, begin_block, validato…

8edfc38

…r) (#359)

adds query data confirm by exact range (#367)

4fe09dc

Merge remote-tracking branch 'upstream/master' into qgb-integration

a73a8f6

# Conflicts: # .gitignore # Makefile # app/app.go # go.mod # go.sum

update go.mod

9e81420

use latest cosmos-sdk qgb branch

cf86c63

fix the tests to use a non zero ethereum address

fb271ad

rach-id marked this pull request as ready for review September 20, 2022 11:53

docs: update the QGB ADR titles

8ec4c68

evan-forbes reviewed Sep 20, 2022

View reviewed changes

rootulp reviewed Sep 20, 2022

View reviewed changes

YazzyYaz mentioned this pull request Sep 20, 2022

Update Validator docs for Celestia-App 0.7.0 with QGB configs celestiaorg/docs#222

Closed

This was referenced Sep 20, 2022

QGB use existing testing infrastructure #745

Closed

QGB Feedback on PR #742 #746

Merged

rach-id and others added 4 commits September 21, 2022 15:48

QGB Update valset and data commitment confirm ADRs to link to ADR-005 (…

3b4ff8e

…#749)

QGB Feedback on PR #742 (#746)

bfa216f

Co-authored-by: Rootul P <rootulp@gmail.com>

Qgb universal nonces ADR update (#748)

69c8d28

Co-authored-by: Rootul P <rootulp@gmail.com>

QGB nits + docs for the merge 2 (#753)

d94943d

Co-authored-by: Rootul P <rootulp@gmail.com>

evan-forbes mentioned this pull request Sep 21, 2022

Incentivized Testnet Tracking Issue #514

Closed

41 tasks

rach-id force-pushed the qgb-integration branch from 7ea840d to d94943d Compare September 22, 2022 09:19

chore: Go mod tidy on QGB branch (#767)

a8b1ad8

rach-id force-pushed the qgb-integration branch from d271574 to a8b1ad8 Compare September 22, 2022 19:38

update proto-gen docker image to a new one and remove manual install …

13a9db9

…of cosmos-proto/protoc-gen-gocosmos during proto gen

rach-id force-pushed the qgb-integration branch from 44861ef to 13a9db9 Compare September 23, 2022 08:59

evan-forbes approved these changes Sep 23, 2022

View reviewed changes

rach-id merged commit 53d688e into main Sep 23, 2022

rach-id deleted the qgb-integration branch September 23, 2022 11:18

rach-id restored the qgb-integration branch October 27, 2022 14:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Merge QGB feature branch into main #742

Merge QGB feature branch into main #742

rach-id commented Sep 20, 2022

codecov-commenter commented Sep 20, 2022 •

edited

evan-forbes commented Sep 20, 2022 •

edited

evan-forbes commented Sep 20, 2022

evan-forbes left a comment

evan-forbes Sep 20, 2022

rach-id Sep 20, 2022

YazzyYaz Sep 20, 2022

evan-forbes Sep 20, 2022

YazzyYaz Sep 20, 2022

YazzyYaz Sep 20, 2022

Bidon15 Sep 21, 2022

evan-forbes Sep 21, 2022

rootulp left a comment

evan-forbes commented Sep 20, 2022

rach-id commented Sep 20, 2022

evan-forbes commented Sep 23, 2022 •

edited

rach-id commented Sep 23, 2022

evan-forbes commented Sep 23, 2022 •

edited

rach-id commented Sep 23, 2022

Merge QGB feature branch into main #742

Merge QGB feature branch into main #742

Conversation

rach-id commented Sep 20, 2022

codecov-commenter commented Sep 20, 2022 • edited

Codecov Report

evan-forbes commented Sep 20, 2022 • edited

evan-forbes commented Sep 20, 2022

evan-forbes left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rootulp left a comment

Choose a reason for hiding this comment

evan-forbes commented Sep 20, 2022

rach-id commented Sep 20, 2022

evan-forbes commented Sep 23, 2022 • edited

rach-id commented Sep 23, 2022

evan-forbes commented Sep 23, 2022 • edited

rach-id commented Sep 23, 2022

codecov-commenter commented Sep 20, 2022 •

edited

evan-forbes commented Sep 20, 2022 •

edited

evan-forbes commented Sep 23, 2022 •

edited

evan-forbes commented Sep 23, 2022 •

edited