ABCI++ vote extensions & FinalizeBlock

[sergio-mena]

This is the continuation of the work started in #9053: backport ABCI++ implementation, which lives in the v0.36.x branch.
Once this tracking issue is completed, the whole ABCI++ interface should be ready for QA process, and then release.

Proposed Path to Done. Summary

The main idea behind this plan is to proceed in the same way as #9053.
The main approach is:

• We will "follow" our work on v0.36.x. By "following a PR", I mean looking at that PR (similar work done for v0.36.x) and copying over only what makes sense. "git cherry-pick" turned out to be our best friend for "following" PRs in ABCI++ Prepare/Process Proposal #9053
• We will be considering only commits related to vote extensions and FinalizeBlock
• We will be skipping PrepareProposal/ProcessProposal related commits (as they are already in main via ABCI++ Prepare/Process Proposal #9053)

The work is structured very similarly to #9053. It consists of two main parts (further described below):

• (1) Preliminary work [Estimation of critical path: XXX days (with 1 thread)]
• (2) Core feature work [Estimation of critical path: XXX days (with 5 threads)]

(1) and (2) can mostly proceed in parallel.

protobufs will be managed as in v0.37.x → They will evolve with the code

Proposed Path to Done. Details

(1) Preliminary Work

We could have two threads here, but as all these tasks serialized are shorter than the critical path in "Core Feature Work", there's no point

• Create feature branch: feature/abci++vef [Estimation: 0 days]
• Restore CI on feature/abci++vef branch [Estimation: 0.5 days]
• e2e tests: port any improvements not included in the core work section (git diff HEAD..v0.36.x -- test/e2e) [Estimation: 2 days] (tracking: e2e: add contexts and e2e generator tests #9426)

(2) Core feature work

We have three threads that can proceed in parallel:

• Vote extensions (*) [Estimation: XXX days]
• FinalizeBlock (*) [Estimation: XXX days]
• Spec and doc work [Estimation: XXX days]

(*) Only when vote extensions and FinalizeBlock are done, can we proceed with the fourth part:

• Final adjustments [Estimation: XXX days, assuming no parallel work done]
  • tasks here could be further parallelized

Vote Extensions

• Old vote extensions
  • Follow abci: Vote Extension 1 #6646, then ABCI Vote Extension 2 #6885
  • Double-check integration of PrepareProposal -- vote extensions
    • Check if there is something missing from abci: PrepareProposal-VoteExtension integration [2nd try] #7821 (or PrepareProposal-VoteExtension integration #6915)
• Sync vote extensions with Spec #9835
  • First cherry-pick Vote extensions: new design #8031 (ignore spec part),
  • Main part: follow abci++: Sync implementation and spec for vote extensions #8141
• Vote extension cleanup #9846
  • follow abci++: Vote extension cleanup #8402
• Vote extension propagation Vote extension propagation #9852
  • follow abci++: Propagate vote extensions (RFC 017) #8433
• Vote extensions: Parameter activation logic #9861
  • Add activation logic: follow abci++: add consensus parameter logic to control vote extension require height #8547
• Vote extensions: Add parameter in proto #9864
  • follow abci++: add proto fields for enabling vote extensions #8587
• Integrate divergences between cherry-picked implementation and manual work in #9620 #9887

FinalizeBlock

• abci: port across finalize block from v0.36 #9427 [Estimation: 4 days]
  • Follow Completed the existing FinalizeBlock PR and rebased to master #7798 (it's bulky, and might depend on other infra PRs, if yes cherry-pick them first)
• Sync FinalizeBlock with spec
  • Description: follow abci: Synchronize FinalizeBlock with the updated specification #7983

Final adjustments

Most of these tasks can be done in parallel, but they are quite short.

• ABCI 2.0: Final adjustments #9916
  • Double-check whether we need something from Fixing handling of contexts in the ABCI++ rebased branch #7768
  • Protobuf: responses should be enums: see what is missing from ABCI++: Update new protos to use enum instead of bool #8158 [Estimation: 0.5 days]
  • Block data exposed to App: follow abci++: only include meaningful header fields in data passed-through to application #8216 and take the parts still missing (mainly FinalizeBlock) [Estimation: 1 day]
  • Double-check if there is something we need from abci++: Remove intermediate protos #8414 (probably not)
  • See what metrics we are still missing from consensus: add additional metrics for abci++ data #8480 (mainly vote extensions)
  • Redundant params in FinalizeBlock, and obsolete methods (BeginBlock, etc). Double check that changes in Removed redundant params in FinalizeBlock #8598 and/or Remove obsolete abci methods, no longer called by ABCI++ Tendermint #8633 are unnecessary
  • Move app_hash from Commit to FinalizeBlock: follow abci: Move app_hash parameter from Commit to FinalizeBlock #8664
  • Compare the status of abci's types.proto in feature/abci++vef and v0.36.x and make sure no changes are related to vote extensions and FinalizeBlock
  • Crash in all cases of undefined app behavior #8206
  • Update description of BlockSync p2p messages in spec #9002
  • Once we are code complete, and all CI tests pass, merge branch back into main

Spec and doc work

Low risk, can proceed completely in parallel with the rest

• ABCI 2.0: Spec work #9917
  • RFC-017: add section on vote extension upgrade path #8484
  • Turn RFC017 into part of the spec
  • Uncomment all parts related to vote extensions, and FinalizeBlock from the spec.
  • Re-merge the spec as of v0.36.x into feature/abci++vef and go through the conflicts (goal: make sure we didn't forget anything)
  • Follow Restore Commit to the ABCI++ spec, and other late modifications #8796 and make sure the changes are present in the spec
    • Restore data field in Commit
    • Rename Misbehaviour to ByzantineValidators (done already?)
  • Final read of spec for overall coherence
• ABCI 2.0: Docs and tutorials #9918
  • See if Revisit all apps/examples and decide how to _demonstrate_ ABCI++ #7660 is still relevant
  • getting-started.md [small, 0.5 days]
  • indexing-transactions.md [small, 0.5 days]
  • abci-cli.md (review only) [small, 0.5 days]
  • what-is-tendermint.md (ABCI section) [small, 0.5 days]
  • go-built-in.md and go.md (same changes) [medium, 2 days]
  • new go-abci-plus-plus tutorial to demonstrate advanced ABCI++ usage [medium 3 days]
    • try to extend the one demonstrating PrepareProposal/ProcessProposal