v0.37 release plan

[thanethomson]

This issue currently targets the v0.37 release.

In order to ship #9053, we need to do the following:

• Migrate master to main #9096
• Update documentation, including the roadmap
• Define a release QA plan
  • Define an integration testing plan with the Cosmos SDK (cc @marbar3778)
• ABCI++ Prepare/Process Proposal #9053
• Additionally, we should aim to include as many of the addressed Q3 priorities in this release as possible:
  • Prioritized mempool follow-ups #9077
  • P2P Stability #9055
  • Short-term data storage improvements #9054
  • Short-term monitoring/logging improvements #9076
  • Release process improvements #9078
• Identify and forward-port any necessary bug/security fixes from v0.35/v0.36 to v0.37
  • abci: Port message length delimiter changes to main #9176
  • abci: remove setOption  #5447
  • blockchain: rename to blocksync service #6755
  • blockchain/v1: remove in favor of v2 #5728
  • proto: deduplicate consensus params #9287
  • Events get mangled when non-utf8 value in Value #6403
• Cut v0.37.x branch
• Cut release candidate
• docs: Enable build for v0.37.x branch #9429
• Execute on the release QA plan and capture the results
  • Capture v0.34 baseline metrics #9123
  • qa: Capture v0.37 pre-release metrics #9333
  • Validate that there's no meaningful, substantial regression from v0.34 to v0.37
  • Execute on the QA plan for integration with the Cosmos SDK
• Cut final v0.37.0 release
  • docs: Ensure that the v0.37 version is available in the Vuepress config
• Prepare v0.37.1 release by merging all of these PRs

Things that still need to be done before we "feature-freeze" v0.37 and cut the v0.37.x branch:

• Merge the feature/abci++ppp branch into main once all relevant work from #9053 is done #9300
• abci++: simplify prepare proposal API #9269
• Forward port discard abci responses change #9286
• Temporarily revert remove lastresulthash from merklization in lastresult hash #9175
• Merge the feature/abci++ppp branch into main once all relevant work from #9053 is done #9300
  • docs: Update v0.33.x to prepare for v0.37 #9248
  • docs: Update v0.34.x to prepare for v0.37 #9244
  • docs: Update all docs to prepare for v0.37 #9243
• Rework of CHANGELOG_PENDING.md and UPGRADING.md to reflect the code merged from feature/abci++ppp #9304
• Events get mangled when non-utf8 value in Value #6403

[tac0turtle]

somethings that are missing

• crypto: ed25519 & sr25519 batch verification #6120
• cli: allow node operator to rollback last state #7033
• inspect: add inspect mode for debugging crashed tendermint node #6785
• fastsync: update the metrics during fast-sync #6590
• blockchain: rename to blocksync service #6755
• abci: remove setOption  #5447
• crypto: adopt zip215 ed25519 verification  #5632
• crypto: Use a different library for ed25519/sr25519 #6526
• blockchain/v1: remove in favor of v2 #5728

Not sure who is leading this endeavour but please look at the change log for 0.35 to identify things that are easy lifts to move to main/0.37 as many people are expecting them.

[tychoish]

I think the crypto changes are potentially pretty isolated (small), offer some amount of ease, and wouldn't be disruptive to the codebase. (#6120, #5632, #6526)

I think the blockchain/blocksync changes are good, and not very risky, (e.g. removing all of the versions of blocksync that don't work, and have as far as I know never worked, which is to say v1 and v2, and renaming it blocksync rather than blockchain in the code.) At the same time, I'm not sure that people run into this problem in practice (and it's easily recoverable when they do,) so they seem like good things to cut scope on. (#6755, #5728)

The reset are a bit more delicate and have less clear answers, but I've attempted to annotate:

• cli: allow node operator to rollback last state #7033 (medium/large, probably not worth doing, unless the corresponding application feature is ready. Unlikely to cause regressions in other areas of the code, but does require reaching deep into tendermint.)
• inspect: add inspect mode for debugging crashed tendermint node #6785 (medium/large, low risk relatively as the change is isolated, but RPC plumbing is enough different in 0.34, that it might be bigger this time.)
• fastsync: update the metrics during fast-sync #6590 (small, worth doing, and well aligned with our observability epic)
• abci: remove setOption  #5447 (small; worth doing but not a clean backport for protobuf compatibility reasons, it seems)

[tac0turtle]

Theses are some that I saw quickly but I bet there are other low lift things in the changelog that could be upstreamed to main

[cmwaters]

Other than the ones already mentioned, I think the following should be considered streaming in the near term:

• Should not unlock locked block when seeing +2/3 prevote for nil #6849
• rpc: implement header and header_by_hash queries #7270
• Performance improvements for the event query API #7338
• Refactor Event Subscription #6634

The rollback command (#7033) has already been backported to v0.34

I think it would be also nice to add the confix tool that Michael wrote

[williambanfield]

@marbar3778 I'm confused by #5728, I thought v1 ended up being the only reactor, v2, and that that v2 should be targeted for removal.

[tac0turtle]

sorry meant the removal of v1 and v2. grabbed that pr as an example

[hdevalence]

Tendermint 0.35 deduplicated some proto definitions between abci and types, notably the ConsensusParams and the BlockParams. Not including this deduplication in 0.37 will make modeling domain types more difficult.

[hdevalence]

At some point between Tendermint 0.34 and 0.35, the SetOption ABCI request was dropped. I believe it was unused; it's not documented anywhere.

[cmwaters]

At some point between Tendermint 0.34 and 0.35, the SetOption ABCI request was dropped. I believe it was unused; it's not documented anywhere.

v0.37 also drops SetOption - yes it's unused.

[cmwaters]

As to the deduplication of some proto definitions we can and should do that. I have opened a PR here: #9287

[ashishkhuraishy]

will the documentation updates also cover fixes to update broken links from the tendermint website?
eg: this is the link which is currently there on Get Started button which redirects to a 404