Cosmovisor improvements

[robert-zaremba]

Cosmovisor is the tool to run Cosmos blockchains. It integrates with the x/upgrade module to organize binaries, optionally download new versions and switch to new version when needed without a downtime. However, it was designed with only one binary in mind: a Cosmos App. Some chains use more binaries (eg all cosmwasm chains require a specific version of wasmd). Updating a chain (cosmos App) may require upgrading app dependencies.

Handling more then one binary by cosmovisor is a breaking change for cosmovisor. Moreover, we will need to think if we want to use the same x/upgrade messages for upgrading a Cosmos App as well as support binaries (eg wasmd).

The desired outcome of this discussion is to spec the Cosmovisor v2 in #9869

[robert-zaremba]

cosmwasm thread.

According to #9869 (comment) we need:

• managing docker from cosmwasm
• allowing cosmwasm to swap some binaries

[aaronc]

I'm sorry where in that thread are you reading that cosmwasm needs support for multiple binaries. Is that true @ethanfrey ?

[robert-zaremba]

App may have other dependencies, and app update may require dependency update. Eg, cosmwasm relays on wasmd -> #9869 (comment)

[aaronc]

But we should confirm how high importance it is before spending time on it. It doesn't sound like cosmwasm requires separate binaries

[aaronc]

The stuff that is needed for cosmovisor v2 IMHO is:

• possibly some different behavior because of coordinated shutdown of tendermint, but this is mostly x/upgrade v2 first
• calling a pre-upgrade command on the binary

I haven't seen evidence that these other requirements are high priority yet.

[robert-zaremba]

Yes, pre-upgrade is definitely needed as well. In fact external binary swap can be implemented as a pre-upgrade.

[aaronc]

Maybe that's true... we need to learn more about how those setups work and how common they are. External tendermint is one example, but it may not be common

[iramiller]

From an operations perspective the cosmovisor process could be doing a lot more to facilitate running a blockchain. There is an opportunity for example to better handle coordination of the upgrade process as well as managing the state of each process according to the version of the binary.

Some of the major ideas that Provenance has been discussing internally from an operations needs assessment:

• Version Management
  • Provide a list of currently available binaries
  • Manage installation/download and issue warnings if this process fails (manual downloads/installs should indicate if they are properly installed and configured prior to an upgrade, errors encountered during download of an upgrade binary should be reported)
  • Isolate configuration appropriately between versions such that configuration can be updated/transistioned during an upgrade process.
• Operational Logging
  • Provide a separation between general operational log messages and the possibly verbose output of the blockchain process itself. Surface/generating a new log stream focused on operations while leaving the existing output in a separate log file.
  • Provide an option for access logs to capture API use
  • Warn when an upgrade is schedule before the upgrade height is reach, aka as soon as the plan is published.
  • Warn when an upgrade is schedule and there is not currently a properly configured/installed binary for the targeted plan
  • Provide information about sync status, warnings when peer count falls, etc
• Maintenance
  • An interface for creating node backup/managing existing backups/ensuring a backup is present and successful before an upgrade is attempted
• Context Management
  The cosmovisor process is also in a unique position to wrap and manage multiple configured environments/node binaries at once. For those who use Kubernetes this might work in a similar fashion to kubectl config use-context ctxname where the ctxname could be a local dev node, testnet, mainnet, various cosmos networks, etc. Benefits of this approach:
  • Simplified management of potentially many different client.toml against specific contexts
  • A command to be executed from any directory with a clearly defined context.
  • The correct binary can be invoked based on the environment being used for all commands
    • For example if there are two different forms for a Msg between versions a user making a cli based transaction would be routed to the correct binary for the current chain height/blockchain they are working with.
  • Managing all configuration through contexts would provide clean separation when multiple environments exist on a single machine. It could also provide a unified configuration interface that would be more approachable for users than the current app.toml/config.toml/client.toml structure that frequently results in simply greping a directory to try and find a piece of config an operator is looking for... not to mention mixing ENV with this.

Note: The scope of the above features is quite a bit larger than just cosmovisor but presenting all of these related ideas together will hopefully provide a bit of helpful context.

[robert-zaremba]

Nice feature list. However lot of work as well. I wonder if some of this tasks could be done by an existing, more general tool.

Warn when an upgrade is schedule before the upgrade height is reach, aka as soon as the plan is published.

I'm not sure if I get it correctly. Upgrade time is defined by a block height:

plan.Height <= ctx.BlockHeight()

[robert-zaremba]

Maybe some docker tool (docker-compose?) could be used to manage contexts and log separation?

[iramiller]

Warn when an upgrade is schedule before the upgrade height is reach, aka as soon as the plan is published.

I'm not sure if I get it correctly. Upgrade time is defined by a block height:

plan.Height <= ctx.BlockHeight()

This idea is derived from the classic 'How far away in time is that upgrade right now?' For an upgrade that is placed a week or so out in the future after the voting periods the estimated blockheight can have a pretty large skew in time as well as of course the typical timezone/localization headaches. This basically is an updating estimate of how far away the event is right now.

[aaronc]

@alexanderbez and I described a way for pre-upgrade to work from cosmovisor's perspective: #9692 (comment). This IMHO is the number 1 thing needed right now and will facilitate the Tendermint v0.35 keys migration as well as migrating app.toml.

[aaronc]

Added a new issue to track this: #9973