Zeebe can backup its data to an external storage without downtime and restore from it

[deepthidevaki]

As a user (operator), I can take backup of a zeebe cluster to and external storage and restore from a backup.

We will implement ZEP: Take backup of zeebe cluster without downtime.

Goal:

• Zeebe will expose an API by which an operator can take backups and manage backups.
• Zeebe will provide a way to restore from a given backup

Addition info

• [EPIC] A technical concept on how to take backup of a Camunda Platform 8 cluster without downtime #9103
• ZEP: Take backup of zeebe cluster without downtime.
• Related to https://github.com/camunda/product-hub/issues/104

Task breakdown

To discuss and decide

• Decide gateway endpoint and API
• Decide which type of storage to support for backup

StreamProcessor

• Allow adding non-engine records in logstream #9612
• Allow adding processors and event appliers for non-engine records #9613
• Add records for checkpoint #9719
• Add processor and event applier for checkpoint record #9720

Snapshotting

• Enable snapshot store to lock snapshot to prevent it from deleting #9626
• Prevent compaction when backup is taken #9627
• Abort snapshot concurrent to a backup #9906
• Add last written position as metadata in snapshot #10115
• Allow retaining more than one snapshot #10206

Backup management

• Define interface for backup manager #9806
• Define interface for BackupStore #9978
• Implement backup manager
  • Backup manager can take a backup #9979
  • Backup manager can get the status of a backup #9981
  • On leader change, new backup manager marks previous ongoing backup as failed #10189
  • Install backup manager with configured backup store #10207
• Implement S3 backup store #10229
  • Backup store can be configured #10220
  • Backup store can save a backup
    • Backup store can save backup metadata #10085
    • Backup store can save snapshot files #10110
    • Backup store can save segment files #10125
  • Backup store can delete backup #10159
  • Backup store can restore backup #10179
  • Backup store can set backup status #10134
  • Backup store can query backup status #10146
  • S3 backup store tests #10184
  • S3 backup store is observable #10218
  • S3 backup store documentation #10219
  • Broker version is part of the backup descriptor #10235
  • Backup status contains timestamps #10238
• Backup manager can delete backup #10208

Communication

• Define communication interface between gateway and broker for backup commands
• Broker can receive checkpoint create command from the gateway #9726
• Broker can receive backup status query #9807
• Broker can receive delete backup command #9808
• Refactor inter-partition communication to use a common sender and receiver #9625
• Update inter-partition message format to allow embedding checkpointId #9721
• Remote command receiver writes checkpoint create record #9809
• Send current checkpointId with all remote commands #9810

Broker

• A broker can take backup #10262
• Users can configure backup storage #10264
• Add a featureFlag to disable support for backup #10369

Gateway

• Add Pause exporting endpoint in gateway #9630
• Add Resume exporting endpoint in gateway #9633
• Implement take backup api in gateway #9901
• Implement monitor backup api in gateway #9902
• Implement delete backup api in gateway #10209

Restore

• Zeebe broker can restore from a backup #10263

Test

• Add integration test for backup and restore #10387
• Write tests for different concurrency scenarios
  • Simulate different concurrency scenarios to take backup. Verify that after restore the inconsistent states due to deployment or message correlation do not exist.
• Test failure scenarios
  • Test scenarios where a leader change/restart happens while taking backup.

Documentation

• Docs for pause/resume exporting operation
• Docs for backup apis

Metrics

• Add metrics for checkpoint processor #10385
• Add metrics for BackupManager #10386

Out of scope

• Providing a client for the backup api
• Supporting multiple types of backup storage

[deepthidevaki]

As we are nearing code freeze for 8.1.0 release, we have left some issues for the next quarter. Following are some issues/improvements that we would not be able to get it in 8.1.0. But we should tackle them sooner than later.

• If the given ID refers to a failed backup (that we know of), take backup command should return an error already, so the user knows to retry with a higher ID. To allow this we may have to let the Backup manager send response for take backup request instead of StreamProcessor. This will let us add more info to the take backup response - such as backup status if it already exists etc.
• Add more details to the existing take backup and get status api responses. The openApi spec describes our desired api.
• Implement api to list all backups
• Implement delete backup api in gateway #10209
• Change restore process (to be discussed).
  • Add api to restore and monitor progress of restore in gateway
  • Users can restore the cluster from a specific backup by
    1. Start zeebe cluster with a clean state
    2. List all completed backups => choose which backup to restore from
    3. send POST restore/backupId to gateway
    4. Monitor progress of restore => failed | inprogress | completed
• Make PartitionStatus actuator available in gateway. This will be useful to monitor status of pause/resume exporting
• Allow retaining more than one snapshot #10206
• Add grafana panels for the backup metrics. feat: add backup metrics to grafana dashboard #10521

[oleschoenburg]

Future Improvement to the S3 backup store implementation:

• Transparent backup compression
• Verify checksums to ensure that backups are not modified or corrupted
• Restructure object layout for more efficient listing of backups
• Ensure compatibility with advanced S3 features such as object locking and versioning

[deepthidevaki]

Closing this. A new issue will be created to keep track of the pending tasks and improvements.