Add the ability to automate and schedule backups #553

frouioui · 2024-05-13T19:59:57Z

Description

This Pull Request adds a new CRD called VitessBackupSchedule. Its main goal is to automate and schedule backups of Vitess, taking backups of the Vitess cluster at regular intervals based on a given cron schedule and Strategy. This new CRD is managed by the VitessCluster, like most other components of the vitess-operator, the VitessCluster controller is responsible for the whole lifecycle (creation, update, deletion) of the VitessBackupSchedule object in the cluster. Inside the VitessCluster it is possible to define several VitessBackupSchedules as a list, allowing for multiple concurrent backup schedules.

Among other things, the VitessBackupSchedule object is responsible for creating Kubernetes's Job at the desired time, based on the user-defined schedule. It also keeps track of older jobs and delete them if they are too old, according to user-defined parameters (successfulJobsHistoryLimit & failedJobsHistoryLimit). The jobs created by the VitessBackupSchedule object will use the vtctld Docker Image and will execute a shell command that is generated based on the user-defined strategies. The end user can define as many backup strategy per schedule, each of them mocks what vtctldclient is able to do, the Backup and BackupShard commands are available, a map of extra flags enable the user to give as many flag as they want to vtctldclient.

A new end-to-end test is added to our BuildKite pipeline as part of this Pull Request to test the proper behavior of this new CRD.

Related PRs

Documentation: Enhance the getting started guide: scheduled backups vitessio/website#1746
Vitessio/vitess: Update operator.yaml and add schedule backup example vitessio/vitess#15969

Demonstration

For this demonstration I have setup a Vitess cluster by following the steps in the getting started guide, until the very last step where we must apply the 306_down_shard_0.yaml file. My cluster is then composed of 2 keyspaces: customer with 2 shards, and commerce unsharded. I then modify the 306... yaml file to contain the new backup schedule, as seen in the snippet right below. We want to create two schedules, one for each keyspace. The keyspace customer will have two backup strategies: one for each shard.

apiVersion: planetscale.com/v2
kind: VitessCluster
metadata:
  name: example
spec:
  backup:
    engine: xtrabackup
    locations:
      - volume:
          hostPath:
            path: /backup
            type: Directory
    schedules:
      - name: "every-minute-customer"
        schedule: "* * * * *"
        resources:
          requests:
            cpu: 100m
            memory: 1024Mi
          limits:
            memory: 1024Mi
        successfulJobsHistoryLimit: 2
        failedJobsHistoryLimit: 3
        strategies:
          - name: BackupShard
            keyspaceShard: "customer/-80"
          - name: BackupShard
            keyspaceShard: "customer/80-"
      - name: "every-minute-commerce"
        schedule: "* * * * *"
        resources:
          requests:
            cpu: 100m
            memory: 1024Mi
          limits:
            memory: 1024Mi
        successfulJobsHistoryLimit: 2
        failedJobsHistoryLimit: 3
        strategies:
          - name: BackupShard
            keyspaceShard: "commerce/-"
  images:

Once the cluster is stable, all tablets are serving and ready, I re-apply my yaml file with the backup configuration:

$ kubectl apply -f test/endtoend/operator/306_down_shard_0.yaml 
vitesscluster.planetscale.com/example configured

Immidiately I can check that the new VitessBackupSchedule objects have been created.

$ kubectl get VitessBackupSchedule 
NAME                                          AGE
example-vbsc-every-minute-commerce-ac6ff735   7s
example-vbsc-every-minute-customer-8aaaa771   7s

Now I want to check the pods where the jobs created by VitessBackupSchedule are running. After about 2 minutes, we can see four pods, two for each schedule. The pods are marked as Completed as they finished their job.

$ kubectl get pods
NAME                                                           READY   STATUS             RESTARTS        AGE
...
example-vbsc-every-minute-commerce-ac6ff735-1715897700-nkfzx   0/1     Completed          0              79s
example-vbsc-every-minute-commerce-ac6ff735-1715897760-qr4hp   0/1     Completed          0              19s
example-vbsc-every-minute-customer-8aaaa771-1715897700-rbsmd   0/1     Completed          0              79s
example-vbsc-every-minute-customer-8aaaa771-1715897760-kzn8t   0/1     Completed          0              19s
...

Now let's check our backup:

$ ls -l vtdataroot/backup/example/commerce/- vtdataroot/backup/example/customer/80- vtdataroot/backup/example/customer/-80 

vtdataroot/backup/example/commerce/-:
total 0
drwxr-xr-x  11 florentpoinsard  staff  352 May 16 16:15 2024-05-16.221502.zone1-0790125915
drwxr-xr-x  11 florentpoinsard  staff  352 May 16 16:16 2024-05-16.221602.zone1-0790125915

vtdataroot/backup/example/customer/-80:
total 0
drwxr-xr-x  11 florentpoinsard  staff  352 May 16 16:15 2024-05-16.221502.zone1-2289928654
drwxr-xr-x  11 florentpoinsard  staff  352 May 16 16:16 2024-05-16.221601.zone1-2289928654

vtdataroot/backup/example/customer/80-:
total 0
drwxr-xr-x  11 florentpoinsard  staff  352 May 16 16:15 2024-05-16.221511.zone1-4277914223
drwxr-xr-x  10 florentpoinsard  staff  320 May 16 16:16 2024-05-16.221609.zone1-2298643297

$ kubectl get vtb --no-headers
example-commerce-x-x-20240516-221502-2f185d5b-1854be28    2m7s
example-commerce-x-x-20240516-221602-2f185d5b-0a248174    67s
example-customer-80-x-20240516-221511-fefbca6f-8ede9c7d   2m7s
example-customer-80-x-20240516-221609-89028361-d9d1c1e4   67s
example-customer-x-80-20240516-221502-887d89ce-2fc618f4   2m7s
example-customer-x-80-20240516-221601-887d89ce-5b5b0acb   66s

Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

GuptaManan100

Rest LGTM

GuptaManan100 · 2024-05-20T08:47:06Z

test/endtoend/operator/operator-latest.yaml

---
-apiVersion: scheduling.k8s.io/v1
-description: The vitess-operator control plane.
-globalDefault: false
-kind: PriorityClass
-metadata:
-  name: vitess-operator-control-plane
-value: 5000
---
-apiVersion: scheduling.k8s.io/v1
-description: Vitess components (vttablet, vtgate, vtctld, etcd)
-globalDefault: false
-kind: PriorityClass
-metadata:
-  name: vitess
-value: 1000


I don't think we want to delete these?

We are not removing this, just moving it earlier in the file. I am using the raw output of kustomize, which I think is what we should do to avoid conflict in the future.

GuptaManan100 · 2024-05-20T08:47:21Z

test/endtoend/operator/operator.yaml

---
-apiVersion: scheduling.k8s.io/v1
-description: The vitess-operator control plane.
-globalDefault: false
-kind: PriorityClass
-metadata:
-  name: vitess-operator-control-plane
-value: 5000
---
-apiVersion: scheduling.k8s.io/v1
-description: Vitess components (vttablet, vtgate, vtctld, etcd)
-globalDefault: false
-kind: PriorityClass
-metadata:
-  name: vitess
-value: 1000


Same for this

Same as #553 (comment)

GuptaManan100 · 2024-05-20T09:10:09Z

pkg/controller/vitessbackupschedule/vitessbackupschedule_controller.go

+var watchResources = []client.Object{
+	&kbatch.Job{},
+}


This variable is only being used once, that too in a for loop when it only has 1 value. Why don't we unfurl that value and use that directly

I have just re-used the same pattern we use throughout the codebase. In other places, even if there is a single element, we use it this way. Making it easy to understand all the resources created by a given controller when reading the top of the file.

Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

pkg/apis/planetscale/v2/vitessbackupschedule_types.go

maxenglander · 2024-05-20T17:25:05Z

pkg/controller/vitessbackupschedule/vitessbackupschedule_controller.go

+	return job, nil
+}
+
+func (r *ReconcileVitessBackupsSchedule) createJobPod(ctx context.Context, vbsc *planetscalev2.VitessBackupSchedule, name string) (pod corev1.PodSpec, err error) {


have we taking the vtctld action timeout into account?

I do not think so, what are you referring to?

vtctldclient has a default command timeout of 1 hour, which is controlled by the --action-timeout flag. We will almost certainly want to increase that.

or make it a flag, like extraFlags, on the backup schedule

I think this is solved as people can set --action-timeout by adding it to the extraFlags field of VitessBackupScheduleStrategy. The flags in set on extraFlags will be passed down to vtctldclient.

might be worth adding a note about that in release notes. i expect it will be a common issue people run in to.

fixed via 9739a94

maxenglander · 2024-05-20T17:34:44Z

pkg/apis/planetscale/v2/vitessbackupschedule_types.go

+	// This field will be ignored if we have picked the strategy BackupShard.
+	// +optional
+	// +kubebuilder:example="zone1-0000000102"
+	TabletAlias string `json:"tabletAlias"`


i kinda question the value of cron-scheduled Backup:

it's not uncommon to delete a PVC/Pod because it's in bad shape, and having to update backup schedules anytime that happens will be annoying.

a image/config rollout may switch a tablet from replica to primary, which means the next time a backup schedule runs it will take the primary offline.

would say it's not worth adding this strategy with these two potential footguns, unless it is a hotly requested feature.

That's a good point. I think it is safe to drop this strategy especially if I add what you are mentioning in #553 (comment). Having two strategies: BackupShard and BackupCluster, would give enough flexibility for most use cases. I am thinking we can even add a third BackupKeyspace strategy, which is a good in-between solution between BackupShard and BackupCluster.

Sounds good. BackupShard is a higher level request that can be intuitively and correctly executed over time as the cluster state changes.

See comment #553 (comment), which explains the new strategies.

maxenglander · 2024-05-20T17:36:26Z

pkg/apis/planetscale/v2/vitessbackupschedule_types.go

+	// This field will be ignored if we have picked the strategy BackupTablet.
+	// +optional
+	// +kubebuilder:example="commerce/-"
+	KeyspaceShard string `json:"keyspaceShard,omitempty"`


If a shard only has 1 replica, then the BackupShard strategy will mean all @replica queries will fail. may be worth having an option like MinHealthyReplicas for that strategy.

The @replica will fail because the tablet will not be able to respond query will being backed up? In that case, yeah it would make sense to add such parameter for the user to configure, with a default to 2 for instance.

Good point. At least when not using online backup methods like xtrabackup.

mattlord

This is looking good! I only had some relatively minor comments. I'll come back and do another pass once you've made whatever changes you like based on the comments from myself and Max.

Thanks!

.gitignore

.buildkite/pipeline.yml

deploy/role.yaml

mattlord · 2024-05-21T16:20:11Z

docs/api/index.html

+<p>Strategy defines how we are going to take a backup.
+If you want to take several backups within the same schedule you can add more items
+to the Strategy list. Each VitessBackupScheduleStrategy will be executed by the same
+kubernetes job. This is useful if for instance you have one schedule, and you want to
+take a backup of all shards in a keyspace and don&rsquo;t want to re-create a second schedule.
+All the VitessBackupScheduleStrategy are concatenated into a single shell command that
+is executed when the Job&rsquo;s container starts.</p>


I'm not sure what this means / looks like. An example somewhere would be nice IMO.

There is already an example in the CRD:

// +kubebuilder:example="-" // +kubebuilder:example="commerce"

In YAML it looks like that, which is what people can find in the examples/tests of the repo:

strategies: - name: BackupShard keyspace: "commerce" shard: "-"

docs/api/index.html

test/endtoend/backup_schedule_test.sh

test/endtoend/operator/101_initial_cluster_backup_schedule.yaml

maxenglander

another thought, might be nice to give users a way to assign annotations, and one or more affinity selection options to the backup runner pods. that way they can influence things scheduling and eviction.

for example, users might not want backup runner pods running on the same nodes as vttablet pods. and they might not want the backup runner pods to get evicted by an unrelated pod after they've been running for a long time.

Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

frouioui · 2024-05-24T20:34:29Z

In commit bc74ab4, I have applied one of the most important suggestion discussed above which is to remove the BackupTablet strategy in favor of BackupKeyspace and BackupCluster. The strategies can be used as follows:

# BackupKeyspace
        strategies:
          - name: BackupKeyspace
            cluster: "example"
            keyspace: "customer"

# BackupCluster
        strategies:
          - name: BackupCluster
            cluster: "example"

Meanwhile, the BackupShard strategy does not change. When ran we can see the following command line argument in the job's pod, which gets executed upon creation of the container:

# BackupKeyspace
Args:
      /bin/sh
      -c
      /vt/bin/vtctldclient --server=example-vtctld-625ee430:15999 BackupShard customer/-80 && /vt/bin/vtctldclient --server=example-vtctld-625ee430:15999 BackupShard customer/80-

# BackupCluster
Args:
      /bin/sh
      -c
      /vt/bin/vtctldclient --server=example-vtctld-625ee430:15999 BackupShard commerce/- && /vt/bin/vtctldclient --server=example-vtctld-625ee430:15999 BackupShard customer/-80 && /vt/bin/vtctldclient --server=example-vtctld-625ee430:15999 BackupShard customer/80-

cc @maxenglander @mattlord

maxenglander · 2024-05-28T19:52:45Z

pkg/apis/planetscale/v2/vitessbackupschedule_types.go

+
+	// Cluster defines on which cluster you want to take the backup.
+	// This field is mandatory regardless of the chosen strategy.
+	Cluster string `json:"cluster"`


i'm not sure i follow why this is necessary. my mental model is that a user defines []VitessBackupScheduleTemplate on the ClusterBackupSpec, and so implicitly each VitessBackupScheduleStrategy will be associated with the cluster where ClusterBackupSpec is defined.

That's a good point @maxenglander, it is pretty useless. I ended up removing that field from VitessBackupScheduleStrategy and adding it to VitessBackupScheduleSpec. The VitessCluster controller will come and fill that new field when it create a new VitessBackupSchedule object, that way VitessBackupSchedule is still be able to select existing components given their cluster names to avoid fetching wrong data in the event where we have multiple VitessCluster running in our K8S cluster.

See b30aa09 for the change.

Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

frouioui · 2024-05-30T18:29:12Z

another thought, might be nice to give users a way to assign annotations, and one or more affinity selection options to the backup runner pods. that way they can influence things scheduling and eviction.

for example, users might not want backup runner pods running on the same nodes as vttablet pods. and they might not want the backup runner pods to get evicted by an unrelated pod after they've been running for a long time.

In e6946fb I have added affinity and annotations in the VitessBackupScheduleTemplate, allowing the user to configure the affinity and annotations they want for their pods that take backups.

maxenglander · 2024-05-30T18:37:42Z

pkg/apis/planetscale/v2/labels.go

@@ -40,6 +40,8 @@ const (
 	TabletPoolNameLabel = LabelPrefix + "/" + "pool-name"
 	// TabletIndexLabel is the key for identifying the index of a Vitess tablet within its pool.
 	TabletIndexLabel = LabelPrefix + "/" + "tablet-index"
+	// BackupScheduleLabel is the key for identifying to which VitessBackupSchedule a Job belongs to.
+	BackupScheduleLabel = LabelPrefix + "/" + "backup-schedule"


Could this also be accomplished with an owner reference?

That might be a little neater, since if you delete a backup schedule, you'd probably also want to delete any currently running pods.

The VitessBackupSchedule is already the owner of its own jobs, which makes it an owner of the pods that are created by the jobs. When deleting a backup schedule all the associated pods and jobs are also removed.

$ kubectl get pods NAME READY STATUS RESTARTS AGE ... example-vbsc-every-minute-94d12263-1717100280-9cscg 0/1 ContainerCreating 0 27s ... $ kubectl get vitessbackupschedule NAME AGE example-vbsc-every-minute-94d12263 90s $ kubectl delete vitessbackupschedule example-vbsc-every-minute-94d12263 vitessbackupschedule.planetscale.com "example-vbsc-every-minute-94d12263" deleted $ kubectl get pods NAME READY STATUS RESTARTS AGE ... example-vbsc-every-minute-94d12263-1717100280-9cscg 1/1 Terminating 1 (31s ago) 67s ... $ kubectl get pods NAME READY STATUS RESTARTS AGE example-90089e05-vitessbackupstorage-subcontroller 1/1 Running 0 2m30s example-commerce-x-x-vtbackup-init-c6db73c9 0/1 Error 2 (26s ago) 2m26s example-commerce-x-x-zone1-vtorc-c13ef6ff-5855667dbc-d5dcl 1/1 Running 1 (73s ago) 2m27s example-etcd-faf13de3-1 1/1 Running 0 2m30s example-etcd-faf13de3-2 1/1 Running 0 2m30s example-etcd-faf13de3-3 1/1 Running 0 2m30s example-vttablet-zone1-0790125915-4e37d9d5 3/3 Running 0 2m27s example-vttablet-zone1-2469782763-bfadd780 3/3 Running 0 2m27s example-vttablet-zone1-2548885007-46a852d0 3/3 Running 0 2m27s example-zone1-vtctld-1d4dcad0-646b8c9c77-vl69s 1/1 Running 1 (73s ago) 2m29s example-zone1-vtgate-bc6cde92-569bbcf4df-lslph 1/1 Running 1 (72s ago) 2m27s vitess-operator-6f87b84f88-6wjlt 1/1 Running 0 2m33s

Could this also be accomplished with an owner reference?

I am not 100% sure of what you mean by that.

cool, this is what i meant by owner reference 👍

The VitessBackupSchedule is already the owner of its own jobs,

maxenglander · 2024-05-30T18:45:20Z

pkg/controller/vitessbackupschedule/vitessbackupschedule_controller.go

+			return err
+		}
+		if jobStartTime.Add(time.Minute * time.Duration(timeout)).Before(time.Now()) {
+			if err := r.client.Delete(ctx, job, client.PropagationPolicy(metav1.DeletePropagationBackground)); (err) != nil {


seems like a good thing to have a metric for

fixed via 46b6967 + 5809cbd

maxenglander · 2024-05-30T18:47:52Z

pkg/controller/vitessbackupschedule/vitessbackupschedule_controller.go

+	return job, nil
+}
+
+func (r *ReconcileVitessBackupsSchedule) createJobPod(ctx context.Context, vbsc *planetscalev2.VitessBackupSchedule, name string) (pod corev1.PodSpec, err error) {


might be worth adding a note about that in release notes. i expect it will be a common issue people run in to.

maxenglander · 2024-05-30T18:51:34Z

pkg/controller/vitessbackupschedule/vitessbackupschedule_controller.go

+					if shardIndex > 0 || ksIndex > 0 {
+						cmd.WriteString(" && ")
+					}
+					createVtctldClientCommand(&cmd, vtctldclientServerArg, strategy.ExtraFlags, ks.name, shard)


am i reading this right that it will be taking a backup of each keyspace and shard in sequence? that doesn't seem ideal to me because if each shard takes an hour to backup, and there are 32 shards, then the backup of the first shard and last shard will be more than a day apart.

i think it would be better if there were at least the option of BackupCluster and BackupKeyspace to backup all keyspaces and shards in parallel.

might be better to limit this PR to only support BackupShard for now, and add support for the other options after more consideration into how to implement BackupKeyspace and BackupCluster.

Let's do that, remove those two strategies as part of this PR and I will work on a subsequent PR to add them back with a better approach. This PR is getting lengthy already.

Fixed via 70ba063

IMO BackupAllShardsInKeyspace and BackupAllShardsInCluster are better names. It may seem nitty, but I think it's important as it reflects what it actually is: independent backups of the shards. i.e. it is NOT a single consistent backup of the keyspace or cluster at any physical or logical point in time.

I ended up removing Keyspace and Cluster strategies in this PR as it will require a bigger refactoring. I am keeping that in mind for when we add them though.

Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

frouioui added 20 commits May 8, 2024 10:30

Add VitessBackupSchedule CRD

7666e88

Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

Added the VitessBackupSchedule controller for the standalone CRD

51f9653

Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

Add VitessBackupSchedule to VitessCluster and clean up the CRD

eec5c02

Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

Clean up VitessCluster controller for VitessBackupSchedule

4586474

Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

Use proper Docker Image in VitessBackupSchedule and re-add fields

fd30010

Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

Add backup strategies and fix VitessCluster reconcile loop for vbsc

b6c0150

Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

Allow for deletion of VitessBackupSchedule

bfe82b0

Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

Enable backup for strategy 'backupShard'

2e4fd65

Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

Enable backup for strategy 'backupTablet'

33c23a0

Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

Refactor VitessBackupSchedule reconciling loop

49166e3

Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

List Jobs using a label filter

b961d35

Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

Support multiple schedules at the same time

9216b59

Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

Remove non-required logging

4177d3d

Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

Allow for multiple backup strategies per schedule

def8ac3

Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

Remove non-required logging

9b1ce77

Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

Merge remote-tracking branch 'origin/main' into scheduled-backups

dc2aa18

Self-review

062d7a0

Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

Re-generate CRDs and operator.yaml

d8cec2a

Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

Add timeout for stale jobs

3347bed

Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

Add E2E test for scheduled backups

cd77eb3

Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

frouioui changed the title ~~Add VitessBackupSchedule~~ VitessBackupSchedule add the ability to automate backups May 16, 2024

frouioui changed the title ~~VitessBackupSchedule add the ability to automate backups~~ Add the ability to automate and schedule backups May 16, 2024

frouioui added 3 commits May 16, 2024 16:23

Modify backup strategy to use a flag map

a129b4c

Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

Revert debug code

614f31d

Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

Fix the backup schedule test to expect 2 every-five-minute pods

95ff978

Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

This was referenced May 17, 2024

Enhance the getting started guide: scheduled backups vitessio/website#1746

Open

Update operator.yaml and add schedule backup example vitessio/vitess#15969

Open

Add default to podJobTimeout

ed10321

Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

frouioui marked this pull request as ready for review May 17, 2024 17:01

frouioui requested a review from mattlord May 17, 2024 17:03

GuptaManan100 approved these changes May 20, 2024

View reviewed changes

Fix clock skew issue

f45dc13

Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

maxenglander reviewed May 20, 2024

View reviewed changes

mattlord reviewed May 21, 2024

View reviewed changes

yields approved these changes May 23, 2024

View reviewed changes

maxenglander reviewed May 23, 2024

View reviewed changes

frouioui added 5 commits May 24, 2024 09:29

Merge remote-tracking branch 'origin/main' into scheduled-backups

bd70ae6

Apply review suggestions

e6e0e45

Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

Change CRD example to be every day instead of every minute

8cbfcad

Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

Revert errexit change in test script

6e20613

Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

Add BackupCluster and BackupKeyspace

bc74ab4

Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

maxenglander reviewed May 28, 2024

View reviewed changes

frouioui added 4 commits May 28, 2024 15:06

Remove Cluster field in Strategy

b30aa09

Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

Add release notes

4dc9ce0

Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

Fix 101 schedule test

9f7cd44

Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

Use env variable for the golang version

54c9512

Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

frouioui mentioned this pull request May 30, 2024

Release of v20.0.0-RC1 vitessio/vitess#16010

Open

30 tasks

Add custom annotations and affinity to backup pods

e6946fb

Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

frouioui requested review from mattlord and maxenglander May 30, 2024 18:29

maxenglander reviewed May 30, 2024

View reviewed changes

frouioui added 4 commits May 30, 2024 14:24

Add more information in the release notes about extraFlags

9739a94

Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

Fix metrics and add new metrics for timed out jobs

46b6967

Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

Revert unwanted change

5809cbd

Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

Revert BackupKeyspace and BackupCluster

70ba063

Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

frouioui requested a review from maxenglander May 30, 2024 21:24

frouioui added 2 commits May 30, 2024 15:25

Merge remote-tracking branch 'origin/main' into scheduled-backups

6dbd28a

Revert panic issue

f74592b

Signed-off-by: Florent Poinsard <florent.poinsard@outlook.fr>

Add the ability to automate and schedule backups #553

Are you sure you want to change the base?

Add the ability to automate and schedule backups #553

Conversation

frouioui commented May 13, 2024 • edited

Description

Related PRs

Demonstration

GuptaManan100 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

frouioui May 24, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

frouioui May 20, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mattlord left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

frouioui May 28, 2024 • edited

Choose a reason for hiding this comment

maxenglander left a comment • edited

Choose a reason for hiding this comment

frouioui commented May 24, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

frouioui commented May 30, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

frouioui May 30, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mattlord May 31, 2024 • edited

Choose a reason for hiding this comment

frouioui May 31, 2024 • edited

Choose a reason for hiding this comment

frouioui commented May 13, 2024 •

edited

frouioui May 24, 2024 •

edited

frouioui May 20, 2024 •

edited

frouioui May 28, 2024 •

edited

maxenglander left a comment •

edited

frouioui May 30, 2024 •

edited

mattlord May 31, 2024 •

edited

frouioui May 31, 2024 •

edited