-
Notifications
You must be signed in to change notification settings - Fork 593
Bazel remote caching disaster recovery plan
Lars Kreutzer edited this page Jan 4, 2023
·
5 revisions
-
Option 1: Delete the entire cache (requires magma AWS access)
- Step 1: Update the service to have desired task count zero. Wait for running tasks to stop.
- Step 2: Manually empty the s3 bucket prefixed with "bazel-remote-cache-".
- Step 3: Update the service to have desired task count one.
-
Option 2: Delete and redeploy the entire remote caching setup (requires magma AWS and ci-infra repo access)
- Step 1: Tear down the bazel-remote infrastructure by running
terraform destroy -target=aws_s3_bucket.S3CacheBucket
(theforce_destroy = true
options needs to have been enabled in the terraform code). - Step 2: Redeploy with
terraform init
andterraform apply
.
- Step 1: Tear down the bazel-remote infrastructure by running
-
Option 3: Invalidate the cache keys (requires CI codeowner approval)
- With bazel-remote this can be done by changing the
--remote_cache
URL e.g. fromhttps://user:pw@url:9090/current-cache
tohttps://user:pw@url:9090/new-cache
. - This needs to be changed in all affected workflows and might require rebasing.
- With bazel-remote this can be done by changing the
-
Option 1: Delete and redeploy the entire remote caching setup (requires magma AWS and ci-infra repo access)
- Tear down the bazel-remote infrastructure by running
terraform destroy -target=aws_s3_bucket.S3CacheBucket
(theforce_destroy = true
options needs to have been enabled in the terraform code). Then redeploy withterraform init
andterraform apply
.
- Tear down the bazel-remote infrastructure by running
- Option 2: Re-implement the GH caches (requires CI codeowner approval).
Magma Website • Docs • Code • Contributing • Wiki