migrate storage from premium to standard for jenkins-infra, jenkins-weekly and jenkins-release

[smerle33]

Service(s)

infra.ci.jenkins.io, release.ci.jenkins.io, weekly.ci.jenkins.io

Summary

a checked with the metrics, standard ZRS hdd will be enough to handle the workload for those 3 controllers.
lets try to save some money.

this will be the occasion to handle the volumes/disk with terraform and remove the Datasource annotation from the helmchart values for controllers.

We will need to create a new Storage Class (on publick8s and privatek8s).

Sidenote: we will have to handle the boostrap permissions from the terraform managed volumes.

Reproduction steps

No response

[smerle33]

WIP (infra and release)

[smerle33]

• Add a new storage class for standard ssd ZRS on private and public - feat(privatek8s/publick8s) add a standard ZRS storage class azure#672

WEEKLY.CI first

• take a snapshot for safety - jenkins-weekly-snapshot-20240515-0907
• add PV/PVC/DISK for weekly.ci with terraform (https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/persistent_volume and https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/persistent_volume_claim). only creating the PVC should be enough: https://kubernetes.io/docs/concepts/storage/persistent-volumes/#dynamic see migrate storage from premium to standard for jenkins-infra, jenkins-weekly and jenkins-release #4044 (comment)
• create a temporary pod with both the new PVC as RW and the Weekly PVC as ReadOnly
• do a timed rsync test between disk

when time ok:

• take a new snapshot for security (remove the one done above)
• disable kubernetes-management builds
• remove the weekly statefulset
• on the temporary pod re-run the rsync (should be fast)
• change the helm values for weekly to use the new PVC : persistence.existingClaim to PVC_NAME
• enable kubernetes-management builds
• merge the PR (changing the chart)
• check that the pod start with the correct PV/PVC
• remove the OLD PV/PVC/DISK (keep the snapshot for a few days)
• remove the snapshot

if all went well, redo for infra.ci/release.ci

[dduportal]

Update:

• New storage classes added on the 2 clusters in feat(privatek8s/publick8s) add a standard ZRS storage class azure#672
• weekly.ci planned to be migrated next week (Monday 29/Tuesday 30)

[dduportal]

Update: on hold until after the 15th of May 2024

[smerle33]

The aim is to be able to change the disk type without recreating everything next time. We choose to create PV/PVC/Disk from terraform instead of just the PVC for that (not following only creating the PVC should be enough: kubernetes.io/docs/concepts/storage/persistent-volumes/#dynamic)
The disk size can be changed on both scenarii.