Design an automated update mechanism for the private repository jenkins-infra/terraform-states

[dduportal]

Service(s)

Other

Summary

The private repository jenkins-infra/terraform-states holds a set of terraform projects we use to:

• define the backend for shared states of each public terraform project
• eventually define the basic resources required to bootstrap the public terraform projects (technical users for instance)

This repository is operated manually as it serves for bootstrap and initial permission:

• Sensitive content which should be separated from the other projects (e.g. NOT running in infra.ci.jenkins.io)
• Requires human (or root accounts) permissions

However we (jenkins infra team) recently acknowledged that some kind of automation is needed for the following use cases:

• Credential expiration (such as Let's Encrypt Azure Service Principal passwords expired for cert.ci.jenkins.io and trusted.ci.jenkins.io #4043). Ref. [infra.ci.jenkins.io] Azure AD Password expired for terraform-production (jenkins-infra/azure) #4045
• Terraform module upgrades: Done manually today as an opportunistic task with a removal of the lock file (rm -f ./<project>/.terraform.lock.hcl) and Terraform Plan + Apply the result (with a careful review of eventual changes).
  • Automation could bump these modules using updatecli
  • BUT no automated check will (and should) be run as the credentials won't be (and should NOT be) available in automation, not mentioning the GPG-encrypted states.

=> The goal is to have PR automatically opened to let the team know there are tasks to be run (and don't forget these) to plan it properly during team meetings

Reproduction steps

No response