CI: 2023 08 18
Zack Galbreath edited this page Aug 18, 2023
·
1 revision
- Alec Scott
- Dan LaManna
- Jacob Nesbitt
- John Parent
- Massimiliano Culpo
- Mike VanDenburgh
- Ryan Krattiger
- Scott Wittenburg
- Todd Gamblin
- Zack Galbreath
- We updated gitlab.spack.io to v16.2.4. This went even more smoothly than previous upgrades due to our recent switch from MinIO to S3.
- We noticed that Spackbot's worker was unresponsive. We ultimately tracked this down to an EC2 instance failing its system checks. After draining the node and restarting it, Spackbot was returned to service.
- We are making progress on the layout and design of our analytics database that will ingest job metrics from GitLab.
- Alec is also working on a database of utilization metrics per GitLab CI job. This data should allow us to dynamically define resource requests on a per-package basis.
- We are investigating the use of Sentry for better notification of error events
- We are experimenting with a couple different strategies for generating a list of specs to prune from our binary mirrors. So far these results look good! We are hoping to deploy an automatic pruning service in the coming weeks.
- We are using Ryan's new timing data feature to design an experiment comparing download speeds between S3 and CloudFront for both our AWS runners and those hosted at UO. We are hoping to collect meaningful data next week.
- John is making good progress on cross-compiling gpg for Windows. He & Max are going to work together to get these prebuilt binaries uploaded into the bootstrap mirror.
- Introduce a hard time limit of 12 hours on builder pods
- Setup persistent storage for Grafana so "starred" dashboards persist across pod restarts
- Continue investigating the use of CloudFront to speedup CI and public buildcache usage.
- Keep progressing towards the following goals:
- Timing Dashboards
- Gitlab CI runners for Windows
- Automatic buildcache pruning