CI Periodic jobs not testing recent kops builds

[rifelpet]

/kind bug

There's multiple issues with the kops pipeline that allow periodic jobs to pick up new kops builds.

Observe these two periodic jobs from weeks apart:

https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/e2e-kops-aws-arm64-ci/1787557790076637184
https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/e2e-kops-aws-arm64-ci/1781518104833036288

Both download the same CI version marker:

I0420 03:02:46.041205   13640 http.go:37] curl https://storage.googleapis.com/kops-ci/bin/latest-ci-updown-green.txt
I0420 03:02:46.117350   13640 http.go:37] curl https://storage.googleapis.com/kops-ci/bin/1.29.0-alpha.4+v1.29.0-alpha.3-378-g22ce1230da/linux/amd64/kops

I0506 19:02:14.140986   13795 http.go:37] curl https://storage.googleapis.com/kops-ci/bin/latest-ci-updown-green.txt
I0506 19:02:14.242580   13795 http.go:37] curl https://storage.googleapis.com/kops-ci/bin/1.29.0-alpha.4+v1.29.0-alpha.3-378-g22ce1230da/linux/amd64/kops

The marker itself hasn't been modified since March

curl -v https://storage.googleapis.com/kops-ci/bin/latest-ci-updown-green.txt 2>&1 | grep last-modified
< last-modified: Thu, 07 Mar 2024 19:28:59 GMT

The updown job that normally updates this has been failing for as far back as testgrid history shows:

https://testgrid.k8s.io/kops-latest#kops-aws-updown

The postsubmit job has also been failing for a few weeks for all merges on the master branch (release branches are working)

https://testgrid.k8s.io/sig-cluster-lifecycle-kops#kops-postsubmit-push-to-staging

We should fix this and keep an eye on the impact to periodic jobs before cutting the new releases.

[rifelpet]

This has been fixed
/close

[k8s-ci-robot]

@rifelpet: Closing this issue.

In response to this:

This has been fixed
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.