Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Deprecate and migrate away from gs://kubernetes-release-dev and gcr.io/kubernetes-ci-images #2318

Closed
42 tasks done
spiffxp opened this issue Jul 9, 2021 · 9 comments
Closed
42 tasks done

Deprecate and migrate away from gs://kubernetes-release-dev and gcr.io/kubernetes-ci-images #2318

spiffxp opened this issue Jul 9, 2021 · 9 comments
Assignees
@spiffxp
Labels
area/artifacts Issues or PRs related to the hosting of release artifacts for subprojects area/prow Setting up or working with prow in general, prow.k8s.io, prow build clusters area/release-eng Issues or PRs related to the Release Engineering subproject kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. kind/deprecation Categorizes issue or PR as related to a feature/enhancement marked for deprecation. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. sig/release Categorizes an issue or PR as relevant to SIG Release. sig/testing Categorizes an issue or PR as relevant to SIG Testing.
Projects
sig-k8s-infra sig-testing issues
Milestone
v1.22

Comments

@spiffxp
Copy link
Member

spiffxp commented Jul 9, 2021 •
edited

Part of umbrella issue to migrate the kubernetes project away from use of GCP project google-containers: #1571

This issue covers the deprecation of and migration away from two google.com assets:

  • the google.com-owned GCS bucket gs://kubernetes-release-dev living in GCP project google-containers, in favor of the community-owned GCS bucket gs://k8s-release-dev living in GCP project k8s-release
  • the google.com-owned GCR repo gcr.io/kubernetes-ci-images, in favor of the community-owned GCR repo gcr.io/k8s-staging-ci-images

Step 1: Set and announce deprecation window

The first part is setting and announcing a deprecation window for CI artifacts placed in gs://kubernetes-release-dev, and their companion images in gcr.io/kubernetes-ci-images e.g.

  • v1.17 was the first release that had CI builds of kubernetes land in gs://k8s-release-dev/ci and gs://kubernetes-release-dev/ci simultaneously
  • v1.18 was the first release that cluster-api and kubeadm started defaulting to gcr.io/k8s-staging-ci-images instead of gcr.io/kubernetes-ci-images
  • v1.20 was the last release that had CI builds land in gs://kubernetes-release-dev/ci-cross
  • v1.22 is the last release to have CI builds of kubernetes land in gs://kubernetes-release-dev and gcr.io/kubernetes-ci-images
  • we will stop using gs://kubernetes-release-dev and gcr.io/kubernetes-ci-images completely once older CI jobs have aged out according to the kubernetes version support policy, or when we see no remaining traffic to gs://kubernetes-release-dev for a period of N weeks

Step 2: Setup community-owned alternatives, and use them for release-blocking / merge-blocking CI

This is a rough breakdown of the work necessary to complete the heavy lifting before we can shard out work to all repos across the project

Step 3: Ensure no references to kubernetes-release-dev within project

The next part of this is ensuring there are no remaining references to kubernetes-release-dev outside of vendor/ directories (except those necessary for the ci-kubernetes-build-.*deprecated.* jobs to satisfy the deprecation window)

  • Migrate documentation to refer to k8s-release-dev instead of kubernetes-release-dev
  • Migrate any remaining scripts / tooling across the project to use k8s-release-dev

This is pretty easily verifiable (with human inspection of results) using https://cs.k8s.io

$ curl -s -X POST   \
  -F "q=kubernetes-release-dev" \
  -F "repos=*" \
  -F "excludeFiles=vendor/"  \
  "https://cs.k8s.io/api/v1/search" \
  | jq -r '.Results | keys | .[]' \
  | sort \
  | sed -e 's/^/- [ ] /g'

Step 4: Ensure no references to kubernetes-ci-images within project

  • Migrate documentation to refer to k8s-staging-ci-images instead of kubernetes-ci-images
  • Migrate any remaining scripts / tooling across the project to use k8s-staging-ci-images
$ curl -s -X POST   \
>   -F "q=kubernetes-ci-images" \
>   -F "repos=*" \
>   -F "excludeFiles=vendor/"  \
>   "https://cs.k8s.io/api/v1/search" \
>   | jq -r '.Results | keys | .[]' \
>   | sort \
>   | sed -e 's/^/- [ ] /g'

Step 5: Complete deprecation

Related issues

/wg k8s-infra

/sig testing
/area prow
FYI @kubernetes/sig-testing-leads

/sig release
/area artifacts
/area release-eng
FYI @kubernetes/release-engineering

/kind cleanup
/kind deprecation
/priority important-soon
/milestone v1.22

/assign
@k8s-ci-robot k8s-ci-robot added wg/k8s-infra sig/testing Categorizes an issue or PR as relevant to SIG Testing. area/prow Setting up or working with prow in general, prow.k8s.io, prow build clusters labels Jul 9, 2021
@k8s-ci-robot k8s-ci-robot added this to the v1.22 milestone Jul 9, 2021
@k8s-ci-robot k8s-ci-robot added sig/release Categorizes an issue or PR as relevant to SIG Release. area/artifacts Issues or PRs related to the hosting of release artifacts for subprojects area/release-eng Issues or PRs related to the Release Engineering subproject kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. kind/deprecation Categorizes issue or PR as related to a feature/enhancement marked for deprecation. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. labels Jul 9, 2021
@spiffxp spiffxp changed the title Migrate kubernetes project away from gs://kubernetes-release-dev to gs://k8s-release-dev Migrate kubernetes project away from gs://kubernetes-release-dev and gcr.io/kubernetes-ci-images Jul 9, 2021
@spiffxp spiffxp mentioned this issue Jul 9, 2021
Open
6 tasks
@spiffxp spiffxp changed the title Migrate kubernetes project away from gs://kubernetes-release-dev and gcr.io/kubernetes-ci-images Deprecate and migrate away from gs://kubernetes-release-dev and gcr.io/kubernetes-ci-images Jul 9, 2021
This was referenced Jul 9, 2021
Merged
Closed
@spiffxp spiffxp added this to Needs Triage in sig-k8s-infra via automation Jul 9, 2021
@spiffxp spiffxp moved this from Needs Triage to In Progress in sig-k8s-infra Jul 9, 2021
This was referenced Jul 9, 2021
Merged
Merged
This was referenced Jul 12, 2021
Closed
Merged
@spiffxp
Copy link
Member Author

spiffxp commented Jul 12, 2021 •
edited

Timeline:

This was referenced Jul 12, 2021
Merged
Merged
Merged
Closed
Merged
Merged
Merged
This was referenced Jul 17, 2021
Merged
Merged
Merged
Merged
Merged
@spiffxp
Copy link
Member Author

spiffxp commented Jul 17, 2021

Went on a bit of a spree today and opened the following:

Where possible I tried to converge toward dl.k8.io instead of a simple s/kubernetes-release-dev/k8s-release-dev, since this is somewhat of a dry-run for the work to migrate gs://kubernetes-release

@spiffxp spiffxp mentioned this issue Jul 19, 2021
Merged
spiffxp added a commit to spiffxp/cloud-provider-gcp that referenced this issue Jul 19, 2021
@spiffxp
use dl.k8s.io where possible
cf2335c 
The only time a kubernetes GCS bucket name should be showing up in a
hardcoded URI is if gsutil is being used (e.g. gsutil cp gs://foo/bar .)

Otherwise, for tools like curl or wget, dl.k8s.io is much nicer for us
as a project, since we can transparently change where that redirects to
without having to change code everywhere

These changes will mean very few changes will be necessary to
accommodate a gs://kubernetes-release -> gs://k8s-release migration
equivalent of the CI migration we're going through right now

These changes also address the gs://kubernetes-release-dev references
currently used by this repo (ref:
kubernetes/k8s.io#2318)
@spiffxp spiffxp mentioned this issue Jul 19, 2021
Merged
@spiffxp
Copy link
Member Author

spiffxp commented Jul 22, 2021

Two more PRs opened / merged:

Need to do another pass and make sure documentation is wrapped up

@CecileRobertMichon CecileRobertMichon mentioned this issue Jul 22, 2021
Closed
@spiffxp spiffxp mentioned this issue Jul 23, 2021
Merged
@spiffxp
Copy link
Member Author

spiffxp commented Jul 23, 2021

I missed that the release-branch cut caused v1.23 builds to start getting pushed:

This was referenced Jul 26, 2021
Open
Closed
@spiffxp spiffxp added this to In Progress in sig-testing issues Jul 27, 2021
@spiffxp spiffxp mentioned this issue Jul 29, 2021
Merged
@spiffxp
Copy link
Member Author

spiffxp commented Jul 29, 2021

This should take care of the last of the non-necessary/historically-archived kubernetes-relesase-dev references:

This was referenced Jul 29, 2021
Merged
Merged
Merged
Merged
@spiffxp
Copy link
Member Author

spiffxp commented Aug 4, 2021 •
edited

kubernetes-release-dev traffic (sent bytes)
Screen Shot 2021-08-04 at 9 13 24 AM

@spiffxp
Copy link
Member Author

spiffxp commented Aug 4, 2021

kubernetes-ci-images traffic (sent bytes)
Screen Shot 2021-08-04 at 9 15 21 AM

@spiffxp
Copy link
Member Author

spiffxp commented Aug 5, 2021

/close
Announcement: https://groups.google.com/g/kubernetes-dev/c/58lP-CIP2zg/m/dS8TTmM4BwAJ

@k8s-ci-robot
Copy link
Contributor

@spiffxp: Closing this issue.

In response to this:

/close
Announcement: https://groups.google.com/g/kubernetes-dev/c/58lP-CIP2zg/m/dS8TTmM4BwAJ

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

sig-k8s-infra automation moved this from In Progress to Done Aug 5, 2021
sig-testing issues automation moved this from In Progress to Done Aug 5, 2021
@spiffxp spiffxp mentioned this issue Aug 5, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@spiffxp spiffxp

Labels
area/artifacts Issues or PRs related to the hosting of release artifacts for subprojects area/prow Setting up or working with prow in general, prow.k8s.io, prow build clusters area/release-eng Issues or PRs related to the Release Engineering subproject kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. kind/deprecation Categorizes issue or PR as related to a feature/enhancement marked for deprecation. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. sig/release Categorizes an issue or PR as relevant to SIG Release. sig/testing Categorizes an issue or PR as relevant to SIG Testing.
Projects
sig-k8s-infra
  
Done
sig-testing issues
  
Done
Milestone
v1.22
Development

No branches or pull requests
2 participants
@spiffxp @k8s-ci-robot