"master upgrade should maintain a functioning cluster" failing #103697

liggitt · 2021-07-14T18:48:10Z

Which jobs are failing:

ci-kubernetes-e2e-gce-stable1-beta-upgrade-master

Which test(s) are failing:

"master upgrade should maintain a functioning cluster"

Since when has it been failing:

Since #99857 merged

On 2021-07-09, the extract step also started failing.

Testgrid link:

testgrid titles are misleading, these are upgrading from 1.21 to 1.22

Reason for failure:

Refactor broke ginkgo usage.

/assign @wojtek-t
/cc @zshihang

Anything else we need to know:

liggitt · 2021-07-14T18:50:41Z

current blocking failure is Failed to set release from https://storage.googleapis.com/k8s-release-dev/ci/k8s-beta.txt (Unexpected HTTP status code: 404)

liggitt · 2021-07-14T18:51:41Z

that looks related to kubernetes/release@908c081

cc @spiffxp

liggitt · 2021-07-14T19:00:42Z

for reference, the auth upgrade jobs which upgrade from ci/latest-1.20 to ci/latest are currently green and are upgrading from v1.20.9-rc.0.20+66e6d5ee1fa946 to v1.22.0-beta.1.157+e375563732a6f5

https://testgrid.k8s.io/sig-auth-gce#upgrade-tests

wojtek-t · 2021-07-14T19:22:32Z

This is super strange. From example failure:
https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-stable1-beta-upgrade-master/1413528078591725568

Kubernetes e2e suite: [sig-cloud-provider-gcp] Upgrade [Feature:Upgrade] master upgrade should maintain a functioning cluster [Feature:MasterUpgrade] expand_less | 7m25s
-- | --
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cloud/gcp/cluster_upgrade.go:57 You may only call BeforeEach from within a Describe, Context or When /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:185

So clearly, the problem is connected from calling NewDefaultFramework.

Now - this is the upgrade auth upgrade job looks:

33 var _ = SIGDescribe("ServiceAccount admission controller migration [Feature:BoundServiceAccountTokenVolume]", func() {
34	f := framework.NewDefaultFramework("serviceaccount-admission-controller-migration")
35	testFrameworks := upgrades.CreateUpgradeFrameworks(upgradeTests)
36
37	ginkgo.Describe("master upgrade", func() {
38		ginkgo.It("should maintain a functioning cluster", func() {
39 			upgCtx, err := common.GetUpgradeContext(f.ClientSet.Discovery())
                        ...

vs the failing cluster upgrade:

51 var _ = SIGDescribe("Upgrade [Feature:Upgrade]", func() {
52 	f := framework.NewDefaultFramework("cluster-upgrade")
53 	testFrameworks := upgrades.CreateUpgradeFrameworks(upgradeTests)
54 
55 	// Create the frameworks here because we can only create them
56	// in a "Describe".
57	ginkgo.Describe("master upgrade", func() {
58		ginkgo.It("should maintain a functioning cluster [Feature:MasterUpgrade]", func() {
59			upgCtx, err := common.GetUpgradeContext(f.ClientSet.Discovery())
                        ..

The literally look the same for me with one failing and the other passing. I will keep looking...

wojtek-t · 2021-07-14T19:38:18Z

Ironically - the tests seem to actually be running, e.g. from the run above from logs:

...
I0709 16:06:37.901] �[0m[sig-cloud-provider-gcp] Upgrade [Feature:Upgrade]�[0m �[90mmaster upgrade�[0m 
I0709 16:06:37.901]   �[1mshould maintain a functioning cluster [Feature:MasterUpgrade]�[0m
I0709 16:06:37.901]   �[37m/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/cloud/gcp/cluster_upgrade.go:57�[0m
I0709 16:06:37.901] [BeforeEach] [sig-cloud-provider-gcp] Upgrade [Feature:Upgrade]
I0709 16:06:37.902]   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:185
I0709 16:06:37.902] �[1mSTEP�[0m: Creating a kubernetes client
I0709 16:06:37.902] Jul  9 16:06:37.892: INFO: >>> kubeConfig: /workspace/.kube/config
I0709 16:06:37.902] �[1mSTEP�[0m: Building a namespace api object, basename cluster-upgrade
I0709 16:06:38.042] Jul  9 16:06:38.042: INFO: No PodSecurityPolicies found; assuming PodSecurityPolicy is disabled.
I0709 16:06:38.042] W0709 16:06:38.042033   10421 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
...
I0709 16:14:00.610] Jul  9 16:14:00.610: INFO: Checking control plane version
I0709 16:14:00.719] Jul  9 16:14:00.718: INFO: Control plane is at version 1.21.3-rc.0.10+5f8f5ab3268b41
I0709 16:14:00.719] �[1mSTEP�[0m: Disruption complete; stopping async validations
I0709 16:14:00.719] �[1mSTEP�[0m: Waiting for async validations to complete
I0709 16:14:00.719] [AfterEach] [sig-cloud-provider-gcp] Upgrade [Feature:Upgrade]
I0709 16:14:00.719]   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:186

So it seems that the tests are actually running (and passing).
It's just at the end we have this strange error reported...

spiffxp · 2021-07-15T06:22:06Z

Failed to set release from https://storage.googleapis.com/k8s-release-dev/ci/k8s-beta.txt (Unexpected HTTP status code: 404)

I can't find a job that sets k8s-beta, so I will copy it, but it will be pointing to a stale build anyway:

$ gsutil cat gs://kubernetes-release-dev/ci/k8s-beta.txt
v1.21.3-rc.0.10+5f8f5ab3268b41
$ gsutil cat gs://kubernetes-release-dev/ci/latest-1.21.txt
v1.21.3-rc.0.26+f872796713459c
$ gsutil cat gs://k8s-release-dev/ci/latest-1.21.txt; echo
v1.21.3-rc.0.26+f872796713459c
$ gsutil cp gs://kubernetes-release-dev/ci/k8s-beta.txt gs://k8s-release-dev/ci
Copying gs://kubernetes-release-dev/ci/k8s-beta.txt [Content-Type=text/plain]...
/ [1 files][   31.0 B/   31.0 B]
Operation completed over 1 objects/31.0 B.

This is our usual fun dance of k8s-beta not meaning anything until a release branch gets cut, ref: kubernetes/sig-release#850

spiffxp · 2021-07-15T06:29:15Z

$ for m in latest latest-1.22 k8s-master; do echo $m: $(gsutil cat gs://k8s-release-dev/ci/$m.txt); done
latest: v1.22.0-beta.2.3+f5bc129a9916a1
latest-1.22: v1.22.0-beta.2.3+f5bc129a9916a1
k8s-master: v1.22.0-beta.2.3+f5bc129a9916a1

I forget exactly how this all plays with job rotation once release-branch jobs are cut. I would personally be fine with moving away from the k8s-foo markers towards hardcoded numbers, but I would rather defer to @kubernetes/release-engineering

/assign @cpanato @puerco @justaugustus

spiffxp · 2021-07-15T06:32:23Z

It looks like kubernetes/test-infra#22790 is what stopped writing gs://kubernetes-release-dev/ci/k8s-beta.txt

spiffxp · 2021-07-15T06:39:39Z

that looks related to kubernetes/release@908c081

cc @spiffxp

That wasn't it, but thanks for tagging me in, because I was definitely the final straw on this particular camel's back. The extract step failing on 2021-07-09 means it's either kubernetes/test-infra#22840 or one of its followup PRs mentioned in kubernetes/k8s.io#2318 (comment)

spiffxp · 2021-07-15T07:42:09Z

#99857 (comment) maybe this is the culprit for the first problem?

wojtek-t · 2021-07-15T07:49:06Z

See #103697 (comment) above [that change was reverted in one of subsequent PRs]

wojtek-t · 2021-07-15T07:50:08Z

Namely - this one: #101118

wojtek-t · 2021-07-15T08:08:22Z

I guess I know what's happening - #101118 has to be cherrypicked back to 1.21.
Will open a cherrypick later today [need to run now.]

wojtek-t · 2021-07-15T11:10:18Z

@liggitt - #103712 is out for review

liggitt · 2021-07-15T14:51:45Z

I can't find a job that sets k8s-beta, so I will copy it, but it will be pointing to a stale build anyway

Thanks, latest run passed the extract step and we're back to the ginkgo describe error. That should be fixed by #103712.

Once https://storage.googleapis.com/k8s-release-dev/ci/k8s-stable1.txt updates to v1.21.3-rc.0.28+4aa451e8458a7c, can you push that to k8s-beta.txt as a stop gap while we figure out whether we should start pushing k8s-beta from CI again or update the job config?

neolit123 · 2021-07-15T15:10:59Z

Just wanted to note that for the kubeadm upgrade jobs we stopped testing against stable and instead we test against the tips of release branches or master. This allows us to also catch problems in backports before a patch release is made. To do that we have to use the latest and latest-foo markers. The foo calculations are currently in the tooling, since there are no latest-1 markers (ala stable1).

liggitt · 2021-07-15T15:13:28Z

it looks like the ci/k8s-stable1.txt marker has the HEAD build for the most recent released minor version branch... isn't that what you want?

neolit123 · 2021-07-15T15:24:46Z

I guess it would. I may have been mislead by the 'stable' in the name. In practice the tooling can now do these calculations and its easy to say 'this upgrade job does latest-1 to latest upgrades', but one still needs to PR test infra. And IIRC the original intent of the *stable markers was mainly to not PR test infra on each release.

liggitt · 2021-07-15T15:31:07Z

looks like CI build v1.21.3-rc.0.28+4aa451e8458a7c is available now

spiffxp · 2021-07-15T15:31:14Z

can you push that to k8s-beta.txt as a stop gap while we figure out whether we should start pushing k8s-beta from CI again or update the job config?

Done

$ gsutil cat gs://k8s-release-dev/ci/latest-1.21.txt; echo
v1.21.3-rc.0.28+4aa451e8458a7c
$ gsutil cp gs://k8s-release-dev/ci/latest-1.21.txt gs://k8s-release-dev/ci/k8s-beta.txt
Copying gs://k8s-release-dev/ci/latest-1.21.txt [Content-Type=text/plain]...
/ [1 files][   30.0 B/   30.0 B]
Operation completed over 1 objects/30.0 B.

liggitt · 2021-07-15T15:31:24Z

awesome, thanks

spiffxp · 2021-07-15T15:40:06Z

And IIRC the original intent of the *stable markers was mainly to not PR test infra on each release.

If we're going to make a decision on what we think version markers should look like, that's what kubernetes/sig-release#850 is for. IMO the contributor clarity gained by hardcoding version numbers would far outweigh the toil of updating them every N months.

neolit123 · 2021-07-15T16:31:43Z

IMO the contributor clarity gained by hardcoding version numbers would far outweigh the toil of updating them every N months.

I agree since some markers have been a bit cryptic. Tooling can help with the regular updates too.

spiffxp · 2021-07-16T00:31:25Z

/assign
keeping an eye on the latest runs

wojtek-t · 2021-07-16T05:48:57Z

OK - so cluster upgrade tests become green again after the change:
https://testgrid.k8s.io/google-gce-upgrade#gce-1.13-1.14-upgrade-cluster&width=25

But it seems that master upgrade is panicing (I don't know how it worked for me before). It's a type - going to send out fix soon.

wojtek-t · 2021-07-16T05:54:09Z

fix in master branch: #103733
cherrypick to 1.21: #103734

liggitt · 2021-07-16T13:58:51Z

#103734 is merged, should resolve the last failure

as long as the job points to ci/k8s-beta.txt, I guess we'll need one more bump of that file once https://storage.googleapis.com/k8s-release-dev/ci/k8s-stable1.txt updates to v1.21.4-rc.0.3+0e1bd6ab564...

liggitt · 2021-07-16T14:26:33Z

also opened kubernetes/test-infra#22915 to fix up the testgrid tab names and make these tests use ci/latest instead of ci/k8s-beta (happy to re-rework that in the future if k8s-beta starts being automatically populated again)

liggitt · 2021-07-16T15:20:49Z

I guess we'll need one more bump of that file once https://storage.googleapis.com/k8s-release-dev/ci/k8s-stable1.txt updates to v1.21.4-rc.0.3+0e1bd6ab564...

looks like it's ready now

spiffxp · 2021-07-16T18:03:57Z

I got pulled away from keyboard, I'll sync shortly though the test-infra PR will probably obviate

spiffxp · 2021-07-16T18:11:30Z

$ gsutil cp gs://kubernetes-release-dev/ci/latest-1.21.txt gs://k8s-release-dev/ci/k8s-beta.txt
Copying gs://kubernetes-release-dev/ci/latest-1.21.txt [Content-Type=text/plain]...
/ [1 files][   30.0 B/   30.0 B]
Operation completed over 1 objects/30.0 B.

$ for m in k8s-beta latest-1.21 k8s-stable1; do echo $m: $(gsutil cat gs://k8s-release-dev/ci/$m.txt); done
k8s-beta: v1.21.4-rc.0.3+0e1bd6ab564a0a
latest-1.21: v1.21.4-rc.0.3+0e1bd6ab564a0a
k8s-stable1: v1.21.4-rc.0.3+0e1bd6ab564a0a

wojtek-t · 2021-07-21T12:29:22Z

OK - so the upgrade itself works now.

However, there are gazilions of storage tests that started failing after Jordan upgraded to use latest.

@liggitt - should we close this one and open a separate bug for the failing storage tests?

liggitt · 2021-07-21T13:34:46Z

@liggitt - should we close this one and open a separate bug for the failing storage tests?

probably so

wojtek-t · 2021-07-21T13:37:50Z

Opened #103822

Closing this one as resolved.

liggitt added the kind/failing-test Categorizes issue or PR as related to a consistently or frequently failing test. label Jul 14, 2021

k8s-ci-robot assigned wojtek-t Jul 14, 2021

k8s-ci-robot added needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Jul 14, 2021

liggitt mentioned this issue Jul 14, 2021

Stop using SIGDescribe for upgrade tests to reflect reality #99857

Merged

liggitt added this to the v1.22 milestone Jul 14, 2021

liggitt added the sig/testing Categorizes an issue or PR as relevant to SIG Testing. label Jul 14, 2021

k8s-ci-robot removed the needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. label Jul 14, 2021

This was referenced Jul 14, 2021

bash: use k8s-release-dev for CI builds kubernetes/release#2158

Merged

Install pdcsi driver by default #102701

Closed

liggitt added priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. triage/accepted Indicates an issue or PR is ready to be actively worked on. labels Jul 14, 2021

k8s-ci-robot removed the needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. label Jul 14, 2021

k8s-ci-robot assigned cpanato, justaugustus and puerco Jul 15, 2021

wojtek-t mentioned this issue Jul 15, 2021

Automated cherry pick of #101118 upstream release 1.21 #103712

Merged

liggitt mentioned this issue Jul 15, 2021

releng: Build job fixups kubernetes/test-infra#22790

Merged

spiffxp mentioned this issue Jul 15, 2021

Deprecate and migrate away from gs://kubernetes-release-dev and gcr.io/kubernetes-ci-images kubernetes/k8s.io#2318

Closed

42 tasks

spiffxp mentioned this issue Jul 15, 2021

Assess version markers used in release-branch job configs kubernetes/sig-release#850

Open

9 tasks

k8s-ci-robot assigned spiffxp Jul 16, 2021

This was referenced Jul 16, 2021

Fix panic in master upgrade tests #103733

Merged

Automated cherry pick of #103733 upstream release 1.21 #103734

Merged

liggitt mentioned this issue Jul 16, 2021

Clean up upgrade/downgrade test configs kubernetes/test-infra#22915

Merged

wojtek-t closed this as completed Jul 21, 2021

liggitt mentioned this issue Apr 4, 2022

e2e: restructure e2e framework and its sub-packages to solve potential cycle imports #109224

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

"master upgrade should maintain a functioning cluster" failing #103697

"master upgrade should maintain a functioning cluster" failing #103697

liggitt commented Jul 14, 2021 •

edited

liggitt commented Jul 14, 2021

liggitt commented Jul 14, 2021

liggitt commented Jul 14, 2021

wojtek-t commented Jul 14, 2021

wojtek-t commented Jul 14, 2021

spiffxp commented Jul 15, 2021

spiffxp commented Jul 15, 2021

spiffxp commented Jul 15, 2021

spiffxp commented Jul 15, 2021 •

edited

spiffxp commented Jul 15, 2021

wojtek-t commented Jul 15, 2021

wojtek-t commented Jul 15, 2021

wojtek-t commented Jul 15, 2021

wojtek-t commented Jul 15, 2021

liggitt commented Jul 15, 2021 •

edited

neolit123 commented Jul 15, 2021 via email •

edited

liggitt commented Jul 15, 2021

neolit123 commented Jul 15, 2021

liggitt commented Jul 15, 2021

spiffxp commented Jul 15, 2021

liggitt commented Jul 15, 2021

spiffxp commented Jul 15, 2021

neolit123 commented Jul 15, 2021

spiffxp commented Jul 16, 2021

wojtek-t commented Jul 16, 2021

wojtek-t commented Jul 16, 2021

liggitt commented Jul 16, 2021 •

edited

liggitt commented Jul 16, 2021 •

edited

liggitt commented Jul 16, 2021

spiffxp commented Jul 16, 2021

spiffxp commented Jul 16, 2021 •

edited

wojtek-t commented Jul 21, 2021

liggitt commented Jul 21, 2021

wojtek-t commented Jul 21, 2021

"master upgrade should maintain a functioning cluster" failing #103697

"master upgrade should maintain a functioning cluster" failing #103697

Comments

liggitt commented Jul 14, 2021 • edited

Which jobs are failing:

Which test(s) are failing:

Since when has it been failing:

Testgrid link:

Reason for failure:

Anything else we need to know:

liggitt commented Jul 14, 2021

liggitt commented Jul 14, 2021

liggitt commented Jul 14, 2021

wojtek-t commented Jul 14, 2021

wojtek-t commented Jul 14, 2021

spiffxp commented Jul 15, 2021

spiffxp commented Jul 15, 2021

spiffxp commented Jul 15, 2021

spiffxp commented Jul 15, 2021 • edited

spiffxp commented Jul 15, 2021

wojtek-t commented Jul 15, 2021

wojtek-t commented Jul 15, 2021

wojtek-t commented Jul 15, 2021

wojtek-t commented Jul 15, 2021

liggitt commented Jul 15, 2021 • edited

neolit123 commented Jul 15, 2021 via email • edited

liggitt commented Jul 15, 2021

neolit123 commented Jul 15, 2021

liggitt commented Jul 15, 2021

spiffxp commented Jul 15, 2021

liggitt commented Jul 15, 2021

spiffxp commented Jul 15, 2021

neolit123 commented Jul 15, 2021

spiffxp commented Jul 16, 2021

wojtek-t commented Jul 16, 2021

wojtek-t commented Jul 16, 2021

liggitt commented Jul 16, 2021 • edited

liggitt commented Jul 16, 2021 • edited

liggitt commented Jul 16, 2021

spiffxp commented Jul 16, 2021

spiffxp commented Jul 16, 2021 • edited

wojtek-t commented Jul 21, 2021

liggitt commented Jul 21, 2021

wojtek-t commented Jul 21, 2021

liggitt commented Jul 14, 2021 •

edited

spiffxp commented Jul 15, 2021 •

edited

liggitt commented Jul 15, 2021 •

edited

neolit123 commented Jul 15, 2021 via email •

edited

liggitt commented Jul 16, 2021 •

edited

liggitt commented Jul 16, 2021 •

edited

spiffxp commented Jul 16, 2021 •

edited