Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

InPlacePodVerticalScaling CI tests fail due to parallel tests stepping over each other in low CPU count CI system #116371

Closed
vinaykul opened this issue Mar 8, 2023 · 5 comments
Closed

InPlacePodVerticalScaling CI tests fail due to parallel tests stepping over each other in low CPU count CI system #116371

vinaykul opened this issue Mar 8, 2023 · 5 comments
Assignees
@vinaykul
Labels
kind/bug Categorizes issue or PR as related to a bug. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. sig/node Categorizes an issue or PR as relevant to SIG Node.
Projects
SIG Node Bugs

Comments

@vinaykul
Copy link
Contributor

vinaykul commented Mar 8, 2023

What happened?

In-place pod resize CI tests were re-enabled in PR kubernetes/test-infra#28928 following merge of PR #102884 These tests are failing and the root cause appears to be pods timing out awaiting scheduling, Unschedulable due to Insufficient cpu.

Running these tests in serial is a quick way to address the problem. It can be followed by a better fix.

What did you expect to happen?

All in-place pod resize E2E tests should pass in CI.

How can we reproduce it (as minimally and precisely as possible)?

https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/ci-cos-cgroupv2-inplace-pod-resize-containerd-main-e2e-gce/1633236905220902912
https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/ci-cos-cgroupv1-inplace-pod-resize-containerd-main-e2e-gce/1633236905296400384

Anything else we need to know?

No response

Kubernetes version

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"27+", GitVersion:"v1.27.0-alpha.3.213+f99c3519929004-dirty", GitCommit:"f99c35199290049ee2420d527c121288ced7a3fb", GitTreeState:"dirty", BuildDate:"2023-03-08T11:41:07Z", GoVersion:"go1.20.1", Compiler:"gc", Platform:"linux/arm64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"27+", GitVersion:"v1.27.0-alpha.3.213+f99c3519929004-dirty", GitCommit:"f99c35199290049ee2420d527c121288ced7a3fb", GitTreeState:"dirty", BuildDate:"2023-03-08T11:41:07Z", GoVersion:"go1.20.1", Compiler:"gc", Platform:"linux/arm64"}```

</details>


### Cloud provider

<details>
GKE CI
</details>


### OS version

<details>

```console
# On Linux: COS
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here

Install tools

Container runtime (CRI) and version (if applicable)

Related plugins (CNI, CSI, ...) and versions (if applicable)
@vinaykul vinaykul added the kind/bug Categorizes issue or PR as related to a bug. label Mar 8, 2023
@k8s-ci-robot k8s-ci-robot added needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Mar 8, 2023
@k8s-ci-robot
Copy link
Contributor

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@vinaykul
Copy link
Contributor Author

vinaykul commented Mar 8, 2023

/assign @vinaykul

@vinaykul
Copy link
Contributor Author

vinaykul commented Mar 8, 2023

/sig node

@k8s-ci-robot k8s-ci-robot added sig/node Categorizes an issue or PR as relevant to SIG Node. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Mar 8, 2023
@vinaykul vinaykul mentioned this issue Mar 8, 2023
Closed
@SergeyKanzhelev SergeyKanzhelev added this to Triage in SIG Node Bugs Mar 8, 2023
@vinaykul
Copy link
Contributor Author

vinaykul commented Mar 8, 2023

IMO, the issue I see when the in-place pod resize CI that was enabled yesterday is multiple tests running in parallel were eating into the limited allocatable CPU on the system (2000m) and causing failures.

An example from a failed test log in
https://storage.googleapis.com/kubernetes-jenkins/logs/ci-cos-cgroupv2-inplace-pod-resize-containerd-main-e2e-gce/1633236905220902912/build-log.txt

I0307 22:56:52.959] ------------------------------
I0307 22:56:52.959] • [FAILED] [304.734 seconds]
I0307 22:56:52.960] [sig-node] Pod InPlace Resize Container [Feature:InPlacePodVerticalScaling] [It] Burstable QoS pod, one container with cpu & memory requests + limits - increase CPU limits only
I0307 22:56:52.960] test/e2e/node/pod_resize.go:1206
I0307 22:56:52.960] 
I0307 22:56:52.960]   Timeline >>
I0307 22:56:52.960]   STEP: Creating a kubernetes client @ 03/07/23 22:51:48.22
I0307 22:56:52.960]   Mar  7 22:51:48.220: INFO: >>> kubeConfig: /workspace/.kube/config
I0307 22:56:52.960]   STEP: Building a namespace api object, basename pod-resize @ 03/07/23 22:51:48.223
I0307 22:56:52.960]   STEP: Waiting for a default service account to be provisioned in namespace @ 03/07/23 22:51:48.443
I0307 22:56:52.960]   STEP: Waiting for kube-root-ca.crt to be provisioned in namespace @ 03/07/23 22:51:48.53
I0307 22:56:52.960]   STEP: Creating a kubernetes client @ 03/07/23 22:51:48.614
I0307 22:56:52.960]   Mar  7 22:51:48.614: INFO: >>> kubeConfig: /workspace/.kube/config
I0307 22:56:52.960]   STEP: Building a namespace api object, basename pod-resize-resource-quota @ 03/07/23 22:51:48.616
I0307 22:56:52.960]   STEP: Waiting for a default service account to be provisioned in namespace @ 03/07/23 22:51:48.837
I0307 22:56:52.960]   STEP: Waiting for kube-root-ca.crt to be provisioned in namespace @ 03/07/23 22:51:48.924
I0307 22:56:52.960]   STEP: Creating a kubernetes client @ 03/07/23 22:51:49.014
I0307 22:56:52.960]   Mar  7 22:51:49.014: INFO: >>> kubeConfig: /workspace/.kube/config
I0307 22:56:52.960]   STEP: Building a namespace api object, basename pod-resize-errors @ 03/07/23 22:51:49.016
I0307 22:56:52.960]   STEP: Waiting for a default service account to be provisioned in namespace @ 03/07/23 22:51:49.182
I0307 22:56:52.960]   STEP: Waiting for kube-root-ca.crt to be provisioned in namespace @ 03/07/23 22:51:49.264
I0307 22:56:52.960]   STEP: creating pod @ 03/07/23 22:51:49.347
I0307 22:56:52.960]   Automatically polling progress:
I0307 22:56:52.960]     [sig-node] Pod InPlace Resize Container [Feature:InPlacePodVerticalScaling] Burstable QoS pod, one container with cpu & memory requests + limits - increase CPU limits only (Spec Runtime: 5m1.128s)
I0307 22:56:52.960]       test/e2e/node/pod_resize.go:1206
I0307 22:56:52.961]       In [It] (Node Runtime: 5m0.001s)
I0307 22:56:52.961]         test/e2e/node/pod_resize.go:1206
I0307 22:56:52.961]         At [By Step] creating pod (Step Runtime: 5m0.001s)
I0307 22:56:52.961]           test/e2e/node/pod_resize.go:1215
I0307 22:56:52.961] 
I0307 22:56:52.961]         Spec Goroutine
I0307 22:56:52.961]         goroutine 915 [select]
I0307 22:56:52.961]           k8s.io/kubernetes/vendor/github.com/onsi/gomega/internal.(*AsyncAssertion).match(0xc00003a620, {0x7f0c650?, 0xc000a6c8b0}, 0x1, {0x0, 0x0, 0x0})
I0307 22:56:52.961]             vendor/github.com/onsi/gomega/internal/async_assertion.go:530
I0307 22:56:52.961]           k8s.io/kubernetes/vendor/github.com/onsi/gomega/internal.(*AsyncAssertion).Should(0xc00003a620, {0x7f0c650, 0xc000a6c8b0}, {0x0, 0x0, 0x0})
I0307 22:56:52.961]             vendor/github.com/onsi/gomega/internal/async_assertion.go:145
I0307 22:56:52.961]           k8s.io/kubernetes/test/e2e/framework.asyncAssertion.Should({{0x7f32a44a2c18, 0xc0019511c0}, {0xc000a6c8a0, 0x1, 0x1}, 0x45d964b800, 0x77359400, 0x0}, {0x7f0c650, 0xc000a6c8b0})
I0307 22:56:52.961]             test/e2e/framework/expect.go:234
I0307 22:56:52.961]           k8s.io/kubernetes/test/e2e/framework/pod.WaitForPodCondition({0x7f32a44a2c18, 0xc0019511c0}, {0x7f62618?, 0xc0013fcd00?}, {0xc0011256c0, 0xf}, {0xc005463ef3, 0x7}, {0x7581506, 0x11}, ...)
I0307 22:56:52.961]             test/e2e/framework/pod/wait.go:228
I0307 22:56:52.961]           k8s.io/kubernetes/test/e2e/framework/pod.WaitTimeoutForPodReadyInNamespace({0x7f32a44a2c18?, 0xc0019511c0?}, {0x7f62618?, 0xc0013fcd00?}, {0xc005463ef3?, 0x0?}, {0xc0011256c0?, 0x2aac985?}, 0x0?)
I0307 22:56:52.961]             test/e2e/framework/pod/wait.go:492
I0307 22:56:52.961]           k8s.io/kubernetes/test/e2e/framework/pod.(*PodClient).CreateSync(0xc000013cb0, {0x7f32a44a2c18, 0xc0019511c0}, 0x7?)
I0307 22:56:52.961]             test/e2e/framework/pod/pod_client.go:106
I0307 22:56:52.961]         > k8s.io/kubernetes/test/e2e/node.doPodResizeTests.func2({0x7f32a44a2c18, 0xc0019511c0})
I0307 22:56:52.961]             test/e2e/node/pod_resize.go:1216
I0307 22:56:52.961]           k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x7f2c708?, 0xc0019511c0})
I0307 22:56:52.961]             vendor/github.com/onsi/ginkgo/v2/internal/node.go:452
I0307 22:56:52.962]           k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
I0307 22:56:52.962]             vendor/github.com/onsi/ginkgo/v2/internal/suite.go:854
I0307 22:56:52.962]           k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/v2/internal.(*Suite).runNode
I0307 22:56:52.962]             vendor/github.com/onsi/ginkgo/v2/internal/suite.go:841
I0307 22:56:52.962] 
I0307 22:56:52.962]         Begin Additional Progress Reports >>
I0307 22:56:52.962]           expected pod to be running and ready, got instead:
I0307 22:56:52.962]               <*v1.Pod | 0xc0020c6480>: 
I0307 22:56:52.962]                   metadata:
I0307 22:56:52.962]                     creationTimestamp: "2023-03-07T22:51:49Z"
I0307 22:56:52.962]                     labels:
I0307 22:56:52.962]                       name: fooPod
I0307 22:56:52.962]                       time: "347833634"
I0307 22:56:52.962]                     managedFields:
I0307 22:56:52.962]                     - apiVersion: v1
I0307 22:56:52.962]                       fieldsType: FieldsV1
I0307 22:56:52.962]                       fieldsV1:
I0307 22:56:52.962]                         f:metadata:
I0307 22:56:52.963]                           f:labels:
I0307 22:56:52.963]                             .: {}
I0307 22:56:52.963]                             f:name: {}
I0307 22:56:52.963]                             f:time: {}
I0307 22:56:52.963]                         f:spec:
I0307 22:56:52.963]                           f:containers:
I0307 22:56:52.963]                             k:{"name":"c1"}:
I0307 22:56:52.963]                               .: {}
I0307 22:56:52.963]                               f:args: {}
I0307 22:56:52.963]                               f:command: {}
I0307 22:56:52.963]                               f:image: {}
I0307 22:56:52.963]                               f:imagePullPolicy: {}
I0307 22:56:52.963]                               f:name: {}
I0307 22:56:52.964]                               f:resizePolicy: {}
I0307 22:56:52.964]                               f:resources:
I0307 22:56:52.964]                                 .: {}
I0307 22:56:52.964]                                 f:limits:
I0307 22:56:52.964]                                   .: {}
I0307 22:56:52.964]                                   f:cpu: {}
I0307 22:56:52.964]                                   f:memory: {}
I0307 22:56:52.964]                                 f:requests:
I0307 22:56:52.964]                                   .: {}
I0307 22:56:52.964]                                   f:cpu: {}
I0307 22:56:52.964]                                   f:memory: {}
I0307 22:56:52.964]                               f:securityContext:
I0307 22:56:52.964]                                 .: {}
I0307 22:56:52.965]                                 f:allowPrivilegeEscalation: {}
I0307 22:56:52.965]                                 f:capabilities:
I0307 22:56:52.965]                                   .: {}
I0307 22:56:52.965]                                   f:drop: {}
I0307 22:56:52.965]                                 f:privileged: {}
I0307 22:56:52.965]                                 f:runAsNonRoot: {}
I0307 22:56:52.965]                                 f:runAsUser: {}
I0307 22:56:52.965]                                 f:seccompProfile:
I0307 22:56:52.965]                                   .: {}
I0307 22:56:52.965]                                   f:type: {}
I0307 22:56:52.965]                               f:terminationMessagePath: {}
I0307 22:56:52.965]                               f:terminationMessagePolicy: {}
I0307 22:56:52.965]                           f:dnsPolicy: {}
I0307 22:56:52.965]                           f:enableServiceLinks: {}
I0307 22:56:52.966]                           f:restartPolicy: {}
I0307 22:56:52.966]                           f:schedulerName: {}
I0307 22:56:52.966]                           f:securityContext: {}
I0307 22:56:52.966]                           f:terminationGracePeriodSeconds: {}
I0307 22:56:52.966]                       manager: e2e.test
I0307 22:56:52.966]                       operation: Update
I0307 22:56:52.966]                       time: "2023-03-07T22:51:49Z"
I0307 22:56:52.966]                     - apiVersion: v1
I0307 22:56:52.966]                       fieldsType: FieldsV1
I0307 22:56:52.966]                       fieldsV1:
I0307 22:56:52.966]                         f:status:
I0307 22:56:52.966]                           f:conditions:
I0307 22:56:52.967]                             .: {}
I0307 22:56:52.967]                             k:{"type":"PodScheduled"}:
I0307 22:56:52.967]                               .: {}
I0307 22:56:52.967]                               f:lastProbeTime: {}
I0307 22:56:52.967]                               f:lastTransitionTime: {}
I0307 22:56:52.967]                               f:message: {}
I0307 22:56:52.967]                               f:reason: {}
I0307 22:56:52.967]                               f:status: {}
I0307 22:56:52.967]                               f:type: {}
I0307 22:56:52.967]                       manager: kube-scheduler
I0307 22:56:52.967]                       operation: Update
I0307 22:56:52.967]                       subresource: status
I0307 22:56:52.968]                       time: "2023-03-07T22:51:49Z"
I0307 22:56:52.968]                     name: testpod
I0307 22:56:52.968]                     namespace: pod-resize-2100
I0307 22:56:52.968]                     resourceVersion: "1161"
I0307 22:56:52.968]                     uid: bade723e-7263-4038-8bc3-e46390517448
I0307 22:56:52.968]                   spec:
I0307 22:56:52.968]                     containers:
I0307 22:56:52.968]                     - args:
I0307 22:56:52.968]                       - -c
I0307 22:56:52.968]                       - trap exit TERM; while true; do sleep 1; done
I0307 22:56:52.968]                       command:
I0307 22:56:52.968]                       - /bin/sh
I0307 22:56:52.968]                       image: registry.k8s.io/e2e-test-images/busybox:1.29-4
I0307 22:56:52.969]                       imagePullPolicy: IfNotPresent
I0307 22:56:52.969]                       name: c1
I0307 22:56:52.969]                       resizePolicy:
I0307 22:56:52.969]                       - policy: RestartNotRequired
I0307 22:56:52.969]                         resourceName: cpu
I0307 22:56:52.969]                       - policy: RestartNotRequired
I0307 22:56:52.969]                         resourceName: memory
I0307 22:56:52.969]                       resources:
I0307 22:56:52.969]                         limits:
I0307 22:56:52.969]                           cpu: 400m
I0307 22:56:52.969]                           memory: 500Mi
I0307 22:56:52.969]                         requests:
I0307 22:56:52.969]                           cpu: 200m
I0307 22:56:52.970]                           memory: 250Mi
I0307 22:56:52.970]                       securityContext:
I0307 22:56:52.970]                         allowPrivilegeEscalation: false
I0307 22:56:52.970]                         capabilities:
I0307 22:56:52.970]                           drop:
I0307 22:56:52.970]                           - ALL
I0307 22:56:52.970]                         privileged: false
I0307 22:56:52.970]                         runAsNonRoot: true
I0307 22:56:52.970]                         runAsUser: 1001
I0307 22:56:52.970]                         seccompProfile:
I0307 22:56:52.970]                           type: RuntimeDefault
I0307 22:56:52.970]                       terminationMessagePath: /dev/termination-log
I0307 22:56:52.970]                       terminationMessagePolicy: File
I0307 22:56:52.971]                       volumeMounts:
I0307 22:56:52.971]                       - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
I0307 22:56:52.971]                         name: kube-api-access-xtrk7
I0307 22:56:52.971]                         readOnly: true
I0307 22:56:52.971]                     dnsPolicy: ClusterFirst
I0307 22:56:52.971]                     enableServiceLinks: true
I0307 22:56:52.971]                     preemptionPolicy: PreemptLowerPriority
I0307 22:56:52.971]                     priority: 0
I0307 22:56:52.971]                     restartPolicy: OnFailure
I0307 22:56:52.971]                     schedulerName: default-scheduler
I0307 22:56:52.972]                     securityContext: {}
I0307 22:56:52.972]                     serviceAccount: default
I0307 22:56:52.972]                     serviceAccountName: default
I0307 22:56:52.972]                     terminationGracePeriodSeconds: 30
I0307 22:56:52.972]                     tolerations:
I0307 22:56:52.972]                     - effect: NoExecute
I0307 22:56:52.972]                       key: node.kubernetes.io/not-ready
I0307 22:56:52.972]                       operator: Exists
I0307 22:56:52.972]                       tolerationSeconds: 300
I0307 22:56:52.972]                     - effect: NoExecute
I0307 22:56:52.973]                       key: node.kubernetes.io/unreachable
I0307 22:56:52.973]                       operator: Exists
I0307 22:56:52.973]                       tolerationSeconds: 300
I0307 22:56:52.973]                     volumes:
I0307 22:56:52.973]                     - name: kube-api-access-xtrk7
I0307 22:56:52.973]                       projected:
I0307 22:56:52.973]                         defaultMode: 420
I0307 22:56:52.973]                         sources:
I0307 22:56:52.973]                         - serviceAccountToken:
I0307 22:56:52.973]                             expirationSeconds: 3607
I0307 22:56:52.973]                             path: token
I0307 22:56:52.974]                         - configMap:
I0307 22:56:52.974]                             items:
I0307 22:56:52.974]                             - key: ca.crt
I0307 22:56:52.974]                               path: ca.crt
I0307 22:56:52.974]                             name: kube-root-ca.crt
I0307 22:56:52.974]                         - downwardAPI:
I0307 22:56:52.974]                             items:
I0307 22:56:52.974]                             - fieldRef:
I0307 22:56:52.974]                                 apiVersion: v1
I0307 22:56:52.974]                                 fieldPath: metadata.namespace
I0307 22:56:52.975]                               path: namespace
I0307 22:56:52.975]                   status:
I0307 22:56:52.975]                     conditions:
I0307 22:56:52.975]                     - lastProbeTime: null
I0307 22:56:52.975]                       lastTransitionTime: "2023-03-07T22:51:49Z"
I0307 22:56:52.975]                       message: '0/2 nodes are available: 1 Insufficient cpu, 1 node(s) were unschedulable.
I0307 22:56:52.975]                         preemption: 0/2 nodes are available: 1 No preemption victims found for incoming
I0307 22:56:52.975]                         pod, 1 Preemption is not helpful for scheduling..'
I0307 22:56:52.975]                       reason: Unschedulable
I0307 22:56:52.975]                       status: "False"
I0307 22:56:52.975]                       type: PodScheduled
I0307 22:56:52.975]                     phase: Pending
I0307 22:56:52.976]                     qosClass: Burstable
I0307 22:56:52.976]         << End Additional Progress Reports
I0307 22:56:52.976]   Mar  7 22:56:49.410: INFO: Failed inside E2E framework:
I0307 22:56:52.976]       k8s.io/kubernetes/test/e2e/framework/pod.WaitForPodCondition({0x7f32a44a2c18, 0xc0019511c0}, {0x7f62618?, 0xc0013fcd00?}, {0xc0011256c0, 0xf}, {0xc005463ef3, 0x7}, {0x7581506, 0x11}, ...)
I0307 22:56:52.976]       	test/e2e/framework/pod/wait.go:228 +0x25f
I0307 22:56:52.976]       k8s.io/kubernetes/test/e2e/framework/pod.WaitTimeoutForPodReadyInNamespace({0x7f32a44a2c18?, 0xc0019511c0?}, {0x7f62618?, 0xc0013fcd00?}, {0xc005463ef3?, 0x0?}, {0xc0011256c0?, 0x2aac985?}, 0x0?)
I0307 22:56:52.976]       	test/e2e/framework/pod/wait.go:492 +0x75
I0307 22:56:52.976]       k8s.io/kubernetes/test/e2e/framework/pod.(*PodClient).CreateSync(0xc000013cb0, {0x7f32a44a2c18, 0xc0019511c0}, 0x7?)
I0307 22:56:52.976]       	test/e2e/framework/pod/pod_client.go:106 +0x97
I0307 22:56:52.976]       k8s.io/kubernetes/test/e2e/node.doPodResizeTests.func2({0x7f32a44a2c18, 0xc0019511c0})
I0307 22:56:52.976]       	test/e2e/node/pod_resize.go:1216 +0x29d
I0307 22:56:52.976]   [FAILED] in [It] - test/e2e/framework/pod/pod_client.go:106 @ 03/07/23 22:56:49.41
I0307 22:56:52.977]   Mar  7 22:56:49.410: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
I0307 22:56:52.977]   Mar  7 22:56:49.495: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
I0307 22:56:52.977]   Mar  7 22:56:49.539: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
I0307 22:56:52.977]   STEP: dump namespace information after failure @ 03/07/23 22:56:49.585
I0307 22:56:52.977]   STEP: Collecting events from namespace "pod-resize-errors-5796". @ 03/07/23 22:56:49.585
I0307 22:56:52.977]   STEP: Found 0 events. @ 03/07/23 22:56:49.626
I0307 22:56:52.977]   Mar  7 22:56:49.667: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
I0307 22:56:52.977]   Mar  7 22:56:49.667: INFO: 
I0307 22:56:52.977]   Mar  7 22:56:49.709: INFO: 
I0307 22:56:52.978]   Logging node info for node bootstrap-e2e-master
I0307 22:56:52.978]   Mar  7 22:56:49.751: INFO: Node Info: &Node{ObjectMeta:{bootstrap-e2e-master    f2c12bc2-c08f-4e00-9187-b8cdc2adb9ed 1799 0 2023-03-07 22:49:20 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:n1-standard-1 beta.kubernetes.io/os:linux cloud.google.com/metadata-proxy-ready:true failure-domain.beta.kubernetes.io/region:us-west1 failure-domain.beta.kubernetes.io/zone:us-west1-b kubernetes.io/arch:amd64 kubernetes.io/hostname:bootstrap-e2e-master kubernetes.io/os:linux node.kubernetes.io/instance-type:n1-standard-1 topology.kubernetes.io/region:us-west1 topology.kubernetes.io/zone:us-west1-b] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-03-07 22:49:20 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:cloud.google.com/metadata-proxy-ready":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{},"f:unschedulable":{}}} } {kube-controller-manager Update v1 2023-03-07 22:49:29 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"10.64.0.0/24\"":{}},"f:taints":{}}} } {kube-controller-manager Update v1 2023-03-07 22:49:29 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}}}} status} {kubelet Update v1 2023-03-07 22:54:46 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:10.64.0.0/24,DoNotUseExternalID:,ProviderID:gce://k8s-jkns-e2e-bazel/us-west1-b/bootstrap-e2e-master,Unschedulable:true,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:,Effect:NoSchedule,TimeAdded:<nil>,},Taint{Key:node.kubernetes.io/unschedulable,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.64.0.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-gce-pd: {{127 0} {<nil>} 127 DecimalSI},cpu: {{1 0} {<nil>} 1 DecimalSI},ephemeral-storage: {{16656896000 0} {<nil>} 16266500Ki BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3858370560 0} {<nil>} 3767940Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-gce-pd: {{127 0} {<nil>} 127 DecimalSI},cpu: {{1 0} {<nil>} 1 DecimalSI},ephemeral-storage: {{14991206376 0} {<nil>} 14991206376 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3596226560 0} {<nil>} 3511940Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-07 22:49:29 +0000 UTC,LastTransitionTime:2023-03-07 22:49:29 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-07 22:54:46 +0000 UTC,LastTransitionTime:2023-03-07 22:49:20 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-07 22:54:46 +0000 UTC,LastTransitionTime:2023-03-07 22:49:20 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-07 22:54:46 +0000 UTC,LastTransitionTime:2023-03-07 22:49:20 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-07 22:54:46 +0000 UTC,LastTransitionTime:2023-03-07 22:49:22 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.138.0.2,},NodeAddress{Type:ExternalIP,Address:34.168.107.101,},NodeAddress{Type:InternalDNS,Address:bootstrap-e2e-master.c.k8s-jkns-e2e-bazel.internal,},NodeAddress{Type:Hostname,Address:bootstrap-e2e-master.c.k8s-jkns-e2e-bazel.internal,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:0f6a701a3a68bf65704b9ae5bf250e3c,SystemUUID:0f6a701a-3a68-bf65-704b-9ae5bf250e3c,BootID:581d7fac-a420-439d-9fa2-9116a8f648c9,KernelVersion:5.10.123+,OSImage:Container-Optimized OS from Google,ContainerRuntimeVersion:containerd://1.7.0-rc.2-6-g29e10a192,KubeletVersion:v1.27.0-alpha.3.188+2bac225e4275da,KubeProxyVersion:v1.27.0-alpha.3.188+2bac225e4275da,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/kube-apiserver-amd64:v1.27.0-alpha.3.188_2bac225e4275da],SizeBytes:120421731,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager-amd64:v1.27.0-alpha.3.188_2bac225e4275da],SizeBytes:112795133,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[registry.k8s.io/kube-scheduler-amd64:v1.27.0-alpha.3.188_2bac225e4275da],SizeBytes:58609136,},ContainerImage{Names:[gke.gcr.io/prometheus-to-sd@sha256:e739643c3939ba0b161425f45a1989eedfc4a3b166db9a7100863296b4c70510 gke.gcr.io/prometheus-to-sd:v0.11.1-gke.1],SizeBytes:48742566,},ContainerImage{Names:[gcr.io/k8s-ingress-image-push/ingress-gce-glbc-amd64@sha256:5db27383add6d9f4ebdf0286409ac31f7f5d273690204b341a4e37998917693b gcr.io/k8s-ingress-image-push/ingress-gce-glbc-amd64:v1.20.1],SizeBytes:36598135,},ContainerImage{Names:[registry.k8s.io/addon-manager/kube-addon-manager@sha256:49cc4e6e4a3745b427ce14b0141476ab339bb65c6bc05033019e046c8727dcb0 registry.k8s.io/addon-manager/kube-addon-manager:v9.1.6],SizeBytes:30464183,},ContainerImage{Names:[registry.k8s.io/kas-network-proxy/proxy-server@sha256:b1389e7014425a1752aac55f5043ef4c52edaef0e223bf4d48ed1324e298087c registry.k8s.io/kas-network-proxy/proxy-server:v0.1.1],SizeBytes:21875112,},ContainerImage{Names:[registry.k8s.io/metadata-proxy@sha256:e914645f22e946bce5165737e1b244e0a296ad1f0f81a9531adc57af2780978a registry.k8s.io/metadata-proxy:v0.1.12],SizeBytes:5301657,},ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:311286,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
I0307 22:56:52.978]   Mar  7 22:56:49.751: INFO: 
I0307 22:56:52.978]   Logging kubelet events for node bootstrap-e2e-master
I0307 22:56:52.978]   Mar  7 22:56:49.793: INFO: 
I0307 22:56:52.978]   Logging pods the kubelet thinks is on node bootstrap-e2e-master
I0307 22:56:52.978]   Mar  7 22:56:49.896: INFO: etcd-server-bootstrap-e2e-master started at 2023-03-07 22:48:39 +0000 UTC (0+1 container statuses recorded)
I0307 22:56:52.978]   Mar  7 22:56:49.896: INFO: 	Container etcd-container ready: true, restart count 0
I0307 22:56:52.978]   Mar  7 22:56:49.896: INFO: konnectivity-server-bootstrap-e2e-master started at 2023-03-07 22:48:39 +0000 UTC (0+1 container statuses recorded)
I0307 22:56:52.979]   Mar  7 22:56:49.896: INFO: 	Container konnectivity-server-container ready: true, restart count 0
I0307 22:56:52.979]   Mar  7 22:56:49.896: INFO: kube-addon-manager-bootstrap-e2e-master started at 2023-03-07 22:48:55 +0000 UTC (0+1 container statuses recorded)
I0307 22:56:52.979]   Mar  7 22:56:49.896: INFO: 	Container kube-addon-manager ready: true, restart count 0
I0307 22:56:52.979]   Mar  7 22:56:49.896: INFO: kube-apiserver-bootstrap-e2e-master started at 2023-03-07 22:48:39 +0000 UTC (0+1 container statuses recorded)
I0307 22:56:52.979]   Mar  7 22:56:49.896: INFO: 	Container kube-apiserver ready: true, restart count 0
I0307 22:56:52.979]   Mar  7 22:56:49.896: INFO: kube-controller-manager-bootstrap-e2e-master started at 2023-03-07 22:48:39 +0000 UTC (0+1 container statuses recorded)
I0307 22:56:52.979]   Mar  7 22:56:49.896: INFO: 	Container kube-controller-manager ready: true, restart count 1
I0307 22:56:52.979]   Mar  7 22:56:49.896: INFO: kube-scheduler-bootstrap-e2e-master started at 2023-03-07 22:48:39 +0000 UTC (0+1 container statuses recorded)
I0307 22:56:52.979]   Mar  7 22:56:49.896: INFO: 	Container kube-scheduler ready: true, restart count 0
I0307 22:56:52.979]   Mar  7 22:56:49.896: INFO: etcd-server-events-bootstrap-e2e-master started at 2023-03-07 22:48:39 +0000 UTC (0+1 container statuses recorded)
I0307 22:56:52.979]   Mar  7 22:56:49.896: INFO: 	Container etcd-container ready: true, restart count 0
I0307 22:56:52.980]   Mar  7 22:56:49.896: INFO: l7-lb-controller-bootstrap-e2e-master started at 2023-03-07 22:48:55 +0000 UTC (0+1 container statuses recorded)
I0307 22:56:52.980]   Mar  7 22:56:49.896: INFO: 	Container l7-lb-controller ready: true, restart count 3
I0307 22:56:52.980]   Mar  7 22:56:49.896: INFO: metadata-proxy-v0.1-8xc8h started at 2023-03-07 22:49:21 +0000 UTC (0+2 container statuses recorded)
I0307 22:56:52.980]   Mar  7 22:56:49.896: INFO: 	Container metadata-proxy ready: true, restart count 0
I0307 22:56:52.980]   Mar  7 22:56:49.896: INFO: 	Container prometheus-to-sd-exporter ready: true, restart count 0
I0307 22:56:52.980]   Mar  7 22:56:50.116: INFO: 
I0307 22:56:52.980]   Latency metrics for node bootstrap-e2e-master
I0307 22:56:52.980]   Mar  7 22:56:50.116: INFO: 
I0307 22:56:52.980]   Logging node info for node bootstrap-e2e-minion-group-h4ft
I0307 22:56:52.980]   Mar  7 22:56:50.236: INFO: Node Info: &Node{ObjectMeta:{bootstrap-e2e-minion-group-h4ft    4d15aad5-e48d-43dc-a8ab-c9e41a8799b8 1762 0 2023-03-07 22:49:25 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:n1-standard-2 beta.kubernetes.io/os:linux cloud.google.com/metadata-proxy-ready:true failure-domain.beta.kubernetes.io/region:us-west1 failure-domain.beta.kubernetes.io/zone:us-west1-b kubernetes.io/arch:amd64 kubernetes.io/hostname:bootstrap-e2e-minion-group-h4ft kubernetes.io/os:linux node.kubernetes.io/instance-type:n1-standard-2 topology.kubernetes.io/region:us-west1 topology.kubernetes.io/zone:us-west1-b] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-03-07 22:49:25 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:cloud.google.com/metadata-proxy-ready":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2023-03-07 22:49:26 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"10.64.1.0/24\"":{}}}} } {kube-controller-manager Update v1 2023-03-07 22:49:42 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}}}} status} {kubelet Update v1 2023-03-07 22:51:58 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status} {node-problem-detector Update v1 2023-03-07 22:54:30 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"CorruptDockerOverlay2\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentContainerdRestart\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentDockerRestart\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentKubeletRestart\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentUnregisterNetDevice\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"KernelDeadlock\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"ReadonlyFilesystem\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status}]},Spec:NodeSpec{PodCIDR:10.64.1.0/24,DoNotUseExternalID:,ProviderID:gce://k8s-jkns-e2e-bazel/us-west1-b/bootstrap-e2e-minion-group-h4ft,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.64.1.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-gce-pd: {{127 0} {<nil>} 127 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{101203873792 0} {<nil>} 98831908Ki BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7812632576 0} {<nil>} 7629524Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-gce-pd: {{127 0} {<nil>} 127 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{91083486262 0} {<nil>} 91083486262 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7550488576 0} {<nil>} 7373524Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:ReadonlyFilesystem,Status:False,LastHeartbeatTime:2023-03-07 22:54:30 +0000 UTC,LastTransitionTime:2023-03-07 22:49:29 +0000 UTC,Reason:FilesystemIsNotReadOnly,Message:Filesystem is not read-only,},NodeCondition{Type:CorruptDockerOverlay2,Status:False,LastHeartbeatTime:2023-03-07 22:54:30 +0000 UTC,LastTransitionTime:2023-03-07 22:49:29 +0000 UTC,Reason:NoCorruptDockerOverlay2,Message:docker overlay2 is functioning properly,},NodeCondition{Type:FrequentUnregisterNetDevice,Status:False,LastHeartbeatTime:2023-03-07 22:54:30 +0000 UTC,LastTransitionTime:2023-03-07 22:49:29 +0000 UTC,Reason:NoFrequentUnregisterNetDevice,Message:node is functioning properly,},NodeCondition{Type:FrequentKubeletRestart,Status:False,LastHeartbeatTime:2023-03-07 22:54:30 +0000 UTC,LastTransitionTime:2023-03-07 22:49:29 +0000 UTC,Reason:NoFrequentKubeletRestart,Message:kubelet is functioning properly,},NodeCondition{Type:FrequentDockerRestart,Status:False,LastHeartbeatTime:2023-03-07 22:54:30 +0000 UTC,LastTransitionTime:2023-03-07 22:49:29 +0000 UTC,Reason:NoFrequentDockerRestart,Message:docker is functioning properly,},NodeCondition{Type:FrequentContainerdRestart,Status:False,LastHeartbeatTime:2023-03-07 22:54:30 +0000 UTC,LastTransitionTime:2023-03-07 22:49:29 +0000 UTC,Reason:NoFrequentContainerdRestart,Message:containerd is functioning properly,},NodeCondition{Type:KernelDeadlock,Status:False,LastHeartbeatTime:2023-03-07 22:54:30 +0000 UTC,LastTransitionTime:2023-03-07 22:49:29 +0000 UTC,Reason:KernelHasNoDeadlock,Message:kernel has no deadlock,},NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-07 22:49:42 +0000 UTC,LastTransitionTime:2023-03-07 22:49:42 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-07 22:51:58 +0000 UTC,LastTransitionTime:2023-03-07 22:49:25 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-07 22:51:58 +0000 UTC,LastTransitionTime:2023-03-07 22:49:25 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-07 22:51:58 +0000 UTC,LastTransitionTime:2023-03-07 22:49:25 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-07 22:51:58 +0000 UTC,LastTransitionTime:2023-03-07 22:49:26 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.138.0.3,},NodeAddress{Type:ExternalIP,Address:34.82.33.24,},NodeAddress{Type:InternalDNS,Address:bootstrap-e2e-minion-group-h4ft.c.k8s-jkns-e2e-bazel.internal,},NodeAddress{Type:Hostname,Address:bootstrap-e2e-minion-group-h4ft.c.k8s-jkns-e2e-bazel.internal,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:9f667b52414f953525f37ab6f912279c,SystemUUID:9f667b52-414f-9535-25f3-7ab6f912279c,BootID:cd68bd53-b051-4eb5-b59d-7129d7624d6c,KernelVersion:5.15.65+,OSImage:Container-Optimized OS from Google,ContainerRuntimeVersion:containerd://1.7.0-rc.2-6-g29e10a192,KubeletVersion:v1.27.0-alpha.3.188+2bac225e4275da,KubeProxyVersion:v1.27.0-alpha.3.188+2bac225e4275da,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.27.0-alpha.3.188_2bac225e4275da],SizeBytes:64938336,},ContainerImage{Names:[gke.gcr.io/prometheus-to-sd@sha256:e739643c3939ba0b161425f45a1989eedfc4a3b166db9a7100863296b4c70510 gke.gcr.io/prometheus-to-sd:v0.11.1-gke.1],SizeBytes:48742566,},ContainerImage{Names:[registry.k8s.io/metrics-server/metrics-server@sha256:6385aec64bb97040a5e692947107b81e178555c7a5b71caa90d733e4130efc10 registry.k8s.io/metrics-server/metrics-server:v0.5.2],SizeBytes:26023008,},ContainerImage{Names:[registry.k8s.io/sig-storage/snapshot-controller@sha256:823c75d0c45d1427f6d850070956d9ca657140a7bbf828381541d1d808475280 registry.k8s.io/sig-storage/snapshot-controller:v6.1.0],SizeBytes:22620891,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:fd636b33485c7826fb20ef0688a83ee0910317dbb6c0c6f3ad14661c1db25def registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.4],SizeBytes:15209393,},ContainerImage{Names:[registry.k8s.io/autoscaling/addon-resizer@sha256:43f129b81d28f0fdd54de6d8e7eacd5728030782e03db16087fc241ad747d3d6 registry.k8s.io/autoscaling/addon-resizer:1.8.14],SizeBytes:10153852,},ContainerImage{Names:[registry.k8s.io/kas-network-proxy/proxy-agent@sha256:939c42e815e6b6af3181f074652c0d18fe429fcee9b49c1392aee7e92887cfef registry.k8s.io/kas-network-proxy/proxy-agent:v0.1.1],SizeBytes:8364694,},ContainerImage{Names:[registry.k8s.io/networking/ingress-gce-404-server-with-metrics-amd64@sha256:7eb7b3cee4d33c10c49893ad3c386232b86d4067de5251294d4c620d6e072b93 registry.k8s.io/networking/ingress-gce-404-server-with-metrics-amd64:v1.10.11],SizeBytes:6463068,},ContainerImage{Names:[registry.k8s.io/metadata-proxy@sha256:e914645f22e946bce5165737e1b244e0a296ad1f0f81a9531adc57af2780978a registry.k8s.io/metadata-proxy:v0.1.12],SizeBytes:5301657,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:311286,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
I0307 22:56:52.981]   Mar  7 22:56:50.236: INFO: 
I0307 22:56:52.981]   Logging kubelet events for node bootstrap-e2e-minion-group-h4ft
I0307 22:56:52.981]   Mar  7 22:56:50.398: INFO: 
I0307 22:56:52.981]   Logging pods the kubelet thinks is on node bootstrap-e2e-minion-group-h4ft
I0307 22:56:52.981]   Mar  7 22:56:50.472: INFO: kube-dns-autoscaler-7b444c59c9-xjdlz started at 2023-03-07 22:49:42 +0000 UTC (0+1 container statuses recorded)
I0307 22:56:52.981]   Mar  7 22:56:50.472: INFO: 	Container autoscaler ready: true, restart count 0
I0307 22:56:52.981]   Mar  7 22:56:50.472: INFO: volume-snapshot-controller-0 started at 2023-03-07 22:49:42 +0000 UTC (0+1 container statuses recorded)
I0307 22:56:52.981]   Mar  7 22:56:50.472: INFO: 	Container volume-snapshot-controller ready: true, restart count 0
I0307 22:56:52.981]   Mar  7 22:56:50.472: INFO: konnectivity-agent-9hp6q started at 2023-03-07 22:49:42 +0000 UTC (0+1 container statuses recorded)
I0307 22:56:52.981]   Mar  7 22:56:50.472: INFO: 	Container konnectivity-agent ready: true, restart count 0
I0307 22:56:52.981]   Mar  7 22:56:50.472: INFO: coredns-55fddfc79-xbcdm started at 2023-03-07 22:49:50 +0000 UTC (0+1 container statuses recorded)
I0307 22:56:52.981]   Mar  7 22:56:50.472: INFO: 	Container coredns ready: true, restart count 0
I0307 22:56:52.981]   Mar  7 22:56:50.472: INFO: metrics-server-v0.5.2-57d47cbf5-qwc4p started at 2023-03-07 22:50:00 +0000 UTC (0+2 container statuses recorded)
I0307 22:56:52.981]   Mar  7 22:56:50.472: INFO: 	Container metrics-server ready: true, restart count 0
I0307 22:56:52.982]   Mar  7 22:56:50.472: INFO: 	Container metrics-server-nanny ready: true, restart count 0
I0307 22:56:52.982]   Mar  7 22:56:50.472: INFO: testpod started at 2023-03-07 22:55:37 +0000 UTC (0+1 container statuses recorded)
I0307 22:56:52.982]   Mar  7 22:56:50.472: INFO: 	Container c1 ready: true, restart count 0
I0307 22:56:52.982]   Mar  7 22:56:50.472: INFO: testpod1 started at 2023-03-07 22:55:26 +0000 UTC (0+1 container statuses recorded)
I0307 22:56:52.982]   Mar  7 22:56:50.472: INFO: 	Container c1 ready: true, restart count 0
I0307 22:56:52.982]   Mar  7 22:56:50.472: INFO: kube-proxy-bootstrap-e2e-minion-group-h4ft started at 2023-03-07 22:49:25 +0000 UTC (0+1 container statuses recorded)
I0307 22:56:52.982]   Mar  7 22:56:50.472: INFO: 	Container kube-proxy ready: true, restart count 0
I0307 22:56:52.982]   Mar  7 22:56:50.472: INFO: testpod started at 2023-03-07 22:55:37 +0000 UTC (0+1 container statuses recorded)
I0307 22:56:52.982]   Mar  7 22:56:50.472: INFO: 	Container c1 ready: true, restart count 0
I0307 22:56:52.982]   Mar  7 22:56:50.472: INFO: testpod started at 2023-03-07 22:55:26 +0000 UTC (0+1 container statuses recorded)
I0307 22:56:52.982]   Mar  7 22:56:50.472: INFO: 	Container c1 ready: true, restart count 0
I0307 22:56:52.982]   Mar  7 22:56:50.472: INFO: testpod2 started at 2023-03-07 22:53:02 +0000 UTC (0+1 container statuses recorded)
I0307 22:56:52.982]   Mar  7 22:56:50.472: INFO: 	Container c2 ready: true, restart count 0
I0307 22:56:52.982]   Mar  7 22:56:50.472: INFO: testpod started at 2023-03-07 22:53:23 +0000 UTC (0+1 container statuses recorded)
I0307 22:56:52.982]   Mar  7 22:56:50.472: INFO: 	Container c1 ready: true, restart count 0
I0307 22:56:52.982]   Mar  7 22:56:50.472: INFO: testpod started at 2023-03-07 22:56:43 +0000 UTC (0+1 container statuses recorded)
I0307 22:56:52.983]   Mar  7 22:56:50.472: INFO: 	Container c1 ready: true, restart count 0
I0307 22:56:52.983]   Mar  7 22:56:50.472: INFO: coredns-55fddfc79-fvl8f started at 2023-03-07 22:49:42 +0000 UTC (0+1 container statuses recorded)
I0307 22:56:52.983]   Mar  7 22:56:50.472: INFO: 	Container coredns ready: true, restart count 0
I0307 22:56:52.983]   Mar  7 22:56:50.472: INFO: l7-default-backend-856d874f49-qx4ww started at 2023-03-07 22:49:42 +0000 UTC (0+1 container statuses recorded)
I0307 22:56:52.983]   Mar  7 22:56:50.472: INFO: 	Container default-http-backend ready: true, restart count 0
I0307 22:56:52.983]   Mar  7 22:56:50.472: INFO: testpod started at 2023-03-07 22:53:43 +0000 UTC (0+1 container statuses recorded)
I0307 22:56:52.983]   Mar  7 22:56:50.472: INFO: 	Container c1 ready: true, restart count 0
I0307 22:56:52.983]   Mar  7 22:56:50.472: INFO: metadata-proxy-v0.1-ww7st started at 2023-03-07 22:49:26 +0000 UTC (0+2 container statuses recorded)
I0307 22:56:52.983]   Mar  7 22:56:50.472: INFO: 	Container metadata-proxy ready: true, restart count 0
I0307 22:56:52.983]   Mar  7 22:56:50.472: INFO: 	Container prometheus-to-sd-exporter ready: true, restart count 0
I0307 22:56:52.983]   Mar  7 22:56:50.472: INFO: testpod started at 2023-03-07 22:56:33 +0000 UTC (0+1 container statuses recorded)
I0307 22:56:52.983]   Mar  7 22:56:50.472: INFO: 	Container c1 ready: true, restart count 0
I0307 22:56:52.983]   Mar  7 22:56:50.472: INFO: testpod1 started at 2023-03-07 22:51:48 +0000 UTC (0+1 container statuses recorded)
I0307 22:56:52.983]   Mar  7 22:56:50.472: INFO: 	Container c1 ready: true, restart count 0
I0307 22:56:52.983]   Mar  7 22:56:50.472: INFO: testpod started at 2023-03-07 22:55:06 +0000 UTC (0+1 container statuses recorded)
I0307 22:56:52.983]   Mar  7 22:56:50.472: INFO: 	Container c1 ready: true, restart count 0
I0307 22:56:52.983]   Mar  7 22:56:50.848: INFO: 
I0307 22:56:52.983]   Latency metrics for node bootstrap-e2e-minion-group-h4ft
I0307 22:56:52.983]   STEP: Destroying namespace "pod-resize-errors-5796" for this suite. @ 03/07/23 22:56:50.848
I0307 22:56:52.984]   STEP: dump namespace information after failure @ 03/07/23 22:56:50.892
I0307 22:56:52.984]   STEP: Collecting events from namespace "pod-resize-resource-quota-2161". @ 03/07/23 22:56:50.892
I0307 22:56:52.984]   STEP: Found 0 events. @ 03/07/23 22:56:50.933
I0307 22:56:52.984]   Mar  7 22:56:51.002: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
I0307 22:56:52.984]   Mar  7 22:56:51.002: INFO: 
I0307 22:56:52.984]   Mar  7 22:56:51.044: INFO:

@vinaykul
Copy link
Contributor Author

vinaykul commented Mar 9, 2023 •
edited

Fixed by PR kubernetes/test-infra#28972

Got our first green runs of in-place pod resize e2e tests in CI jobs and baseline is established!

https://testgrid.k8s.io/sig-node-containerd#cos-cgroupv2-inplace-pod-resize-containerd-e2e-serial
https://testgrid.k8s.io/sig-node-containerd#cos-cgroupv1-inplace-pod-resize-containerd-e2e-serial

@vinaykul vinaykul closed this as completed Mar 9, 2023
SIG Node Bugs automation moved this from Triage to Done Mar 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@vinaykul vinaykul

Labels
kind/bug Categorizes issue or PR as related to a bug. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. sig/node Categorizes an issue or PR as relevant to SIG Node.
Projects
SIG Node Bugs
Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Run all in-place pod resize tests in serial vinaykul/kubernetes
2 participants
@vinaykul @k8s-ci-robot