Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Run all in-place pod resize tests in serial #116372

Closed
wants to merge 1 commit into from
Closed

Run all in-place pod resize tests in serial #116372

wants to merge 1 commit into from

Conversation

vinaykul
Copy link
Contributor

@vinaykul vinaykul commented Mar 8, 2023

What type of PR is this?

/kind bug
/kind failing-test

What this PR does / why we need it: In-place pod resize CI tests were re-enabled in PR kubernetes/test-infra#28928 following merge of PR #102884 These tests are failing and the root cause appears to be pods timing out awaiting scheduling, Unschedulable due to Insufficient cpu.

Running these tests in serial may be a quick way to address the problem.

Which issue(s) this PR fixes: #116371 116371

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

@k8s-ci-robot
Copy link
Contributor

Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our release note process to remove it.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added kind/bug Categorizes issue or PR as related to a bug. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. kind/failing-test Categorizes issue or PR as related to a consistently or frequently failing test. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Mar 8, 2023
@k8s-ci-robot k8s-ci-robot added area/test sig/node Categorizes an issue or PR as relevant to SIG Node. sig/testing Categorizes an issue or PR as relevant to SIG Testing. labels Mar 8, 2023
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: vinaykul
Once this PR has been reviewed and has the lgtm label, please assign sergeykanzhelev for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. label Mar 8, 2023
@k8s-ci-robot
Copy link
Contributor

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added the needs-priority Indicates a PR lacks a `priority/foo` label and requires one. label Mar 8, 2023
@vinaykul
Copy link
Contributor Author

vinaykul commented Mar 8, 2023

/test pull-kubernetes-e2e-gce-cos-alpha-features

@vinaykul
Copy link
Contributor Author

vinaykul commented Mar 8, 2023

/assign @bobbypage

Hi David,

Could you please look at this small change and LMK if [Serial] does what I think it does?

IMO, the issue I see in the in-place pod resize CI that was enabled yesterday is multiple tests running in parallel were eating into the limited allocatable CPU on the system (2000m) and causing failures.

An example from a failed test log in
https://storage.googleapis.com/kubernetes-jenkins/logs/ci-cos-cgroupv2-inplace-pod-resize-containerd-main-e2e-gce/1633236905220902912/build-log.txt

I0307 22:56:52.959] ------------------------------
I0307 22:56:52.959] • [FAILED] [304.734 seconds]
I0307 22:56:52.960] [sig-node] Pod InPlace Resize Container [Feature:InPlacePodVerticalScaling] [It] Burstable QoS pod, one container with cpu & memory requests + limits - increase CPU limits only
I0307 22:56:52.960] test/e2e/node/pod_resize.go:1206
I0307 22:56:52.960] 
I0307 22:56:52.960]   Timeline >>
I0307 22:56:52.960]   STEP: Creating a kubernetes client @ 03/07/23 22:51:48.22
I0307 22:56:52.960]   Mar  7 22:51:48.220: INFO: >>> kubeConfig: /workspace/.kube/config
I0307 22:56:52.960]   STEP: Building a namespace api object, basename pod-resize @ 03/07/23 22:51:48.223
I0307 22:56:52.960]   STEP: Waiting for a default service account to be provisioned in namespace @ 03/07/23 22:51:48.443
I0307 22:56:52.960]   STEP: Waiting for kube-root-ca.crt to be provisioned in namespace @ 03/07/23 22:51:48.53
I0307 22:56:52.960]   STEP: Creating a kubernetes client @ 03/07/23 22:51:48.614
I0307 22:56:52.960]   Mar  7 22:51:48.614: INFO: >>> kubeConfig: /workspace/.kube/config
I0307 22:56:52.960]   STEP: Building a namespace api object, basename pod-resize-resource-quota @ 03/07/23 22:51:48.616
I0307 22:56:52.960]   STEP: Waiting for a default service account to be provisioned in namespace @ 03/07/23 22:51:48.837
I0307 22:56:52.960]   STEP: Waiting for kube-root-ca.crt to be provisioned in namespace @ 03/07/23 22:51:48.924
I0307 22:56:52.960]   STEP: Creating a kubernetes client @ 03/07/23 22:51:49.014
I0307 22:56:52.960]   Mar  7 22:51:49.014: INFO: >>> kubeConfig: /workspace/.kube/config
I0307 22:56:52.960]   STEP: Building a namespace api object, basename pod-resize-errors @ 03/07/23 22:51:49.016
I0307 22:56:52.960]   STEP: Waiting for a default service account to be provisioned in namespace @ 03/07/23 22:51:49.182
I0307 22:56:52.960]   STEP: Waiting for kube-root-ca.crt to be provisioned in namespace @ 03/07/23 22:51:49.264
I0307 22:56:52.960]   STEP: creating pod @ 03/07/23 22:51:49.347
I0307 22:56:52.960]   Automatically polling progress:
I0307 22:56:52.960]     [sig-node] Pod InPlace Resize Container [Feature:InPlacePodVerticalScaling] Burstable QoS pod, one container with cpu & memory requests + limits - increase CPU limits only (Spec Runtime: 5m1.128s)
I0307 22:56:52.960]       test/e2e/node/pod_resize.go:1206
I0307 22:56:52.961]       In [It] (Node Runtime: 5m0.001s)
I0307 22:56:52.961]         test/e2e/node/pod_resize.go:1206
I0307 22:56:52.961]         At [By Step] creating pod (Step Runtime: 5m0.001s)
I0307 22:56:52.961]           test/e2e/node/pod_resize.go:1215
I0307 22:56:52.961] 
I0307 22:56:52.961]         Spec Goroutine
I0307 22:56:52.961]         goroutine 915 [select]
I0307 22:56:52.961]           k8s.io/kubernetes/vendor/github.com/onsi/gomega/internal.(*AsyncAssertion).match(0xc00003a620, {0x7f0c650?, 0xc000a6c8b0}, 0x1, {0x0, 0x0, 0x0})
I0307 22:56:52.961]             vendor/github.com/onsi/gomega/internal/async_assertion.go:530
I0307 22:56:52.961]           k8s.io/kubernetes/vendor/github.com/onsi/gomega/internal.(*AsyncAssertion).Should(0xc00003a620, {0x7f0c650, 0xc000a6c8b0}, {0x0, 0x0, 0x0})
I0307 22:56:52.961]             vendor/github.com/onsi/gomega/internal/async_assertion.go:145
I0307 22:56:52.961]           k8s.io/kubernetes/test/e2e/framework.asyncAssertion.Should({{0x7f32a44a2c18, 0xc0019511c0}, {0xc000a6c8a0, 0x1, 0x1}, 0x45d964b800, 0x77359400, 0x0}, {0x7f0c650, 0xc000a6c8b0})
I0307 22:56:52.961]             test/e2e/framework/expect.go:234
I0307 22:56:52.961]           k8s.io/kubernetes/test/e2e/framework/pod.WaitForPodCondition({0x7f32a44a2c18, 0xc0019511c0}, {0x7f62618?, 0xc0013fcd00?}, {0xc0011256c0, 0xf}, {0xc005463ef3, 0x7}, {0x7581506, 0x11}, ...)
I0307 22:56:52.961]             test/e2e/framework/pod/wait.go:228
I0307 22:56:52.961]           k8s.io/kubernetes/test/e2e/framework/pod.WaitTimeoutForPodReadyInNamespace({0x7f32a44a2c18?, 0xc0019511c0?}, {0x7f62618?, 0xc0013fcd00?}, {0xc005463ef3?, 0x0?}, {0xc0011256c0?, 0x2aac985?}, 0x0?)
I0307 22:56:52.961]             test/e2e/framework/pod/wait.go:492
I0307 22:56:52.961]           k8s.io/kubernetes/test/e2e/framework/pod.(*PodClient).CreateSync(0xc000013cb0, {0x7f32a44a2c18, 0xc0019511c0}, 0x7?)
I0307 22:56:52.961]             test/e2e/framework/pod/pod_client.go:106
I0307 22:56:52.961]         > k8s.io/kubernetes/test/e2e/node.doPodResizeTests.func2({0x7f32a44a2c18, 0xc0019511c0})
I0307 22:56:52.961]             test/e2e/node/pod_resize.go:1216
I0307 22:56:52.961]           k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x7f2c708?, 0xc0019511c0})
I0307 22:56:52.961]             vendor/github.com/onsi/ginkgo/v2/internal/node.go:452
I0307 22:56:52.962]           k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
I0307 22:56:52.962]             vendor/github.com/onsi/ginkgo/v2/internal/suite.go:854
I0307 22:56:52.962]           k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/v2/internal.(*Suite).runNode
I0307 22:56:52.962]             vendor/github.com/onsi/ginkgo/v2/internal/suite.go:841
I0307 22:56:52.962] 
I0307 22:56:52.962]         Begin Additional Progress Reports >>
I0307 22:56:52.962]           expected pod to be running and ready, got instead:
I0307 22:56:52.962]               <*v1.Pod | 0xc0020c6480>: 
I0307 22:56:52.962]                   metadata:
I0307 22:56:52.962]                     creationTimestamp: "2023-03-07T22:51:49Z"
I0307 22:56:52.962]                     labels:
I0307 22:56:52.962]                       name: fooPod
I0307 22:56:52.962]                       time: "347833634"
I0307 22:56:52.962]                     managedFields:
I0307 22:56:52.962]                     - apiVersion: v1
I0307 22:56:52.962]                       fieldsType: FieldsV1
I0307 22:56:52.962]                       fieldsV1:
I0307 22:56:52.962]                         f:metadata:
I0307 22:56:52.963]                           f:labels:
I0307 22:56:52.963]                             .: {}
I0307 22:56:52.963]                             f:name: {}
I0307 22:56:52.963]                             f:time: {}
I0307 22:56:52.963]                         f:spec:
I0307 22:56:52.963]                           f:containers:
I0307 22:56:52.963]                             k:{"name":"c1"}:
I0307 22:56:52.963]                               .: {}
I0307 22:56:52.963]                               f:args: {}
I0307 22:56:52.963]                               f:command: {}
I0307 22:56:52.963]                               f:image: {}
I0307 22:56:52.963]                               f:imagePullPolicy: {}
I0307 22:56:52.963]                               f:name: {}
I0307 22:56:52.964]                               f:resizePolicy: {}
I0307 22:56:52.964]                               f:resources:
I0307 22:56:52.964]                                 .: {}
I0307 22:56:52.964]                                 f:limits:
I0307 22:56:52.964]                                   .: {}
I0307 22:56:52.964]                                   f:cpu: {}
I0307 22:56:52.964]                                   f:memory: {}
I0307 22:56:52.964]                                 f:requests:
I0307 22:56:52.964]                                   .: {}
I0307 22:56:52.964]                                   f:cpu: {}
I0307 22:56:52.964]                                   f:memory: {}
I0307 22:56:52.964]                               f:securityContext:
I0307 22:56:52.964]                                 .: {}
I0307 22:56:52.965]                                 f:allowPrivilegeEscalation: {}
I0307 22:56:52.965]                                 f:capabilities:
I0307 22:56:52.965]                                   .: {}
I0307 22:56:52.965]                                   f:drop: {}
I0307 22:56:52.965]                                 f:privileged: {}
I0307 22:56:52.965]                                 f:runAsNonRoot: {}
I0307 22:56:52.965]                                 f:runAsUser: {}
I0307 22:56:52.965]                                 f:seccompProfile:
I0307 22:56:52.965]                                   .: {}
I0307 22:56:52.965]                                   f:type: {}
I0307 22:56:52.965]                               f:terminationMessagePath: {}
I0307 22:56:52.965]                               f:terminationMessagePolicy: {}
I0307 22:56:52.965]                           f:dnsPolicy: {}
I0307 22:56:52.965]                           f:enableServiceLinks: {}
I0307 22:56:52.966]                           f:restartPolicy: {}
I0307 22:56:52.966]                           f:schedulerName: {}
I0307 22:56:52.966]                           f:securityContext: {}
I0307 22:56:52.966]                           f:terminationGracePeriodSeconds: {}
I0307 22:56:52.966]                       manager: e2e.test
I0307 22:56:52.966]                       operation: Update
I0307 22:56:52.966]                       time: "2023-03-07T22:51:49Z"
I0307 22:56:52.966]                     - apiVersion: v1
I0307 22:56:52.966]                       fieldsType: FieldsV1
I0307 22:56:52.966]                       fieldsV1:
I0307 22:56:52.966]                         f:status:
I0307 22:56:52.966]                           f:conditions:
I0307 22:56:52.967]                             .: {}
I0307 22:56:52.967]                             k:{"type":"PodScheduled"}:
I0307 22:56:52.967]                               .: {}
I0307 22:56:52.967]                               f:lastProbeTime: {}
I0307 22:56:52.967]                               f:lastTransitionTime: {}
I0307 22:56:52.967]                               f:message: {}
I0307 22:56:52.967]                               f:reason: {}
I0307 22:56:52.967]                               f:status: {}
I0307 22:56:52.967]                               f:type: {}
I0307 22:56:52.967]                       manager: kube-scheduler
I0307 22:56:52.967]                       operation: Update
I0307 22:56:52.967]                       subresource: status
I0307 22:56:52.968]                       time: "2023-03-07T22:51:49Z"
I0307 22:56:52.968]                     name: testpod
I0307 22:56:52.968]                     namespace: pod-resize-2100
I0307 22:56:52.968]                     resourceVersion: "1161"
I0307 22:56:52.968]                     uid: bade723e-7263-4038-8bc3-e46390517448
I0307 22:56:52.968]                   spec:
I0307 22:56:52.968]                     containers:
I0307 22:56:52.968]                     - args:
I0307 22:56:52.968]                       - -c
I0307 22:56:52.968]                       - trap exit TERM; while true; do sleep 1; done
I0307 22:56:52.968]                       command:
I0307 22:56:52.968]                       - /bin/sh
I0307 22:56:52.968]                       image: registry.k8s.io/e2e-test-images/busybox:1.29-4
I0307 22:56:52.969]                       imagePullPolicy: IfNotPresent
I0307 22:56:52.969]                       name: c1
I0307 22:56:52.969]                       resizePolicy:
I0307 22:56:52.969]                       - policy: RestartNotRequired
I0307 22:56:52.969]                         resourceName: cpu
I0307 22:56:52.969]                       - policy: RestartNotRequired
I0307 22:56:52.969]                         resourceName: memory
I0307 22:56:52.969]                       resources:
I0307 22:56:52.969]                         limits:
I0307 22:56:52.969]                           cpu: 400m
I0307 22:56:52.969]                           memory: 500Mi
I0307 22:56:52.969]                         requests:
I0307 22:56:52.969]                           cpu: 200m
I0307 22:56:52.970]                           memory: 250Mi
I0307 22:56:52.970]                       securityContext:
I0307 22:56:52.970]                         allowPrivilegeEscalation: false
I0307 22:56:52.970]                         capabilities:
I0307 22:56:52.970]                           drop:
I0307 22:56:52.970]                           - ALL
I0307 22:56:52.970]                         privileged: false
I0307 22:56:52.970]                         runAsNonRoot: true
I0307 22:56:52.970]                         runAsUser: 1001
I0307 22:56:52.970]                         seccompProfile:
I0307 22:56:52.970]                           type: RuntimeDefault
I0307 22:56:52.970]                       terminationMessagePath: /dev/termination-log
I0307 22:56:52.970]                       terminationMessagePolicy: File
I0307 22:56:52.971]                       volumeMounts:
I0307 22:56:52.971]                       - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
I0307 22:56:52.971]                         name: kube-api-access-xtrk7
I0307 22:56:52.971]                         readOnly: true
I0307 22:56:52.971]                     dnsPolicy: ClusterFirst
I0307 22:56:52.971]                     enableServiceLinks: true
I0307 22:56:52.971]                     preemptionPolicy: PreemptLowerPriority
I0307 22:56:52.971]                     priority: 0
I0307 22:56:52.971]                     restartPolicy: OnFailure
I0307 22:56:52.971]                     schedulerName: default-scheduler
I0307 22:56:52.972]                     securityContext: {}
I0307 22:56:52.972]                     serviceAccount: default
I0307 22:56:52.972]                     serviceAccountName: default
I0307 22:56:52.972]                     terminationGracePeriodSeconds: 30
I0307 22:56:52.972]                     tolerations:
I0307 22:56:52.972]                     - effect: NoExecute
I0307 22:56:52.972]                       key: node.kubernetes.io/not-ready
I0307 22:56:52.972]                       operator: Exists
I0307 22:56:52.972]                       tolerationSeconds: 300
I0307 22:56:52.972]                     - effect: NoExecute
I0307 22:56:52.973]                       key: node.kubernetes.io/unreachable
I0307 22:56:52.973]                       operator: Exists
I0307 22:56:52.973]                       tolerationSeconds: 300
I0307 22:56:52.973]                     volumes:
I0307 22:56:52.973]                     - name: kube-api-access-xtrk7
I0307 22:56:52.973]                       projected:
I0307 22:56:52.973]                         defaultMode: 420
I0307 22:56:52.973]                         sources:
I0307 22:56:52.973]                         - serviceAccountToken:
I0307 22:56:52.973]                             expirationSeconds: 3607
I0307 22:56:52.973]                             path: token
I0307 22:56:52.974]                         - configMap:
I0307 22:56:52.974]                             items:
I0307 22:56:52.974]                             - key: ca.crt
I0307 22:56:52.974]                               path: ca.crt
I0307 22:56:52.974]                             name: kube-root-ca.crt
I0307 22:56:52.974]                         - downwardAPI:
I0307 22:56:52.974]                             items:
I0307 22:56:52.974]                             - fieldRef:
I0307 22:56:52.974]                                 apiVersion: v1
I0307 22:56:52.974]                                 fieldPath: metadata.namespace
I0307 22:56:52.975]                               path: namespace
I0307 22:56:52.975]                   status:
I0307 22:56:52.975]                     conditions:
I0307 22:56:52.975]                     - lastProbeTime: null
I0307 22:56:52.975]                       lastTransitionTime: "2023-03-07T22:51:49Z"
I0307 22:56:52.975]                       message: '0/2 nodes are available: 1 Insufficient cpu, 1 node(s) were unschedulable.
I0307 22:56:52.975]                         preemption: 0/2 nodes are available: 1 No preemption victims found for incoming
I0307 22:56:52.975]                         pod, 1 Preemption is not helpful for scheduling..'
I0307 22:56:52.975]                       reason: Unschedulable
I0307 22:56:52.975]                       status: "False"
I0307 22:56:52.975]                       type: PodScheduled
I0307 22:56:52.975]                     phase: Pending
I0307 22:56:52.976]                     qosClass: Burstable
I0307 22:56:52.976]         << End Additional Progress Reports
I0307 22:56:52.976]   Mar  7 22:56:49.410: INFO: Failed inside E2E framework:
I0307 22:56:52.976]       k8s.io/kubernetes/test/e2e/framework/pod.WaitForPodCondition({0x7f32a44a2c18, 0xc0019511c0}, {0x7f62618?, 0xc0013fcd00?}, {0xc0011256c0, 0xf}, {0xc005463ef3, 0x7}, {0x7581506, 0x11}, ...)
I0307 22:56:52.976]       	test/e2e/framework/pod/wait.go:228 +0x25f
I0307 22:56:52.976]       k8s.io/kubernetes/test/e2e/framework/pod.WaitTimeoutForPodReadyInNamespace({0x7f32a44a2c18?, 0xc0019511c0?}, {0x7f62618?, 0xc0013fcd00?}, {0xc005463ef3?, 0x0?}, {0xc0011256c0?, 0x2aac985?}, 0x0?)
I0307 22:56:52.976]       	test/e2e/framework/pod/wait.go:492 +0x75
I0307 22:56:52.976]       k8s.io/kubernetes/test/e2e/framework/pod.(*PodClient).CreateSync(0xc000013cb0, {0x7f32a44a2c18, 0xc0019511c0}, 0x7?)
I0307 22:56:52.976]       	test/e2e/framework/pod/pod_client.go:106 +0x97
I0307 22:56:52.976]       k8s.io/kubernetes/test/e2e/node.doPodResizeTests.func2({0x7f32a44a2c18, 0xc0019511c0})
I0307 22:56:52.976]       	test/e2e/node/pod_resize.go:1216 +0x29d
I0307 22:56:52.976]   [FAILED] in [It] - test/e2e/framework/pod/pod_client.go:106 @ 03/07/23 22:56:49.41
I0307 22:56:52.977]   Mar  7 22:56:49.410: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
I0307 22:56:52.977]   Mar  7 22:56:49.495: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
I0307 22:56:52.977]   Mar  7 22:56:49.539: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
I0307 22:56:52.977]   STEP: dump namespace information after failure @ 03/07/23 22:56:49.585
I0307 22:56:52.977]   STEP: Collecting events from namespace "pod-resize-errors-5796". @ 03/07/23 22:56:49.585
I0307 22:56:52.977]   STEP: Found 0 events. @ 03/07/23 22:56:49.626
I0307 22:56:52.977]   Mar  7 22:56:49.667: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
I0307 22:56:52.977]   Mar  7 22:56:49.667: INFO: 
I0307 22:56:52.977]   Mar  7 22:56:49.709: INFO: 
I0307 22:56:52.978]   Logging node info for node bootstrap-e2e-master
I0307 22:56:52.978]   Mar  7 22:56:49.751: INFO: Node Info: &Node{ObjectMeta:{bootstrap-e2e-master    f2c12bc2-c08f-4e00-9187-b8cdc2adb9ed 1799 0 2023-03-07 22:49:20 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:n1-standard-1 beta.kubernetes.io/os:linux cloud.google.com/metadata-proxy-ready:true failure-domain.beta.kubernetes.io/region:us-west1 failure-domain.beta.kubernetes.io/zone:us-west1-b kubernetes.io/arch:amd64 kubernetes.io/hostname:bootstrap-e2e-master kubernetes.io/os:linux node.kubernetes.io/instance-type:n1-standard-1 topology.kubernetes.io/region:us-west1 topology.kubernetes.io/zone:us-west1-b] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-03-07 22:49:20 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:cloud.google.com/metadata-proxy-ready":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{},"f:unschedulable":{}}} } {kube-controller-manager Update v1 2023-03-07 22:49:29 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"10.64.0.0/24\"":{}},"f:taints":{}}} } {kube-controller-manager Update v1 2023-03-07 22:49:29 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}}}} status} {kubelet Update v1 2023-03-07 22:54:46 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:10.64.0.0/24,DoNotUseExternalID:,ProviderID:gce://k8s-jkns-e2e-bazel/us-west1-b/bootstrap-e2e-master,Unschedulable:true,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:,Effect:NoSchedule,TimeAdded:<nil>,},Taint{Key:node.kubernetes.io/unschedulable,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.64.0.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-gce-pd: {{127 0} {<nil>} 127 DecimalSI},cpu: {{1 0} {<nil>} 1 DecimalSI},ephemeral-storage: {{16656896000 0} {<nil>} 16266500Ki BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3858370560 0} {<nil>} 3767940Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-gce-pd: {{127 0} {<nil>} 127 DecimalSI},cpu: {{1 0} {<nil>} 1 DecimalSI},ephemeral-storage: {{14991206376 0} {<nil>} 14991206376 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3596226560 0} {<nil>} 3511940Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-07 22:49:29 +0000 UTC,LastTransitionTime:2023-03-07 22:49:29 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-07 22:54:46 +0000 UTC,LastTransitionTime:2023-03-07 22:49:20 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-07 22:54:46 +0000 UTC,LastTransitionTime:2023-03-07 22:49:20 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-07 22:54:46 +0000 UTC,LastTransitionTime:2023-03-07 22:49:20 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-07 22:54:46 +0000 UTC,LastTransitionTime:2023-03-07 22:49:22 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.138.0.2,},NodeAddress{Type:ExternalIP,Address:34.168.107.101,},NodeAddress{Type:InternalDNS,Address:bootstrap-e2e-master.c.k8s-jkns-e2e-bazel.internal,},NodeAddress{Type:Hostname,Address:bootstrap-e2e-master.c.k8s-jkns-e2e-bazel.internal,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:0f6a701a3a68bf65704b9ae5bf250e3c,SystemUUID:0f6a701a-3a68-bf65-704b-9ae5bf250e3c,BootID:581d7fac-a420-439d-9fa2-9116a8f648c9,KernelVersion:5.10.123+,OSImage:Container-Optimized OS from Google,ContainerRuntimeVersion:containerd://1.7.0-rc.2-6-g29e10a192,KubeletVersion:v1.27.0-alpha.3.188+2bac225e4275da,KubeProxyVersion:v1.27.0-alpha.3.188+2bac225e4275da,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/kube-apiserver-amd64:v1.27.0-alpha.3.188_2bac225e4275da],SizeBytes:120421731,},ContainerImage{Names:[registry.k8s.io/kube-controller-manager-amd64:v1.27.0-alpha.3.188_2bac225e4275da],SizeBytes:112795133,},ContainerImage{Names:[registry.k8s.io/etcd@sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83 registry.k8s.io/etcd:3.5.7-0],SizeBytes:101639218,},ContainerImage{Names:[registry.k8s.io/kube-scheduler-amd64:v1.27.0-alpha.3.188_2bac225e4275da],SizeBytes:58609136,},ContainerImage{Names:[gke.gcr.io/prometheus-to-sd@sha256:e739643c3939ba0b161425f45a1989eedfc4a3b166db9a7100863296b4c70510 gke.gcr.io/prometheus-to-sd:v0.11.1-gke.1],SizeBytes:48742566,},ContainerImage{Names:[gcr.io/k8s-ingress-image-push/ingress-gce-glbc-amd64@sha256:5db27383add6d9f4ebdf0286409ac31f7f5d273690204b341a4e37998917693b gcr.io/k8s-ingress-image-push/ingress-gce-glbc-amd64:v1.20.1],SizeBytes:36598135,},ContainerImage{Names:[registry.k8s.io/addon-manager/kube-addon-manager@sha256:49cc4e6e4a3745b427ce14b0141476ab339bb65c6bc05033019e046c8727dcb0 registry.k8s.io/addon-manager/kube-addon-manager:v9.1.6],SizeBytes:30464183,},ContainerImage{Names:[registry.k8s.io/kas-network-proxy/proxy-server@sha256:b1389e7014425a1752aac55f5043ef4c52edaef0e223bf4d48ed1324e298087c registry.k8s.io/kas-network-proxy/proxy-server:v0.1.1],SizeBytes:21875112,},ContainerImage{Names:[registry.k8s.io/metadata-proxy@sha256:e914645f22e946bce5165737e1b244e0a296ad1f0f81a9531adc57af2780978a registry.k8s.io/metadata-proxy:v0.1.12],SizeBytes:5301657,},ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:311286,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
I0307 22:56:52.978]   Mar  7 22:56:49.751: INFO: 
I0307 22:56:52.978]   Logging kubelet events for node bootstrap-e2e-master
I0307 22:56:52.978]   Mar  7 22:56:49.793: INFO: 
I0307 22:56:52.978]   Logging pods the kubelet thinks is on node bootstrap-e2e-master
I0307 22:56:52.978]   Mar  7 22:56:49.896: INFO: etcd-server-bootstrap-e2e-master started at 2023-03-07 22:48:39 +0000 UTC (0+1 container statuses recorded)
I0307 22:56:52.978]   Mar  7 22:56:49.896: INFO: 	Container etcd-container ready: true, restart count 0
I0307 22:56:52.978]   Mar  7 22:56:49.896: INFO: konnectivity-server-bootstrap-e2e-master started at 2023-03-07 22:48:39 +0000 UTC (0+1 container statuses recorded)
I0307 22:56:52.979]   Mar  7 22:56:49.896: INFO: 	Container konnectivity-server-container ready: true, restart count 0
I0307 22:56:52.979]   Mar  7 22:56:49.896: INFO: kube-addon-manager-bootstrap-e2e-master started at 2023-03-07 22:48:55 +0000 UTC (0+1 container statuses recorded)
I0307 22:56:52.979]   Mar  7 22:56:49.896: INFO: 	Container kube-addon-manager ready: true, restart count 0
I0307 22:56:52.979]   Mar  7 22:56:49.896: INFO: kube-apiserver-bootstrap-e2e-master started at 2023-03-07 22:48:39 +0000 UTC (0+1 container statuses recorded)
I0307 22:56:52.979]   Mar  7 22:56:49.896: INFO: 	Container kube-apiserver ready: true, restart count 0
I0307 22:56:52.979]   Mar  7 22:56:49.896: INFO: kube-controller-manager-bootstrap-e2e-master started at 2023-03-07 22:48:39 +0000 UTC (0+1 container statuses recorded)
I0307 22:56:52.979]   Mar  7 22:56:49.896: INFO: 	Container kube-controller-manager ready: true, restart count 1
I0307 22:56:52.979]   Mar  7 22:56:49.896: INFO: kube-scheduler-bootstrap-e2e-master started at 2023-03-07 22:48:39 +0000 UTC (0+1 container statuses recorded)
I0307 22:56:52.979]   Mar  7 22:56:49.896: INFO: 	Container kube-scheduler ready: true, restart count 0
I0307 22:56:52.979]   Mar  7 22:56:49.896: INFO: etcd-server-events-bootstrap-e2e-master started at 2023-03-07 22:48:39 +0000 UTC (0+1 container statuses recorded)
I0307 22:56:52.979]   Mar  7 22:56:49.896: INFO: 	Container etcd-container ready: true, restart count 0
I0307 22:56:52.980]   Mar  7 22:56:49.896: INFO: l7-lb-controller-bootstrap-e2e-master started at 2023-03-07 22:48:55 +0000 UTC (0+1 container statuses recorded)
I0307 22:56:52.980]   Mar  7 22:56:49.896: INFO: 	Container l7-lb-controller ready: true, restart count 3
I0307 22:56:52.980]   Mar  7 22:56:49.896: INFO: metadata-proxy-v0.1-8xc8h started at 2023-03-07 22:49:21 +0000 UTC (0+2 container statuses recorded)
I0307 22:56:52.980]   Mar  7 22:56:49.896: INFO: 	Container metadata-proxy ready: true, restart count 0
I0307 22:56:52.980]   Mar  7 22:56:49.896: INFO: 	Container prometheus-to-sd-exporter ready: true, restart count 0
I0307 22:56:52.980]   Mar  7 22:56:50.116: INFO: 
I0307 22:56:52.980]   Latency metrics for node bootstrap-e2e-master
I0307 22:56:52.980]   Mar  7 22:56:50.116: INFO: 
I0307 22:56:52.980]   Logging node info for node bootstrap-e2e-minion-group-h4ft
I0307 22:56:52.980]   Mar  7 22:56:50.236: INFO: Node Info: &Node{ObjectMeta:{bootstrap-e2e-minion-group-h4ft    4d15aad5-e48d-43dc-a8ab-c9e41a8799b8 1762 0 2023-03-07 22:49:25 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:n1-standard-2 beta.kubernetes.io/os:linux cloud.google.com/metadata-proxy-ready:true failure-domain.beta.kubernetes.io/region:us-west1 failure-domain.beta.kubernetes.io/zone:us-west1-b kubernetes.io/arch:amd64 kubernetes.io/hostname:bootstrap-e2e-minion-group-h4ft kubernetes.io/os:linux node.kubernetes.io/instance-type:n1-standard-2 topology.kubernetes.io/region:us-west1 topology.kubernetes.io/zone:us-west1-b] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] [] [{kubelet Update v1 2023-03-07 22:49:25 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:cloud.google.com/metadata-proxy-ready":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {kube-controller-manager Update v1 2023-03-07 22:49:26 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"10.64.1.0/24\"":{}}}} } {kube-controller-manager Update v1 2023-03-07 22:49:42 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}}}} status} {kubelet Update v1 2023-03-07 22:51:58 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:images":{}}} status} {node-problem-detector Update v1 2023-03-07 22:54:30 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"CorruptDockerOverlay2\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentContainerdRestart\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentDockerRestart\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentKubeletRestart\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentUnregisterNetDevice\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"KernelDeadlock\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"ReadonlyFilesystem\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status}]},Spec:NodeSpec{PodCIDR:10.64.1.0/24,DoNotUseExternalID:,ProviderID:gce://k8s-jkns-e2e-bazel/us-west1-b/bootstrap-e2e-minion-group-h4ft,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.64.1.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-gce-pd: {{127 0} {<nil>} 127 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{101203873792 0} {<nil>} 98831908Ki BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7812632576 0} {<nil>} 7629524Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-gce-pd: {{127 0} {<nil>} 127 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{91083486262 0} {<nil>} 91083486262 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7550488576 0} {<nil>} 7373524Ki BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:ReadonlyFilesystem,Status:False,LastHeartbeatTime:2023-03-07 22:54:30 +0000 UTC,LastTransitionTime:2023-03-07 22:49:29 +0000 UTC,Reason:FilesystemIsNotReadOnly,Message:Filesystem is not read-only,},NodeCondition{Type:CorruptDockerOverlay2,Status:False,LastHeartbeatTime:2023-03-07 22:54:30 +0000 UTC,LastTransitionTime:2023-03-07 22:49:29 +0000 UTC,Reason:NoCorruptDockerOverlay2,Message:docker overlay2 is functioning properly,},NodeCondition{Type:FrequentUnregisterNetDevice,Status:False,LastHeartbeatTime:2023-03-07 22:54:30 +0000 UTC,LastTransitionTime:2023-03-07 22:49:29 +0000 UTC,Reason:NoFrequentUnregisterNetDevice,Message:node is functioning properly,},NodeCondition{Type:FrequentKubeletRestart,Status:False,LastHeartbeatTime:2023-03-07 22:54:30 +0000 UTC,LastTransitionTime:2023-03-07 22:49:29 +0000 UTC,Reason:NoFrequentKubeletRestart,Message:kubelet is functioning properly,},NodeCondition{Type:FrequentDockerRestart,Status:False,LastHeartbeatTime:2023-03-07 22:54:30 +0000 UTC,LastTransitionTime:2023-03-07 22:49:29 +0000 UTC,Reason:NoFrequentDockerRestart,Message:docker is functioning properly,},NodeCondition{Type:FrequentContainerdRestart,Status:False,LastHeartbeatTime:2023-03-07 22:54:30 +0000 UTC,LastTransitionTime:2023-03-07 22:49:29 +0000 UTC,Reason:NoFrequentContainerdRestart,Message:containerd is functioning properly,},NodeCondition{Type:KernelDeadlock,Status:False,LastHeartbeatTime:2023-03-07 22:54:30 +0000 UTC,LastTransitionTime:2023-03-07 22:49:29 +0000 UTC,Reason:KernelHasNoDeadlock,Message:kernel has no deadlock,},NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2023-03-07 22:49:42 +0000 UTC,LastTransitionTime:2023-03-07 22:49:42 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2023-03-07 22:51:58 +0000 UTC,LastTransitionTime:2023-03-07 22:49:25 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2023-03-07 22:51:58 +0000 UTC,LastTransitionTime:2023-03-07 22:49:25 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2023-03-07 22:51:58 +0000 UTC,LastTransitionTime:2023-03-07 22:49:25 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2023-03-07 22:51:58 +0000 UTC,LastTransitionTime:2023-03-07 22:49:26 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.138.0.3,},NodeAddress{Type:ExternalIP,Address:34.82.33.24,},NodeAddress{Type:InternalDNS,Address:bootstrap-e2e-minion-group-h4ft.c.k8s-jkns-e2e-bazel.internal,},NodeAddress{Type:Hostname,Address:bootstrap-e2e-minion-group-h4ft.c.k8s-jkns-e2e-bazel.internal,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:9f667b52414f953525f37ab6f912279c,SystemUUID:9f667b52-414f-9535-25f3-7ab6f912279c,BootID:cd68bd53-b051-4eb5-b59d-7129d7624d6c,KernelVersion:5.15.65+,OSImage:Container-Optimized OS from Google,ContainerRuntimeVersion:containerd://1.7.0-rc.2-6-g29e10a192,KubeletVersion:v1.27.0-alpha.3.188+2bac225e4275da,KubeProxyVersion:v1.27.0-alpha.3.188+2bac225e4275da,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[registry.k8s.io/kube-proxy-amd64:v1.27.0-alpha.3.188_2bac225e4275da],SizeBytes:64938336,},ContainerImage{Names:[gke.gcr.io/prometheus-to-sd@sha256:e739643c3939ba0b161425f45a1989eedfc4a3b166db9a7100863296b4c70510 gke.gcr.io/prometheus-to-sd:v0.11.1-gke.1],SizeBytes:48742566,},ContainerImage{Names:[registry.k8s.io/metrics-server/metrics-server@sha256:6385aec64bb97040a5e692947107b81e178555c7a5b71caa90d733e4130efc10 registry.k8s.io/metrics-server/metrics-server:v0.5.2],SizeBytes:26023008,},ContainerImage{Names:[registry.k8s.io/sig-storage/snapshot-controller@sha256:823c75d0c45d1427f6d850070956d9ca657140a7bbf828381541d1d808475280 registry.k8s.io/sig-storage/snapshot-controller:v6.1.0],SizeBytes:22620891,},ContainerImage{Names:[registry.k8s.io/coredns/coredns@sha256:017727efcfeb7d053af68e51436ce8e65edbc6ca573720afb4f79c8594036955 registry.k8s.io/coredns/coredns:v1.10.0],SizeBytes:15273057,},ContainerImage{Names:[registry.k8s.io/cpa/cluster-proportional-autoscaler@sha256:fd636b33485c7826fb20ef0688a83ee0910317dbb6c0c6f3ad14661c1db25def registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.4],SizeBytes:15209393,},ContainerImage{Names:[registry.k8s.io/autoscaling/addon-resizer@sha256:43f129b81d28f0fdd54de6d8e7eacd5728030782e03db16087fc241ad747d3d6 registry.k8s.io/autoscaling/addon-resizer:1.8.14],SizeBytes:10153852,},ContainerImage{Names:[registry.k8s.io/kas-network-proxy/proxy-agent@sha256:939c42e815e6b6af3181f074652c0d18fe429fcee9b49c1392aee7e92887cfef registry.k8s.io/kas-network-proxy/proxy-agent:v0.1.1],SizeBytes:8364694,},ContainerImage{Names:[registry.k8s.io/networking/ingress-gce-404-server-with-metrics-amd64@sha256:7eb7b3cee4d33c10c49893ad3c386232b86d4067de5251294d4c620d6e072b93 registry.k8s.io/networking/ingress-gce-404-server-with-metrics-amd64:v1.10.11],SizeBytes:6463068,},ContainerImage{Names:[registry.k8s.io/metadata-proxy@sha256:e914645f22e946bce5165737e1b244e0a296ad1f0f81a9531adc57af2780978a registry.k8s.io/metadata-proxy:v0.1.12],SizeBytes:5301657,},ContainerImage{Names:[registry.k8s.io/e2e-test-images/busybox@sha256:2e0f836850e09b8b7cc937681d6194537a09fbd5f6b9e08f4d646a85128e8937 registry.k8s.io/e2e-test-images/busybox:1.29-4],SizeBytes:731990,},ContainerImage{Names:[registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d registry.k8s.io/pause:3.8],SizeBytes:311286,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:nil,},}
I0307 22:56:52.981]   Mar  7 22:56:50.236: INFO: 
I0307 22:56:52.981]   Logging kubelet events for node bootstrap-e2e-minion-group-h4ft
I0307 22:56:52.981]   Mar  7 22:56:50.398: INFO: 
I0307 22:56:52.981]   Logging pods the kubelet thinks is on node bootstrap-e2e-minion-group-h4ft
I0307 22:56:52.981]   Mar  7 22:56:50.472: INFO: kube-dns-autoscaler-7b444c59c9-xjdlz started at 2023-03-07 22:49:42 +0000 UTC (0+1 container statuses recorded)
I0307 22:56:52.981]   Mar  7 22:56:50.472: INFO: 	Container autoscaler ready: true, restart count 0
I0307 22:56:52.981]   Mar  7 22:56:50.472: INFO: volume-snapshot-controller-0 started at 2023-03-07 22:49:42 +0000 UTC (0+1 container statuses recorded)
I0307 22:56:52.981]   Mar  7 22:56:50.472: INFO: 	Container volume-snapshot-controller ready: true, restart count 0
I0307 22:56:52.981]   Mar  7 22:56:50.472: INFO: konnectivity-agent-9hp6q started at 2023-03-07 22:49:42 +0000 UTC (0+1 container statuses recorded)
I0307 22:56:52.981]   Mar  7 22:56:50.472: INFO: 	Container konnectivity-agent ready: true, restart count 0
I0307 22:56:52.981]   Mar  7 22:56:50.472: INFO: coredns-55fddfc79-xbcdm started at 2023-03-07 22:49:50 +0000 UTC (0+1 container statuses recorded)
I0307 22:56:52.981]   Mar  7 22:56:50.472: INFO: 	Container coredns ready: true, restart count 0
I0307 22:56:52.981]   Mar  7 22:56:50.472: INFO: metrics-server-v0.5.2-57d47cbf5-qwc4p started at 2023-03-07 22:50:00 +0000 UTC (0+2 container statuses recorded)
I0307 22:56:52.981]   Mar  7 22:56:50.472: INFO: 	Container metrics-server ready: true, restart count 0
I0307 22:56:52.982]   Mar  7 22:56:50.472: INFO: 	Container metrics-server-nanny ready: true, restart count 0
I0307 22:56:52.982]   Mar  7 22:56:50.472: INFO: testpod started at 2023-03-07 22:55:37 +0000 UTC (0+1 container statuses recorded)
I0307 22:56:52.982]   Mar  7 22:56:50.472: INFO: 	Container c1 ready: true, restart count 0
I0307 22:56:52.982]   Mar  7 22:56:50.472: INFO: testpod1 started at 2023-03-07 22:55:26 +0000 UTC (0+1 container statuses recorded)
I0307 22:56:52.982]   Mar  7 22:56:50.472: INFO: 	Container c1 ready: true, restart count 0
I0307 22:56:52.982]   Mar  7 22:56:50.472: INFO: kube-proxy-bootstrap-e2e-minion-group-h4ft started at 2023-03-07 22:49:25 +0000 UTC (0+1 container statuses recorded)
I0307 22:56:52.982]   Mar  7 22:56:50.472: INFO: 	Container kube-proxy ready: true, restart count 0
I0307 22:56:52.982]   Mar  7 22:56:50.472: INFO: testpod started at 2023-03-07 22:55:37 +0000 UTC (0+1 container statuses recorded)
I0307 22:56:52.982]   Mar  7 22:56:50.472: INFO: 	Container c1 ready: true, restart count 0
I0307 22:56:52.982]   Mar  7 22:56:50.472: INFO: testpod started at 2023-03-07 22:55:26 +0000 UTC (0+1 container statuses recorded)
I0307 22:56:52.982]   Mar  7 22:56:50.472: INFO: 	Container c1 ready: true, restart count 0
I0307 22:56:52.982]   Mar  7 22:56:50.472: INFO: testpod2 started at 2023-03-07 22:53:02 +0000 UTC (0+1 container statuses recorded)
I0307 22:56:52.982]   Mar  7 22:56:50.472: INFO: 	Container c2 ready: true, restart count 0
I0307 22:56:52.982]   Mar  7 22:56:50.472: INFO: testpod started at 2023-03-07 22:53:23 +0000 UTC (0+1 container statuses recorded)
I0307 22:56:52.982]   Mar  7 22:56:50.472: INFO: 	Container c1 ready: true, restart count 0
I0307 22:56:52.982]   Mar  7 22:56:50.472: INFO: testpod started at 2023-03-07 22:56:43 +0000 UTC (0+1 container statuses recorded)
I0307 22:56:52.983]   Mar  7 22:56:50.472: INFO: 	Container c1 ready: true, restart count 0
I0307 22:56:52.983]   Mar  7 22:56:50.472: INFO: coredns-55fddfc79-fvl8f started at 2023-03-07 22:49:42 +0000 UTC (0+1 container statuses recorded)
I0307 22:56:52.983]   Mar  7 22:56:50.472: INFO: 	Container coredns ready: true, restart count 0
I0307 22:56:52.983]   Mar  7 22:56:50.472: INFO: l7-default-backend-856d874f49-qx4ww started at 2023-03-07 22:49:42 +0000 UTC (0+1 container statuses recorded)
I0307 22:56:52.983]   Mar  7 22:56:50.472: INFO: 	Container default-http-backend ready: true, restart count 0
I0307 22:56:52.983]   Mar  7 22:56:50.472: INFO: testpod started at 2023-03-07 22:53:43 +0000 UTC (0+1 container statuses recorded)
I0307 22:56:52.983]   Mar  7 22:56:50.472: INFO: 	Container c1 ready: true, restart count 0
I0307 22:56:52.983]   Mar  7 22:56:50.472: INFO: metadata-proxy-v0.1-ww7st started at 2023-03-07 22:49:26 +0000 UTC (0+2 container statuses recorded)
I0307 22:56:52.983]   Mar  7 22:56:50.472: INFO: 	Container metadata-proxy ready: true, restart count 0
I0307 22:56:52.983]   Mar  7 22:56:50.472: INFO: 	Container prometheus-to-sd-exporter ready: true, restart count 0
I0307 22:56:52.983]   Mar  7 22:56:50.472: INFO: testpod started at 2023-03-07 22:56:33 +0000 UTC (0+1 container statuses recorded)
I0307 22:56:52.983]   Mar  7 22:56:50.472: INFO: 	Container c1 ready: true, restart count 0
I0307 22:56:52.983]   Mar  7 22:56:50.472: INFO: testpod1 started at 2023-03-07 22:51:48 +0000 UTC (0+1 container statuses recorded)
I0307 22:56:52.983]   Mar  7 22:56:50.472: INFO: 	Container c1 ready: true, restart count 0
I0307 22:56:52.983]   Mar  7 22:56:50.472: INFO: testpod started at 2023-03-07 22:55:06 +0000 UTC (0+1 container statuses recorded)
I0307 22:56:52.983]   Mar  7 22:56:50.472: INFO: 	Container c1 ready: true, restart count 0
I0307 22:56:52.983]   Mar  7 22:56:50.848: INFO: 
I0307 22:56:52.983]   Latency metrics for node bootstrap-e2e-minion-group-h4ft
I0307 22:56:52.983]   STEP: Destroying namespace "pod-resize-errors-5796" for this suite. @ 03/07/23 22:56:50.848
I0307 22:56:52.984]   STEP: dump namespace information after failure @ 03/07/23 22:56:50.892
I0307 22:56:52.984]   STEP: Collecting events from namespace "pod-resize-resource-quota-2161". @ 03/07/23 22:56:50.892
I0307 22:56:52.984]   STEP: Found 0 events. @ 03/07/23 22:56:50.933
I0307 22:56:52.984]   Mar  7 22:56:51.002: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
I0307 22:56:52.984]   Mar  7 22:56:51.002: INFO: 
I0307 22:56:52.984]   Mar  7 22:56:51.044: INFO:

@bart0sh bart0sh added this to Waiting on Author in SIG Node PR Triage Mar 8, 2023
@bart0sh bart0sh moved this from Waiting on Author to Triage in SIG Node PR Triage Mar 8, 2023
@SergeyKanzhelev
Copy link
Member

/assign

@SergeyKanzhelev SergeyKanzhelev moved this from Triage to PRs - Needs Reviewer in SIG Node CI/Test Board Mar 8, 2023
@SergeyKanzhelev SergeyKanzhelev moved this from Triage to Waiting on Author in SIG Node PR Triage Mar 8, 2023
@SergeyKanzhelev SergeyKanzhelev moved this from Waiting on Author to Needs Reviewer in SIG Node PR Triage Mar 8, 2023
@bobbypage
Copy link
Member

bobbypage commented Mar 8, 2023 •
edited

This sounds reasonable to me, but do we already have a pre-existing serial cluster e2e alpha job where this would run?

edit: I found ci-kubernetes-e2e-gci-gce-serial (https://github.com/kubernetes/test-infra/blob/master/config/jobs/kubernetes/sig-cloud-provider/gcp/gcp-gce.yaml#L815-L849) but it does not have alpha feature gates enabled. Perhaps we need to add a new sig node job that uses containerd master + serial + alpha feature gates?

@vinaykul
Copy link
Contributor Author

vinaykul commented Mar 8, 2023

This sounds reasonable to me, but do we already have a pre-existing serial cluster e2e alpha job where this would run?

edit: I found ci-kubernetes-e2e-gci-gce-serial (https://github.com/kubernetes/test-infra/blob/master/config/jobs/kubernetes/sig-cloud-provider/gcp/gcp-gce.yaml#L815-L849) but it does not have alpha feature gates enabled. Perhaps we need to add a new sig node job that uses containerd master + serial + alpha feature gates?

We could just expand this one if it works well for in-place pod resize (and focus it down to cgroupv2 as we phase out of cgv1 to judiciously use resources). My initial thoughts were to remove this once containerd catches up to this feature, but there will likely be other new features that would need this coverage.

@bobbypage
Copy link
Member

bobbypage commented Mar 9, 2023 •
edited

another idea... we already have cos-cgroupv1-inplace-pod-resize-containerd-e2e and cos-cgroupv2-inplace-pod-resize-containerd-e2e. Perhaps we can reuse those? i.e. replace minStartupPods there to --minStartupPods=1 (to ensure it will run in serial) and keep the existing focus of \[Feature:InPlacePodVerticalScaling\] (xref: https://github.com/kubernetes/test-infra/blob/master/config/jobs/kubernetes/sig-node/containerd.yaml#L1407). That way these two jobs will be the primary ones for this feature.

If later we want to add serial + all feature, we can also add another job for that? wdyt?

@vinaykul
Copy link
Contributor Author

vinaykul commented Mar 9, 2023

another idea... we already have cos-cgroupv1-inplace-pod-resize-containerd-e2e and cos-cgroupv2-inplace-pod-resize-containerd-e2e. Perhaps we can reuse those? i.e. replace minStartupPods there to --minStartupPods=1 (to ensure it will run in serial) and keep the existing focus of \[Feature:InPlacePodVerticalScaling\] (xref: https://github.com/kubernetes/test-infra/blob/master/config/jobs/kubernetes/sig-node/containerd.yaml#L1407). That way these two jobs will be the primary ones for this feature.

If later we want to add serial + all feature, we can also add another job for that? wdyt?

I didn't realize that setting minStartupPods=1 is sufficient to get all the tests to run in serial. I'll create a test-infra PR for this shortly.

@vinaykul
Copy link
Contributor Author

vinaykul commented Mar 9, 2023

/hold
Checking whether PR kubernetes/test-infra#28972 might be the right way to address this.

@k8s-ci-robot k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 9, 2023
@vinaykul
Copy link
Contributor Author

vinaykul commented Mar 9, 2023

PR kubernetes/test-infra#28972 happens to be the right fix.

We got our first green runs of in-place pod resize e2e tests in CI jobs and baseline is established!
https://testgrid.k8s.io/sig-node-containerd#cos-cgroupv2-inplace-pod-resize-containerd-e2e-serial
https://testgrid.k8s.io/sig-node-containerd#cos-cgroupv1-inplace-pod-resize-containerd-e2e-serial

/close

@k8s-ci-robot
Copy link
Contributor

@vinaykul: Closed this PR.

In response to this:

PR kubernetes/test-infra#28972 happens to be the right fix.

We got our first green runs of in-place pod resize e2e tests in CI jobs and baseline is established!
https://testgrid.k8s.io/sig-node-containerd#cos-cgroupv2-inplace-pod-resize-containerd-e2e-serial
https://testgrid.k8s.io/sig-node-containerd#cos-cgroupv1-inplace-pod-resize-containerd-e2e-serial

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

SIG Node CI/Test Board automation moved this from PRs - Needs Reviewer to Done Mar 9, 2023
SIG Node PR Triage automation moved this from Needs Reviewer to Done Mar 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@krmayankk krmayankk Awaiting requested review from krmayankk

@SergeyKanzhelev SergeyKanzhelev Awaiting requested review from SergeyKanzhelev

Assignees

@bobbypage bobbypage

@SergeyKanzhelev SergeyKanzhelev

Labels
area/test cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. kind/bug Categorizes issue or PR as related to a bug. kind/failing-test Categorizes issue or PR as related to a consistently or frequently failing test. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. sig/node Categorizes an issue or PR as relevant to SIG Node. sig/testing Categorizes an issue or PR as relevant to SIG Testing. size/S Denotes a PR that changes 10-29 lines, ignoring generated files.
Projects
SIG Node CI/Test Board
Done
SIG Node CI/Test Board
Archived in project
SIG Node PR Triage
Archived in project
SIG Node PR Triage
Done
Milestone
No milestone
4 participants
@vinaykul @k8s-ci-robot @SergeyKanzhelev @bobbypage