[Flaky test] ci-kubernetes-unit

[Nivedita-coder]

Which jobs are flaking?

release-master-blocking

• ci-kubernetes-unit

Which tests are flaking?

• ci-kubernetes-unit.Overall
• k8s.io/kubernetes/pkg/kubelet/nodeshutdown: TestLocalStorage
• k8s.io/kubernetes/pkg/kubelet/nodeshutdown: TestFeatureEnabled
• k8s.io/kubernetes/pkg/schedular/framework/plugins/volumebinding.TestBindPodVolumes
• k8s.io/kubernetes/pkg/schedular/framework/plugins/volumebinding.TestBindPodVolumes/binding-static-pv-succeeds-after-time

Since when has it been flaking?

07-04 22:51 EEST

Testgrid link

https://testgrid.k8s.io/sig-release-master-blocking#ci-kubernetes-unit

Reason for failure (if possible)

{Failed === CONT testing.go:1312: race detected during execution of test FAIL FAIL k8s.io/kubernetes/pkg/kubelet/nodeshutdown 2.384s }

Anything else we need to know?

No response

Relevant SIG(s)

/sig testing
cc @kubernetes/ci-signal

[endocrimes]

/triage accepted
/assign

[MadhavJivrajani]

@pohly maybe you can help provide insights here (at least for the nodeshutdown tests since they are related to race conditions due to logging)!

Logs with details about the race: https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/ci-kubernetes-unit/1544926366053437440

Seems like the race arises due to t being passed to and used for receiving logs here:

kubernetes/vendor/k8s.io/klog/v2/ktesting/setup.go

Line 43 in 2a017f9

func NewTestContext(tl TL) (logr.Logger, context.Context) {

Read happens here:
https://github.com/golang/go/blob/3a7cec27738c51347f4e5a466410a0b2089391b7/src/testing/testing.go#L882

Write:
https://github.com/golang/go/blob/3a7cec27738c51347f4e5a466410a0b2089391b7/src/testing/testing.go#L1424-L1426

[pohly]

This is a dupe of #110854. A revert is pending as a stop-gap solution, but I also expect an updated klog which is more resilient against leaked goroutines (the actual problem) soon.

[shuheiktgw]

Hello @pohly 👋 I'm a 1.25 CI Signal Shadow and would you let us know if this issue should be a blocker for the 1.25.0-alpha.3 release on July 9th? I believe this is not a blocker but just in case.

[pohly]

The klog update went in, the race in the logging path should be gone now. The kubelet shutdown test still has other data races, but they don't seem to occur in the CI.

[Nivedita-coder]

job is passing :)
/close

[k8s-ci-robot]

@Nivedita-coder: Closing this issue.

In response to this:

job is passing :)
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.