Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix flake on TestRecordOperation unit test #105809

Closed
wants to merge 1 commit into from
Closed

fix flake on TestRecordOperation unit test #105809

wants to merge 1 commit into from

Conversation

aojea
Copy link
Member

@aojea aojea commented Oct 21, 2021 •
edited

/kind flake

go test -timeout 100s -run ^TestRecordOperation$ k8s.io/kubernetes/pkg/kubelet/kuberuntime -v -count 10
=== RUN   TestRecordOperation
--- PASS: TestRecordOperation (0.00s)
=== RUN   TestRecordOperation
--- PASS: TestRecordOperation (0.00s)
=== RUN   TestRecordOperation
--- PASS: TestRecordOperation (0.00s)
=== RUN   TestRecordOperation
--- PASS: TestRecordOperation (0.00s)
=== RUN   TestRecordOperation
--- PASS: TestRecordOperation (0.00s)
=== RUN   TestRecordOperation
--- PASS: TestRecordOperation (0.00s)
=== RUN   TestRecordOperation
--- PASS: TestRecordOperation (0.00s)
=== RUN   TestRecordOperation
--- PASS: TestRecordOperation (0.00s)
=== RUN   TestRecordOperation
--- PASS: TestRecordOperation (0.00s)
=== RUN   TestRecordOperation
--- PASS: TestRecordOperation (0.00s)
PASS
ok      k8s.io/kubernetes/pkg/kubelet/kuberuntime       0.038s

NONE

Fixes #104940

@k8s-ci-robot k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. kind/flake Categorizes issue or PR as related to a flaky test. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. area/kubelet sig/node Categorizes an issue or PR as relevant to SIG Node. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Oct 21, 2021
@aojea aojea mentioned this pull request Oct 21, 2021
Closed
@aojea
Copy link
Member Author

aojea commented Oct 21, 2021

/assign @pacoxu @logicalhan

@CatherineF-dev CatherineF-dev mentioned this pull request Oct 21, 2021
Closed
@CatherineF-dev CatherineF-dev mentioned this pull request Oct 21, 2021
Closed
@ehashman ehashman added this to PRs - Needs Reviewer in SIG Node CI/Test Board Oct 28, 2021
ehashman
ehashman reviewed Oct 28, 2021
View reviewed changes
Copy link
Member

@ehashman ehashman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/priority important-longterm
/triage accepted
/lgtm

@k8s-ci-robot k8s-ci-robot added priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-priority Indicates a PR lacks a `priority/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Oct 28, 2021
@ehashman ehashman moved this from PRs - Needs Reviewer to PRs - Needs Approver in SIG Node CI/Test Board Oct 28, 2021
@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 28, 2021
pacoxu
pacoxu approved these changes Oct 29, 2021
View reviewed changes
Copy link
Member

@pacoxu pacoxu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm
/assign @mrunalp

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: aojea, pacoxu
To complete the pull request process, please assign yujuhong after the PR has been reviewed.
You can assign the PR to them by writing /assign @yujuhong in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

MikeSpreitzer
MikeSpreitzer reviewed Oct 29, 2021
View reviewed changes
pkg/kubelet/kuberuntime/instrumented_services_test.go
legacyregistry.MustRegister(metrics.RuntimeOperationsErrors)
},
)
defer legacyregistry.Reset()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks problematic to me, it assumes that there are no concurrent tests using the legacyregistry.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how can we deal with it then?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/cc @logicalhan

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's first be explicit about whether there is a problem.
Are there test scenarios that have concurrent test func invocations in one process?

If so then the implication is that any given test func X can not assume that a registered metric is being affected only by the current invocation of X.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These tests are executed one by one. They don't overlap with each other.

I printed begin time and end time for each test in
https://gist.github.com/CatherineF-dev/7557f0abdabe5e13ba1b37c7fa15523c

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The sample_and_watermark_test.go updated in #105886 could make and use a private registry in each test func invocation.

Same for the metrics_test.go in that same PR.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a bit busy these, anyone feels that can take over this PR?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can take it. Since #105886 is a similar issue.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

During testing, I found one advantage for using the global registry (legacyregistry). It could help detecting duplicated metrics name.

pkg1: metrics_a
pkg2: metrics_a
Both pkg1 and pkg2 emit metrics_a. It will be detected by the global registry, which is pretty useful.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Duplicate metric names from registrations from non-test code will be detected by other unit, integration, and end-to-end tests.

@SergeyKanzhelev SergeyKanzhelev moved this from PRs - Needs Approver to PRs Waiting on Author in SIG Node CI/Test Board Oct 30, 2021
@pacoxu pacoxu mentioned this pull request Nov 1, 2021
Closed
@MikeSpreitzer
Copy link
Member

@MikeSpreitzer

@aojea
Copy link
Member Author

aojea commented Nov 4, 2021

/assign @CatherineF-dev
Thanks for taking over
/hold
please add @MikeSpreitzer and the other people here as reviewers

@k8s-ci-robot k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 4, 2021
@k8s-ci-robot
Copy link
Contributor

@aojea: GitHub didn't allow me to assign the following users: CatherineF-dev.

Note that only kubernetes members, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

/assign @CatherineF-dev
Thanks for taking over
/hold
please add @MikeSpreitzer and the other people here as reviewers

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@CatherineF-dev
Copy link
Contributor

CatherineF-dev commented Nov 9, 2021 •
edited

The below is solved.

Still in trying replacing legacyregistry with local registry for mux.Handle("/metrics", legacyregistry.Handler()).

I tried the following one, but failed

handler := promhttp.InstrumentMetricHandler(prometheus.DefaultRegisterer, promhttp.HandlerFor(localRegistry, promhttp.HandlerOpts{}))
mux.Handle("/metrics", handler)

legacyregistry.Handler() is defined in registry.go

@CatherineF-dev
Copy link
Contributor

Hi @aojea, I uploaded instrumented_services_test.go to https://gist.github.com/CatherineF-dev/37616e11c1d0f331dd1bee40fa56d89a
Could you copy these changes? Or should I push it to aojea:flake_kuberuntime_metrics?

Not sure whether I have the permission to push to aojea:flake_kuberuntime_metrics.

@aojea
Copy link
Member Author

aojea commented Nov 9, 2021

Hi @aojea, I uploaded instrumented_services_test.go to https://gist.github.com/CatherineF-dev/37616e11c1d0f331dd1bee40fa56d89a Could you copy these changes? Or should I push it to aojea:flake_kuberuntime_metrics?

Not sure whether I have the permission to push to aojea:flake_kuberuntime_metrics.

create your own PR , you are the one doing the effort :)

@CatherineF-dev CatherineF-dev mentioned this pull request Nov 10, 2021
Merged
@k8s-ci-robot
Copy link
Contributor

@aojea: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 17, 2021
@aojea
Copy link
Member Author

aojea commented Dec 8, 2021

duplicate

@aojea aojea closed this Dec 8, 2021
SIG Node CI/Test Board automation moved this from PRs Waiting on Author to Done Dec 8, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@endocrimes endocrimes endocrimes left review comments

@pacoxu pacoxu pacoxu approved these changes

@ehashman ehashman ehashman left review comments

@MikeSpreitzer MikeSpreitzer MikeSpreitzer left review comments

@CatherineF-dev CatherineF-dev CatherineF-dev left review comments

@sjenning sjenning Awaiting requested review from sjenning

@logicalhan logicalhan Awaiting requested review from logicalhan

Assignees

@mrunalp mrunalp

@logicalhan logicalhan

@pacoxu pacoxu

@ehashman ehashman

Labels
area/kubelet cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. kind/flake Categorizes issue or PR as related to a flaky test. lgtm "Looks good to me", indicates that a PR is ready to be merged. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. release-note-none Denotes a PR that doesn't merit a release note. sig/node Categorizes an issue or PR as relevant to SIG Node. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
SIG Node CI/Test Board
Done
SIG Node CI/Test Board
Archived in project
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

UT failure: panic: duplicate metrics collector registration attempted
9 participants
@aojea @k8s-ci-robot @MikeSpreitzer @CatherineF-dev @logicalhan @endocrimes @pacoxu @ehashman @mrunalp