Fix sample_and_watermark_test.go for bad luck, repeated test #106325

MikeSpreitzer · 2021-11-10T20:36:20Z

What type of PR is this?

/kind bug
/kind failing-test

What this PR does / why we need it:

This PR fixes the unit test for sample-and-watermark histograms in two ways.
First, it makes the test correctly handle the cases where the first step of the fake clock does not cross the sampling threshold.
Second, it make the test work correctly if it is repeated in the same process and even if multiple invocations are run concurrently in the same process.

This is part of the campaign to make unit tests safe for repetition (#104940).

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

The second half of this is a simpler alternative to #105886 .

Does this PR introduce a user-facing change?

NONE

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

MikeSpreitzer · 2021-11-10T20:37:25Z

@kubernetes/sig-api-machinery-bugs
/cc @wojtek-t
/cc @tkashem
@Catherine-F

logicalhan · 2021-11-10T21:03:38Z

staging/src/k8s.io/apiserver/pkg/util/flowcontrol/metrics/sample_and_watermark_test.go

@@ -95,9 +95,9 @@ func TestSampler(t *testing.T) {
 }

 /* getHistogramCount returns the count of the named histogram */
-func getHistogramCount(regs Registerables, metricName string) (int64, error) {
+func getHistogramCount(registry compbasemetrics.KubeRegistry, metricName string, allowNotFound bool) (int64, error) {


I'd actually change this so that allowNotFound is not a parameter on the function and return 0, nil as default fallback behavior. My reasons:

cleaner function signature

this is a private function which seems to be used exactly once in this file

this is quite possibly just actually the correct behavior.

The force-push to 9ec70d16b11 simplified that function without losing that bit of information if the test ever does fail that way.

MikeSpreitzer · 2021-11-11T02:53:32Z

/retest

logicalhan · 2021-11-11T17:50:03Z

staging/src/k8s.io/apiserver/pkg/util/flowcontrol/metrics/sample_and_watermark_test.go

 		}
 		return int64(*hist.SampleCount), nil
 	}
-	return 0, fmt.Errorf("not found, considered=%#+v", considered)
+	return 0, errMetricNotFound


why return an error, if it's not found, then 0 is the correct count.

Because there are two ways to get zero: metric not found, or metric found and contains zero. No need to lose that distinction.

I actually don't think the metric exists until it gets written to...

Yes, that is part of what is expected.
My point here is let's not needlessly discard a bit of information about why the get method returned zero, since the point of a test is to not assume that everything goes as expected. If that get method ever returns a zero when zero is not what's expected, it can be helpful to have a bit of explanatin of why for the zero.

If a metric can't exist until it's written to, then the error condition is actually not here, it exists in like 119. If int64(*hist.SampleCount) is equal to zero, this is a condition we do not expect and that should be an error.

I am having trouble parsing "... is actually not here, it exists ...".

This is a behavioral unit test of the sample-and-watermark histograms including their underlying machinery. While us developers expect that the HistogramVec has no metrics before it is written, the point of a behavioral unit test is to not assume more than is necessary. The current revision of this PR can distinguish between different pathologies that lead to an unexpected zero. That seems better to me than not helping to identify what went wrong, in the case of an unexpected zero.

While us developers expect that the HistogramVec has no metrics before it is written...

This is a reasonable expectation given that this is how underlying Prometheus implementation actually works.

I am saying this is how it should look:

for _, mf := range mfs { thisName := mf.GetName() if thisName != metricName { continue } metric := mf.GetMetric()[0] hist := metric.GetHistogram() if hist == nil { return 0, errors.New("dto.Metric has nil Histogram") } if hist.SampleCount == nil { return 0, errors.New("dto.Histogram has nil SampleCount") } count := int64(*hist.SampleCount) if count == 0 { return 0, errors.New("we should never have a 0 samplecount here") } return count, nil } return 0, nil

I think it is unnecessarily specific for this client of the Prometheus go library to insist that a HistogramVec whose label slice is empty start out in a state where the suggested code in the previous comment executes the return 0, nil statement. Remember that calling NewHistogram produces a Histogram with a sample count of zero. So such a thing is perfectly fine, semantically. A HistogramVec whose label slice is empty can only ever have one Histogram in it. If the HistogramVec implementation were to choose to create the only possible Histogram in this case eagerly, who cares? Maybe somebody with other Prometheus use cases in mind, but I do not think that clients of sample-and-watermark histograms would care.

MikeSpreitzer · 2021-11-11T19:22:13Z

/test pull-kubernetes-kubemark-e2e-gce-scale

fedebongio · 2021-11-11T21:05:50Z

/triage accepted

MikeSpreitzer · 2021-11-13T03:59:17Z

/retest

marseel · 2021-11-15T15:37:08Z

/test pull-kubernetes-kubemark-e2e-gce-scale

logicalhan

/lgtm
/approve

k8s-ci-robot · 2021-11-15T20:36:26Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: logicalhan, MikeSpreitzer

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~staging/src/k8s.io/apiserver/pkg/util/flowcontrol/OWNERS~~ [MikeSpreitzer]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2021-11-16T11:38:46Z

@MikeSpreitzer: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-kubernetes-kubemark-e2e-gce-scale	9ec70d16b115e1cf1cb84dbd57619ac1a9303533	link	true	`/test pull-kubernetes-kubemark-e2e-gce-scale`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

wojtek-t · 2021-11-16T13:04:59Z

Heh... - the test was triggerred exactly when kubemark got broken by CSI migration for ~2h yesterday.

/test pull-kubernetes-kubemark-e2e-gce-scale

MikeSpreitzer · 2021-11-16T14:02:10Z

Maybe this is not the best vehicle for testing CI; I made #106413 for that purpose.
Is there anyway to abandon the pull-kubernetes-kubemark-e2e-gce-scale job for this PR?

wojtek-t · 2021-11-16T14:26:30Z

Is there anyway to abandon the pull-kubernetes-kubemark-e2e-gce-scale job for this PR?

I think that force-pushing the commit (e.g. rebasing on master) should do the job.

MikeSpreitzer · 2021-11-16T18:34:07Z

The force-push to 06e1716 is a rebase onto master, for the purpose of canceling the request for the pull-kubernetes-kubemark-e2e-gce-scale job.

logicalhan

/lgtm

k8s-ci-robot requested review from lavalamp and yue9944882 November 10, 2021 20:37

k8s-ci-robot added area/apiserver sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Nov 10, 2021

k8s-ci-robot requested review from tkashem and wojtek-t November 10, 2021 20:37

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 10, 2021

logicalhan reviewed Nov 10, 2021

View reviewed changes

MikeSpreitzer force-pushed the iterable-sawh-test branch from 3b2cca3 to 9ec70d1 Compare November 10, 2021 22:58

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Nov 10, 2021

logicalhan reviewed Nov 11, 2021

View reviewed changes

k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Nov 11, 2021

This was referenced Nov 14, 2021

apf: kubemark-500 scale test fails with mutating request estimator enabled #105804

Closed

pull-kubernetes-kubemark-e2e-gce-scale fails when it should not and gathers too little info kubernetes/test-infra#24349

Closed

logicalhan approved these changes Nov 15, 2021

View reviewed changes

k8s-ci-robot assigned logicalhan Nov 15, 2021

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 15, 2021

Fix sample_and_watermark_test.go for bad luck, repeated test

06e1716

MikeSpreitzer force-pushed the iterable-sawh-test branch from 9ec70d1 to 06e1716 Compare November 16, 2021 18:33

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 16, 2021

logicalhan reviewed Nov 16, 2021

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 16, 2021

k8s-ci-robot merged commit d32be90 into kubernetes:master Nov 16, 2021

k8s-ci-robot added this to the v1.23 milestone Nov 16, 2021

MikeSpreitzer deleted the iterable-sawh-test branch November 16, 2021 22:40

mmiranda96 mentioned this pull request Nov 17, 2021

[Flaky test] [sig-node] containerd-NodeConformance #106469

Closed

chymy mentioned this pull request Aug 10, 2023

Using Golang 1.20 to run test cases pkg/util/flowcontrol/metrics/sample_and_watermark_test.go occasionally fails #119744

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix sample_and_watermark_test.go for bad luck, repeated test #106325

Fix sample_and_watermark_test.go for bad luck, repeated test #106325

MikeSpreitzer commented Nov 10, 2021

MikeSpreitzer commented Nov 10, 2021

logicalhan Nov 10, 2021

MikeSpreitzer Nov 10, 2021

MikeSpreitzer commented Nov 11, 2021

logicalhan Nov 11, 2021

MikeSpreitzer Nov 11, 2021

logicalhan Nov 11, 2021

MikeSpreitzer Nov 11, 2021 •

edited

logicalhan Nov 11, 2021

MikeSpreitzer Nov 11, 2021

logicalhan Nov 11, 2021 •

edited

MikeSpreitzer Nov 12, 2021 •

edited

MikeSpreitzer commented Nov 11, 2021

fedebongio commented Nov 11, 2021

MikeSpreitzer commented Nov 13, 2021

marseel commented Nov 15, 2021

logicalhan left a comment

k8s-ci-robot commented Nov 15, 2021

k8s-ci-robot commented Nov 16, 2021

wojtek-t commented Nov 16, 2021

MikeSpreitzer commented Nov 16, 2021

wojtek-t commented Nov 16, 2021

MikeSpreitzer commented Nov 16, 2021

logicalhan left a comment

Fix sample_and_watermark_test.go for bad luck, repeated test #106325

Fix sample_and_watermark_test.go for bad luck, repeated test #106325

Conversation

MikeSpreitzer commented Nov 10, 2021

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

MikeSpreitzer commented Nov 10, 2021

logicalhan Nov 10, 2021

Choose a reason for hiding this comment

MikeSpreitzer Nov 10, 2021

Choose a reason for hiding this comment

MikeSpreitzer commented Nov 11, 2021

logicalhan Nov 11, 2021

Choose a reason for hiding this comment

MikeSpreitzer Nov 11, 2021

Choose a reason for hiding this comment

logicalhan Nov 11, 2021

Choose a reason for hiding this comment

MikeSpreitzer Nov 11, 2021 • edited

Choose a reason for hiding this comment

logicalhan Nov 11, 2021

Choose a reason for hiding this comment

MikeSpreitzer Nov 11, 2021

Choose a reason for hiding this comment

logicalhan Nov 11, 2021 • edited

Choose a reason for hiding this comment

MikeSpreitzer Nov 12, 2021 • edited

Choose a reason for hiding this comment

MikeSpreitzer commented Nov 11, 2021

fedebongio commented Nov 11, 2021

MikeSpreitzer commented Nov 13, 2021

marseel commented Nov 15, 2021

logicalhan left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Nov 15, 2021

k8s-ci-robot commented Nov 16, 2021

wojtek-t commented Nov 16, 2021

MikeSpreitzer commented Nov 16, 2021

wojtek-t commented Nov 16, 2021

MikeSpreitzer commented Nov 16, 2021

logicalhan left a comment

Choose a reason for hiding this comment

MikeSpreitzer Nov 11, 2021 •

edited

logicalhan Nov 11, 2021 •

edited

MikeSpreitzer Nov 12, 2021 •

edited