Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Race condition with Exemplar in Counter #1145

Closed
fstab opened this issue Oct 7, 2022 · 1 comment
Closed

Race condition with Exemplar in Counter #1145

fstab opened this issue Oct 7, 2022 · 1 comment

Comments

@fstab
Copy link
Member

fstab commented Oct 7, 2022

I found an inconsistent concurrency behavior in client_java, and I think that client_golang has the same issue. Here's the current implementation of the counter's Write() function:

func (c *counter) Write(out *dto.Metric) error {
	val := c.get()

	var exemplar *dto.Exemplar
	if e := c.exemplar.Load(); e != nil {
		exemplar = e.(*dto.Exemplar)
	}

	return populateMetric(CounterValue, val, c.labelPairs, exemplar, out)
}

Counter implements selfCollector. If I understand correctly, this means that the counter continues to be updated while Write() is being called.

So if an update happens after reading val but before reading exemplar, you might see an examplar for an observation that's not represented in val.

I guess to fix this it would be sufficient to read exemplar first and then val, because then the worst case you get an exemplar for a previous observation, but you won't get an exemplar for a not-yet-existing observation.

fstab added a commit that referenced this issue Oct 7, 2022
Potential fix for #1145.

Signed-off-by: Fabian Stäber <fabian@fstab.de>
@fstab
Copy link
Member Author

fstab commented Oct 7, 2022

PR for the trivial fix. I hope I'm not missing anything #1146.

fstab added a commit that referenced this issue Oct 17, 2022
* Fix race condition with Exemplar in Counter

Potential fix for #1145.

Signed-off-by: Fabian Stäber <fabian@fstab.de>

* Fix race condition with Exemplar in Counter

Signed-off-by: Fabian Stäber <fabian@fstab.de>

Signed-off-by: Fabian Stäber <fabian@fstab.de>
@fstab fstab closed this as completed Oct 17, 2022
bwplotka pushed a commit that referenced this issue Nov 1, 2022
* Fix race condition with Exemplar in Counter

Potential fix for #1145.

Signed-off-by: Fabian Stäber <fabian@fstab.de>

* Fix race condition with Exemplar in Counter

Signed-off-by: Fabian Stäber <fabian@fstab.de>

Signed-off-by: Fabian Stäber <fabian@fstab.de>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant