Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cluster should always aggregate metrics in a same order #539

Open
matej21 opened this issue Feb 3, 2023 · 0 comments
Open

Cluster should always aggregate metrics in a same order #539

matej21 opened this issue Feb 3, 2023 · 0 comments

Comments

@matej21
Copy link

matej21 commented Feb 3, 2023

The issue with the current metrics aggregation is that it occurs in the order in which individual workers respond to the GET_METRICS_REQ message. This may cause variations in the aggregation results when the metrics contain floats, such as a histogram.

For example, if workers respond to the first request with values 0.5848208, 0.5479198, 0.3437699 (which sum to 1.4765105), and then respond to a second request with the same values but in a different order (0.3437699+0.5848208+0.5479198), the result of the aggregation will be1.4765104999999998 in JS, due to float errors.

This can trigger a "reset" detection in Prometheus and severely impact the accuracy of the graphs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants