New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Ruler: Implement flag for max-source-resolution in the rule query #7077
Comments
Working on this, will try to come with a PR. |
@KingRebo38 What's the query are you running in Ruler? |
@yeya24 The Queries are something like this: rules:
- expr: increase(kong_http_requests_total[14d])
record: kong_http_requests_total:rate:14d
- expr: increase(kong_http_requests_total[30d])
record: kong_http_requests_total:rate:30d
...
and so on I also think the instance level flag might cause some issues, but for now I would solve it by sharding the ruler instances. |
@KingRebo38 you can optimize these rules by reusing data from others. rules:
- expr: increase(kong_http_requests_total[1d])
record: kong_http_requests_total:rate:1d
- expr: sum_over_time(kong_http_requests_total:rate:1d[14d]) # sum the 1d-rate over 14d
record: kong_http_requests_total:rate:14d
- expr: sum_over_time(kong_http_requests_total:rate:1d[28d]) # sum the 1d-rate over 28d
record: kong_http_requests_total:rate:28d With this you basically have your own downsampling. As long as they are part of the same rule group, evaluation order is respected. Edit: Ah, if your raw resolution retention is 7 days it won't work :( |
@KingRebo38 does it work if you set max resource resolution to |
@douglascamata If I set it to auto, it defaults to the raw data |
You can try then setting |
Is your proposal related to a problem?
My feature request is related to the following bug ticket: #7070
After further investigation I came to the conclusion that the setting max_source_resolution in the http request is the solution.
The ruler, as it stands now, only calculates data using auto downsampling. This is an issue, as the querier then only uses raw data for calculating rules. As I want to calculate metrics aggregated across a long time period this becomes a problem, since my raw data only exists for 7 days.
For example with the following query: increase(kong_http_requests_total[14d])
The ruler will only be able to calculate the last 7 days instead.
Describe the solution you'd like
The solution I found was setting max_source_resolution=5m. This will force the querier to use the downsampled data which allows the ruler to get all data it needs for the aggregation.
I tested this by setting MaxSourceResolution manually in the code of the ruler and creating my own image:
The flag could be something like: --query.max-source-resolution
After this is implemented, I am expecting to see my requests calculated like this:
Ignore the gaps please. As you can see, the metric is calculated correctly again, before the switch we made.
For more context, please read the bug ticket.
The text was updated successfully, but these errors were encountered: