Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ruler: Implement flag for max-source-resolution in the rule query #7077

Open
KingRebo38 opened this issue Jan 19, 2024 · 7 comments · May be fixed by #7080
Open

Ruler: Implement flag for max-source-resolution in the rule query #7077

KingRebo38 opened this issue Jan 19, 2024 · 7 comments · May be fixed by #7080

Comments

@KingRebo38
Copy link

Is your proposal related to a problem?

My feature request is related to the following bug ticket: #7070

After further investigation I came to the conclusion that the setting max_source_resolution in the http request is the solution.
The ruler, as it stands now, only calculates data using auto downsampling. This is an issue, as the querier then only uses raw data for calculating rules. As I want to calculate metrics aggregated across a long time period this becomes a problem, since my raw data only exists for 7 days.
For example with the following query: increase(kong_http_requests_total[14d])
The ruler will only be able to calculate the last 7 days instead.

Describe the solution you'd like

The solution I found was setting max_source_resolution=5m. This will force the querier to use the downsampled data which allows the ruler to get all data it needs for the aggregation.

I tested this by setting MaxSourceResolution manually in the code of the ruler and creating my own image:

	v, warns, err := promClient.PromqlQueryInstant(ctx, endpoints[i], qs, t, promclient.QueryOptions{
		Deduplicate:             true,
		PartialResponseStrategy: partialResponseStrategy,
		Method:                  httpMethod,
		DoNotAddThanosParams:    doNotAddThanosParams,
		MaxSourceResolution:     "5m", // this would need to be retrieved by a flag

The flag could be something like: --query.max-source-resolution

After this is implemented, I am expecting to see my requests calculated like this:
image

Ignore the gaps please. As you can see, the metric is calculated correctly again, before the switch we made.

For more context, please read the bug ticket.

@kartikaysaxena
Copy link
Contributor

Working on this, will try to come with a PR.

@yeya24
Copy link
Contributor

yeya24 commented Jan 19, 2024

@KingRebo38 What's the query are you running in Ruler?
We are thinking about detecting the resolution rather than setting up the flag manually.
A ruler instance level flag might not work well if you have other rules which don't need 5m resolution

@KingRebo38
Copy link
Author

@yeya24 The Queries are something like this:

rules:
  - expr: increase(kong_http_requests_total[14d])
    record: kong_http_requests_total:rate:14d
  - expr: increase(kong_http_requests_total[30d])
    record: kong_http_requests_total:rate:30d
    ...
    and so on

I also think the instance level flag might cause some issues, but for now I would solve it by sharding the ruler instances.

@douglascamata
Copy link
Contributor

douglascamata commented Jan 23, 2024

@KingRebo38 you can optimize these rules by reusing data from others.

rules:
  - expr: increase(kong_http_requests_total[1d])
    record: kong_http_requests_total:rate:1d
  - expr: sum_over_time(kong_http_requests_total:rate:1d[14d]) # sum the 1d-rate over 14d
    record: kong_http_requests_total:rate:14d
  - expr: sum_over_time(kong_http_requests_total:rate:1d[28d]) # sum the 1d-rate over 28d
    record: kong_http_requests_total:rate:28d

With this you basically have your own downsampling. As long as they are part of the same rule group, evaluation order is respected.

Edit: Ah, if your raw resolution retention is 7 days it won't work :(

@douglascamata
Copy link
Contributor

@KingRebo38 does it work if you set max resource resolution to auto?

@KingRebo38
Copy link
Author

@douglascamata If I set it to auto, it defaults to the raw data

@douglascamata
Copy link
Contributor

You can try then setting --query-range.request-downsampled on your Query Frontend.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants