Prometheus Data Source - Expand Rules: When multiple recording rules match a metric-name the first match isn't always correct

[ellisda]

What happened?

FWIW - I'm writing this up after discussing a customer's experience with @itsmylife

I was looking at an Mimir alert rule definition, and chose to "Explore" the query. The Explore GUI detected that the query was referring to a recorded series and suggested I could "expand rules", which I did. The results weren't what I expected because I have more than one recording rule that writes into the metric name, each made unique by additional prometheus labels on the recorded series.

What did you expect to happen?

I expected the "expand rules" function to find the rule that populated the series I was querying or to give me some kind of warning / error that it found multiple matches and might not work for me query.

I also expected the result of the "expand rules" action to be a promql query that no longer contained the uuid=... label matcher, because that label was added as the result of the recording rule and didn't exist in the raw metrics (i.e. I can't run the expression from the rule AND include the "uuid" label matcher, that uuid label won't exist on the raw metrics, only the recorded series)

Screenshare.-.2024-05-10.11_56_39.AM.mp4

Did this work before?

No, I don't think this worked before. I chose to write it up as a bug rather than a feature request, because I thought it would be easier to understand the issue this way.

How do we reproduce it?

1. Produce multiple recording rules that write to the same metric name (ex: recorded_series) , where each adds labels to the rule object (ex: uuid=<unique value per rule>)
2. Explore a promql query like recorded_series{uuid="foo"} and click "expand rules"
3. Confirm that the expanded query comes from the first recording rule that matches metric name recorded_series rather than from the one that had rule.Labels[uuid] = "foo"

Is the bug inside a dashboard panel?

no. It's in the Explore GUI

Environment (with versions)?

Grafana: 11.1.0
OS: Cloud
Browser: chrome

Grafana platform?

I use Grafana Cloud

Datasource(s)?

prometheus