9.1.6 warn - rule declares one or many reserved labels. Those rules labels will be ignored

[mhollstein]

What happened:
After update from Grafana 9.0.5 to 9.1.6 there are many entries in the grafana.log file like this:
logger=ngalert t=2022-09-23T07:21:00.353596477Z level=warn msg="rule declares one or many reserved labels. Those rules labels will be ignored" labels="alertname=test ssl"

What you expected to happen:
No warnings like in Grafana 9.0.5

How to reproduce it (as minimally and precisely as possible):
use the new alerting: unified_alerting enabled
update from Grafana 9.0.5 to 9.1.6

Anything else we need to know?:
Grafana itself added these two labels to all alarms during migration from alerting to unified alerting:
-alertname
-rule_uid

Now it seems that Grafana since version > 9.0.5 will add an entry in the logfile for the alertname label.
logger=ngalert t=2022-09-23T07:21:00.353596477Z level=warn msg="rule declares one or many reserved labels. Those rules labels will be ignored" labels="alertname=test ssl"

Environment:

• Grafana version: 9.1.6

• Data source type & version:

• OS Grafana is installed on: as Docker image: grafana/grafana:9.1.6

• User OS & Browser: Windows 10, Browser: Chrome

• Grafana plugins:
  grafana-clock-panel/
  grafana-image-renderer/
  grafana-piechart-panel/
  grafana-polystat-panel/
  grafana-simple-json-datasource/
  grafana-singlestat-panel/
  jdbranham-diagram-panel/
  monitoringartist-monitoringart-datasource/
  natel-plotly-panel/
  novalabs-annotations-panel/
  radensolutions-netxms-datasource/
  ryantxu-ajax-panel/
  savantly-heatmap-panel/
  vonage-status-panel/

• Others:

In the documentation, the only reserverd label for Grafana at this time 23.09.2022 is this label:
grafana_folder but not "alertname".
https://grafana.com/docs/grafana/latest/alerting/fundamentals/annotation-label/how-to-use-labels/

The (1) ist the label alertname that Grafana added during migration to unified alerting
The (2) seems to be another, identical internal Grafana label alertname.

By the way... what does it meen excactly: Those rules labels will be ignored?
Is the custom label "alertname" ignored while the same internal label alertname exists?
Is the result of the rule alertname="SSL Cert Expiry" in my example? (internal wins)

What is with other additional custom labels like rule_uid? Are they evaluated and an alarm is send even if a log warning is given about alertname?

[WojciechKuk]

I have the same issue

[ShawnPollock]

Same here

[ik1s]

+1
9.04 -> 9.1.4

[CSimpiFoN]

Same here on 9.1.6.

[btorsaldivar]

Same here, getting tons of logs on v9.1.6. Also new dashboard is not getting pick up, using json with the dashboards provisioning provider....

logger=ngalert t=2022-09-30T09:18:47.329258692Z level=warn msg="rule declares one or many reserved labels. Those rules labels will be ignored" labels="alertname=XXX  (DB): Failed Connections alert"

[hansimueller66]

The same here - 9.1.7 didn't changed anything - several times a second logs the error appear

[marbuilder]

Same here - 9.1.7

[yuri-tceretian]

I think this happens when during migration from legacy alerting we add a label
https://github.com/grafana/grafana/blob/9f45e2e7060e3878706e70c1465f1403b6e96082/pkg/services/sqlstore/migrations/ualert/alert_rule.go#L103-L102

which is treated by the state manager code as user input.

I think we can do the following:

1. Ignore if the value of the user-defined label is the same as the override.
2. Do not log a warning but log debug instead.

[yuri-tceretian]

I think that warning still makes sense in this case. It will make it visible that the labels\annotations in the notifications are not the same as the user configured.
For example, there can be a label specified in the rule that is also provided by the evaluation result. My opinion is that a warning message is a pretty obvious indicator of a problem, and can save time while troubleshooting.

What do you think about a migration that would automatically remove the label alertname from the rule specification?
That label is ignored and gets overwritten with the rule's title anyway.
Please, either 👍🏻 or leave a comment if you think this is an ok solution. Alternative approaches are welcome too

[mhollstein]

Do not log a warning but log debug instead is a good idea. Remove alertname on migration is a good idea too if it goes to the title. Please have a look at invalid chars in old alertname and convert them too. I‘ve found out that / and \ are not allowed. But the migration had these chars put in. As a result you have to remove these chars before you can save a modified alert rule.

[grafanabot]

This issue was removed from the 9.1.8 milestone because 9.1.8 is currently being released.

[fcorneli]

I'm also seeing this on 9.2.2.

[gurumee92]

I'm also seeing this on 9.2.2. too.

However, there are places where the alarm is triggered and others where it is not

sum by(role) (count_over_time({k8s_pod_name=~".*-grafana-.*"} |= `rule declares one or many reserved labels. Those rules labels will be ignored` [1m]))

last query result: (roleB not triggered alarm but roleA, roleC triggered althhogh they almost same configuration)

• roleA - 275
• roleB - 150 (alarm not triggered)
• roleC - 130

[jeroenhabets]

Is anything stopping you from adding a migration like this for SQLite3? (For the users like us who have already migrated so fix for #56509 comes too late)

UPDATE alert_rule SET labels=json_remove(labels, '$.alertname') 
    WHERE labels IS NOT NULL AND json_extract(labels, '$.alertname') IS NOT NULL;

[d-shi]

Let me know if I should make a new issue for this. I am getting this warning log, but for a different label than alertname so is probably unrelated to migration. I'm wondering if there is a way to tell which alert rule this message is for. We have hundreds of alerts defined all with this same label that is in the warning, so I'm not sure where to narrow down the problem to. Thanks!

[yuri-tceretian]

@d-shi, good catch! that's definitely a bug. The log context should contain rule UID. I will fix it.

[yuri-tceretian]

as for the automatic removal of the label alertname, we cannot reach a consensus about the migration and still discuss the alternatives.
For now alternatives are:

1. remove the label on save.
2. reject the request if the rule contains a reserved label (we have a few besides alertname).

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last year. It will be closed in 30 days if no further activity occurs. Please feel free to leave a comment if you believe the issue is still relevant. Thank you for your contributions!

[github-actions]

This issue has been automatically closed because it has not had any further activity in the last 30 days. Thank you for your contributions!

[shlomitubul]

this issue is still relevant