Metrics relabeling doesn't respect __tmp prefix

[herewasmike]

What did you do?
I'm adding metric relabeling with some condition involved
` metric_relabel_configs:

• source_labels: [pod, id]
  separator: ;
  regex: .+;/kubepods/.+
  target_label: __tmp_is_pod
  replacement: "true"
  action: replace
• source_labels: [__tmp_is_pod, id]
  separator: ;
  regex: true;.+
  target_label: id
  replacement: $1`

What did you expect to see?
Documentation states

If a relabeling step needs to store a label value only temporarily (as the input to a subsequent relabeling step), use the __tmp label name prefix. This prefix is guaranteed to never be used by Prometheus itself.

And for metrics relabeling there's only

It has the same configuration format and actions as target relabeling.

So one can expect that this label won't be ingested into prometheus (please correct me if I misread documentation)

What did you see instead? Under which circumstances?

Environment

• Prometheus version:
  Version | 2.26.0
  Revision | 3cafc58
  `

[roidelapluie]

The documentation here says:

https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config
Labels starting with __ will be removed from the label set after target relabeling is completed.

We could make it clearer that it only affect target relabeling, e.g. relabel_configs.

It does not affect:

• metric_relabel_configs
• write_relabel_configs
• alert_relabel_configs

you would need to remove it with another action

- regex: __tmp_.*
  action: labeldrop

[herewasmike]

Thank you for promt reply.
It appears I did misread documentation, though it would be really nice to have this reworded a bit to avoid any confusions.
I'm aware of labeldrop but thought that it may come handy to have some temporary labels in metric relabelings as well.

[roidelapluie]

The sentence: This prefix is guaranteed to never be used by Prometheus itself.actually means that we will not introduce new "magic" labels that start with __tmp. We sometimes add new labels that start with __, like __scrape_interval__ that was added recently.