Best Practices for changing configuration & labels

[offermannu]

Hi everyone,

The installation of Prometheus in a medium or larger system landscape takes some time. Meanwhile, the configuration will be constantly changing, so the metrics collected during this time will be somewhat incompatible.
I think that this is normal, and the data of the development phase can only be used in a limited way - if at all.
If you look at the time series in - e.g. - Grafana, you get "strange" effects if you extend the time range to include the setup phase.

But how to proceed if you change Prometheus configurations and especially labels later?
In relational databases, schema migration scripts are executed in such cases, which may also change the data.
Is something like this possible or desirable in Prometheus?

I was thinking of adding a version label "ver=XYZ" to all targets, which will be changed if the configuration changes in an incompatible way. In Grafana, one could add a variable "version" to the dashboards, in which the current version is entered. All PROMQL queries would need to be extended to include the expression {ver="$version}.
The effect would be that queries in Grafana would only consider the compatible time series, but the question is whether the effort is worth it.

So my question is: are there best practices for Prometehus to deal with incompatible data or configuration changes?