absent_over_time should emit all the labels of that metric

[ankurnigam]

Hi Team,

Absent and absent_over_time both are emitting only vector value but we expect absent_over_time should emit all the labels of the metric.

[alburthoffman]

what do u mean by emitting all labels?

[roidelapluie]

Hello, I understand your request, however, given the way Prometheus works, this is not possible. We only have access to instant data.

How would this even behave? would you query the entire history?

you can simulate what you want by using unless and offset: mymetric offset 1h unless mymetric will, during one hour, showing disappearing labels.

[ankurnigam]

correct. I was looking for something like absent by (instance) (my_metric). Also , I don't see present_over_time function , is this introduced recently?

[roidelapluie]

yes, in 2.29

[ankurnigam]

does it retain all labels of the metric?

[ankurnigam]

also do we have something else present as well in 2.29 ?

[roidelapluie]

yes it retains all the labels.

what do you mean with your second question?

[ankurnigam]

Actually our problem is that we have redesigned prometheus to use push mechanism instead of pull, so we are fetching metrics locally on the server and then we have another agent which is pushing metrics to prometheus . This is the standard cloud industry follows.

[RichiH]

Prometheus Remote Write is based on the assumption that the data goes through a Prometheus engine at some point between scraping and being pushed, it's not clearly defined otherwise.

I would suggest Cortex or Thanos as a long-term push target, instead.

[ankurnigam]

Even though we will have thanos but if function will not work then it would be a problem as absent over time doesn't give you labels , it just give you vector value so you cannot compare it dynamically.