Incorrect sum aggregations with recording rule

[kul]

What did you do?

Applied following addition rules,

  - name: agg-recording-rules
    groups:
      - name: agg-rules
        rules:
          - record: my_requests_drop_opt0_total
            expr: sum without (colo, instance, job, prometheus, service) (my_requests_drop_total)

What did you expect to see?

I was expecting more or less the same rate and increase calculation over both original and recorded metric.

What did you see instead? Under which circumstances?

With less load (i.e. instances emitting the metrics) , the rate and increase was in line. For example the cardinality of the original metrics (my_requests_drop_total) at a point in time typically were 38k.

However with increased load i.e cardinality of 200k , the aggregation rules gave strange results eg. here is how the rate looked

System information

No response

Prometheus version

kube-prometheus-stack-35.5.78
Prometheus: 2.35.0 

Prometheus configuration file

No response

Alertmanager version

No response

Alertmanager configuration file

No response

Logs

No response

[prymitive]

I was expecting more or less the same rate and increase calculation over both original and recorded metric.

Your assumption is wrong. Summing together counters loses information about counter resets, so it cannot be accurate.

[prymitive]

https://www.robustperception.io/rate-then-sum-never-sum-then-rate/