Unary negation of range vectors (rate and resets of decreasing metrics)

[ruuda]

Proposal

Allow computing rate(), resets(), and increase() for metrics that count down rather than up.

One way to achieve this, would be to allow unary negation on range vectors. Currently rate(-decreasing_metric[5m]) and rate(0 - decreasing_metric[5m]) cause a parse error, but if this were allowed, it would enable computing the rates of decreasing metrics. Note, -rate(decreasing_metric[5m]) is not at all the same thing!

Negating a range vector would negate all of its values.

Use case. Why is this important?

I have an exporter that monitors the balance of an account. The balance is decreasing continuously, as a system is spending from the account. Periodically, the account gets topped up. I want to compute its spending rate, and only spending, regardless of when the account gets topped up. If I compute this using delta or deriv, then if the account was topped up inside the time window selected for the range vector, then the spending rate is no longer accurate, and may even go negative.

[roidelapluie]

Unfortunately, a bank account is fundamentally a gauge, which is incompatible with rate(). I have read your message and you are aware of that, but I fundamentally think that even if we would allow unary negation of vectors, you would stumble accross other issues.

As a workaround, you could try:

clamp_min(decreasing_metric - decreasing_metric offset 5m, 0) but it would not take into account "counter resets".
Another possibility would be to use subqueries:
rate((-decreasing_metric)[5m:]), with other tradeoffs.

[ruuda]

Aah, a subquery is a nice workaround, thanks!

[beorn7]

Note, however, that the counter reset semantics is probably different than what you expect. If a counter reset is detected, the PromQL engine assumes that the counter has started from zero again, as can be seen nicely in this test:

prometheus/promql/testdata/functions.test

Lines 86 to 94 in 5a4c373

	# Test for increase() with counter reset.
	# When the counter is reset, it always starts at 0.
	# So the sequence 3 2 (decreasing counter = reset) is interpreted the same as 3 0 1 2.
	# Prometheus assumes it missed the intermediate values 0 and 1.
	load 5m
	http_requests{path="/foo"} 0 1 2 3 2 3 4
	eval instant at 30m increase(http_requests[30m])
	{path="/foo"} 7

This doesn't match the semantics of "top-up" behavior, even if the proposed increase((-accountbalance)[5m]) was possible. In fact, I'm not even sure what the counter reset detection does if it sees negative values of a counter.

Independent of the use case, I have been thinking lately about changing the semantics of PromQL (in a future release) so that operations that are only acting on single samples (unary negation, arithmetic with a scalar, comparison with a scalar, …) would return the timestamp unchanged rather than the timestamp of the evaluation (which is partially inspired by #8966 (comment), but also other thoughts about the fundamental structure of Prometheus sample values). Those "single sample" operations wouldn't need subqueries. rate((-decreasing_metric)[5m]) would just work (although, as said, not in the intended way for an account balance), but also something like count_over_time((some_metric > 42)[5m]) to count the samples in some_metric exceeding 42.

(This would still not solve the original problem of #8966. For that, we needed new functions like select_first(some_metric[5m]) or select_max(some_metric[5m]), which wouldn't be aggregations over time but select a single sample and return it with its timestamp unchanged. And in the latter case, it also needs a definition how to "break ties", i.e. whether it returns the earliest or latest sample if more than one are having the same maximum value.)