Unit test not considering the entire time series

[martinohansen]

What did you do?

Created the following test case:

rule_files:
    - alerts.yml

evaluation_interval: 1m

tests:
    - name: uricheck-exporter
      interval: 1m
      input_series:
        - series: 'http_status{target="flap-max"}'
          values: '503x2 200x2 503x2 200x2 503x2 200x2'
        - series: 'http_status{target="flap-min"}'
          values: '200x2 -2x2 200x2 -2x2 200x2 -2x2'

      promql_expr_test:
        - expr: max by(target)(http_status) >= 500 or min by (target)(http_status) <= 0
          eval_time: 10m
          exp_samples:
            - labels: '{target="flap-max"}'
              value: 503
            - labels: '{target="flap-min"}'
              value: -2

What did you expect to see?

I would expect this test case to pass: 1) the max over the entire series at 10m is indeed 503 and 2) the min over the entire series at 10m is indeed -2.

What did you see instead? Under which circumstances?

It fails. It looks like the test case is not considering the entire series but just the series at the "eval_time" mark.

$ promtool test rules tests/bug-report.yml 
Unit Testing:  tests/bug-report.yml
  FAILED:
    expr: "max by(target)(http_status) >= 500 or min by (target)(http_status) <= 0", time: 10m,
        exp:"{target=\"flap-max\"} 5.03E+02, {target=\"flap-min\"} -2E+00"
        got:"{target=\"flap-min\"} -2E+00"

[roidelapluie]

You are looking for:
max by(target)(max_over_time(http_status[10m]))

Same for min.

Without the over time, indeed only the instant you query is used.

[martinohansen]

Perfect, thanks a lot!