OPA force cache not working with system.authz policy

[HarshPathakhp]

Short Description -
We have a http.send() call which is making a GET request to a remote server, and then force cache its response for 60 seconds. This call is being made in system.authz policy as part of the allow rule. We noticed that OPA is not really caching the response in system.authz, but it is able to cache from a non-system regular policy.

Specifications -

1. OPA version: 0.41.0
2. Able to reproduce issue locally on MacOS

How To Reproduce -
I have attached a toy example, containing 3 bundles, with each bundle being considered as an independent example.
opa-locust-test.zip

To reproduce this issue, following the following steps -

1. For each bundle, do the following -
   - Run opa run -s --authentication=token --authorization=basic -b <add bundle directory name>
   - Once OPA has started running (listening on port 8181), run locust at the root of the directory which contains the locustfile. (You may need to install locust using pip install locust
   - Locust will open a UI at port 8089 locally, go to the UI and enter the number of users as 100, and spawn rate as 10. Locust will start making requests, we can stop the testing from the UI itself after say 3000 requests have been made.
   - The metrics for the load test can be seen by going to Download data and then Download Report. This report will show the request latency distribution and various percentile values at P50, P95, etc.

Actual Behavior -
For the bundles authz_no_cache and authz_force_cache, the request times are going above 5 seconds majorly. The bundle demo_force_cache is showing clear signs of caching, with responses returned in in double digit milliseconds.

Expected Behavior -
I expected the bundle authz_force_cache to show clear signs of caching take place.

[srenatus]

✔️ Reproduced, thanks for the report.

Simpler example:

package system.authz

default allow = false

allow = true {
    response = http.send({
        "url": "https://enbwru0mbg6ca.x.pipedream.net/",
        "method": "GET",
        "force_cache": true,
        "force_cache_duration_seconds": 60
    })
    print(response)
    response.status_code == 200
}

using opa run -s --log-level=debug authz.rego --authorization=basic and three curl "http://127.0.0.1:8181/v1/policies/foo" calls in quick succession, I find three requests sent.

🔍 I'll have a closer look.

[srenatus]

#4838 should fix this.