Trying to user OPA with Istio in Minikube, rego rules not working like expected

[AnnikaTestumgebung]

Hi,
I am still quite new to the OPA topic, but I have already carried out examples, created rules, etc. Now I'm trying to apply OPA to my test application. I have a Node.js application in the minicube with Istio. Now I want to apply the OPA website example to my application first:
https://www.openpolicyagent.org/docs/latest/envoy-tutorial-istio/

I followed all these steps and it's working. Now I am trying to user OPA for my test application in minikube. So I changed:

• I have my microservices in minikube
• added Istio
• added OPA with quick_start

I would now like to have a rule that only allows access to /photogallery via my local web application: test.com/fogogalerie should be allowed, all other paths should not.

When I add the rule:

apiVersion: v1
kind: ConfigMap
metadata:
  name: opa-policy
data:
  policy.rego: |
    package istio.authz

    import input.attributes.request.http as http_request
    import input.parsed_path

    default allow = false

    allow {
        parsed_path[0] == "health"
        http_request.method == "GET"
    }

    allow {
      "path" == "/fotogalerie" 
    }

I get this in the browser:

upstream connect error or disconnect/reset before headers. reset reason: connection termination

when I got to https://test.com/fotogalerie

So why is it not working?

What I tried so far:

• I changed the service mehs as mentioned
• tried different rules
• default allow = true is working: access is allowed to the whole page
• I have also tried to apply the rule specifically only to the web server pod via Authorization Policy:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: webserver
spec:
  selector:
    matchLabels:
      app: webserver
  action: CUSTOM
  provider: 
    name: opa-ext-authz-grpc
  rules:
  - to:
    - operation:
        notPaths: ["/health"]

Desired result: I would like to perform authorization queries with OPA, e.g. depending on which browser is being used, which role is currently logged on, etc. But first I would be satisfied if access via /photogallery is allowed and all other accesses are not, so that I can see that OPA also works with my setting.

I am thankful for any help.

[anderseknert]

Hi there @AnnikaTestumgebung 👋

Hmm, if setting default allow := true works, I would take that to mean that at least OPA is configured and working correctly. I'd check the logs for the Istio and OPA containers to see what they say.

Not related to your issue, but there's a mistake in the rule here, where you're comparing two strings that will never be equal:

allow {
  "path" == "/fotogalerie" 
}`

I guess you meant to get the path from the input here.

Using Regal for linting is a good way to catch mistakes like that :)

[AnnikaTestumgebung]

OK, thank you very much for the tip! And yes, that was a bit of a bad example, I've tested many different rules, but I am not quite sure if the rego really works as it should in my case. With the example above I just wanted to test whether it is possible via OPA that my application is only accessible via the path /fotogalerie and all other paths are not accessible, simply as a test.

I have now tested this example here with Regal and with
https://play.openpolicyagent.org/ (1 file linted. No violations found.)

`apiVersion: v1
kind: ConfigMap
metadata:
name: opa-policy
data:
policy.rego: |
package istio.authz

import rego.v1

default allow := false

is_firefox_user_agent if {
  input.user_agent == "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:124.0) Gecko/20100101 Firefox/124.0"
}

allow if {
  is_firefox_user_agent
}

allow if input.parsed_path[0] == "fotogalerie"

`

I am not getting any error messages from the Istio or OPA containers in my pods. Now I would expect to be able to reach my application with Firefox (I also tried it without the last line of the rule ...), but I always get this error:

upstream connect error or disconnect/reset before headers. reset reason: connection termination

The istio-logs of the pod seems to be good ("GET /health?plugins HTTP/1.1" 200 - via_upstream"...) the access still is denied (from istio logs) UAEX ext_authz_denied ... inbound|8001|| 127.0.0.6:55875 10.244.1.236:8001 .. outbound_.8001_._.webserver-service.default.svc.cluster.local

And the opa logs do not refer to any error ({"client_addr"...":"info","msg":"Received request.","req_id":75,"req_method":"GET","req_path":"/health","time":"2024-04-12T09:43:26Z"}")

Of cause I could again try to change the rego, but that's what happens since I started to get OPA into my project (I started to work more intensively with OPA 2 weeks ago, maybe that's a bit too little time, but I want to use OPA in my thesis and I'm running out of time.)

And by the way: thanks for your detailed videos on OPA :)

[anderseknert]

I'm verry happy to hear that! There's nothing wrong with your policy, so I don't think this is an OPA issue. I'm afraid I don't know enough about Istio to be of much help with that though. Perhaps @ashutosh-narkar or @srenatus could have some ideas.

[anderseknert]

({"client_addr"...":"info","msg":"Received request.","req_id":75,"req_method":"GET","req_path":"/health","time":"2024-04-12T09:43:26Z"}")

Do you only see requests to the /health endpoint in the logs, or others too?

[srenatus]

I've been summoned 👀 Do you know if you can change the arguments that envoy is started with? If so, adding

--component-log-level ext_authz:trace

to its arguments would give you more detailed logs about the communication between envoy and opa-envoy-plugin. This helped me in the past, but I don't know enough Istio to tell you how to wire it up :/

[AnnikaTestumgebung]

Wowh! I didn't expect to get so much support, thank you! I'm glad that I can rule out the rules themselves, that directs the solving to the configuration with Istio, I guess. Although I did the tutorial here and that worked. I only deleted all the deployments of the example used by the tutorial here of the example and added my application to minikube and also my istio gateway ... And I am just trying to use my own rego rules. However, as written, this does not work.

Perhaps also interesting: if I change the rego in the configmap to default allow = true and remove everything else, I have access to the application ...

About the logs:
At first I thought I was only seeing the /health logs as they are sent every few seconds, but in the file are the logs of the OPA-container in my web server pod as well. I assume that the OPA container is processing requests from the web server.

logs-from-opa-istio-in-webserver-55849f9587-9jbnl.log
logs-from-istio-proxy-in-webserver-55849f9587-9jbnl.log

[AnnikaTestumgebung]

I'm going to try it out now, thanks!

[srenatus]

Sorry I just now remembered I hadn't looked at your shared logs last week. Looking at them now, there are evaluated policies, as can be seen from the Decision Logs -- so it's probably not a connection issue between envoy and opa-envoy-plugin:

{"decision_id":"a4bf0b89-6f98-40ad-98fe-b1b873c10926","input":{"attributes":{"destination":{"address":{"socketAddress":{"address":"10.244.2.35","portValue":8001}},"principal":"spiffe://cluster.local/ns/default/sa/webserver","service":"outbound_.8001_._.webserver-service.default.svc.cluster.local"},"source":{"address":{"socketAddress":{"address":"10.244.2.23","portValue":53044}},"principal":"spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account"}},"parsed_body":null,"parsed_path":[""],"parsed_query":{},"truncated_body":false,"version":{"encoding":"protojson","ext_authz":"v3"}},"labels":{"id":"cd33d3ba-d102-42e7-9343-dde8d410f9b5","version":"0.63.0-envoy-3"},"level":"info","metrics":{"timer_rego_query_eval_ns":1313125,"timer_server_handler_ns":4276500},"msg":"Decision Log","path":"istio/authz/allow","result":false,"time":"2024-04-12T19:10:24Z","timestamp":"2024-04-12T19:10:24.187534545Z","type":"openpolicyagent.org/decision_logs"}
{"decision_id":"85d19cc4-6d98-4c6c-86cf-83399f4ed4c2","input":{"attributes":{"destination":{"address":{"socketAddress":{"address":"10.244.2.35","portValue":8001}},"principal":"spiffe://cluster.local/ns/default/sa/webserver","service":"outbound_.8001_._.webserver-service.default.svc.cluster.local"},"source":{"address":{"socketAddress":{"address":"10.244.2.23","portValue":53048}},"principal":"spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account"}},"parsed_body":null,"parsed_path":[""],"parsed_query":{},"truncated_body":false,"version":{"encoding":"protojson","ext_authz":"v3"}},"labels":{"id":"cd33d3ba-d102-42e7-9343-dde8d410f9b5","version":"0.63.0-envoy-3"},"level":"info","metrics":{"timer_rego_query_eval_ns":914625,"timer_server_handler_ns":3786542},"msg":"Decision Log","path":"istio/authz/allow","result":false,"time":"2024-04-12T19:10:25Z","timestamp":"2024-04-12T19:10:25.295093545Z","type":"openpolicyagent.org/decision_logs"}

But they both contain

"result":false

so if that's not what you wanted, given the input that's also part of the decision log, let's revisit your policy 🔍

[AnnikaTestumgebung]

Unfortunately, I have not found out exactly where to insert these lines (--component-log-level ext_authz:trace). I tried to insert the argument directly into the istio-gateway via Edit in the minikube dahsboard to the istio-ingressgateway, but that doesn't work. How did you insert the argument?

And I would like to know how I can make the rego policy only affect the webserver pod?

I'm only asking because even that hasn't helped me to call the application only via firefox, as I intended as a test for OPA.

With the hope that my questions here will bring me closer to a solution:

If I add the policy mentioned above to allow access only with firefox browser, it happens that the whole application gets the error above: upstream connect error or disconnect/reset before headers. reset reason: connection termination
So I tried to apply an AuthorizationPolicy additional just to the webserver with that:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: webserver
spec:
  selector:
    matchLabels:
      app: webserver
  action: CUSTOM
  provider: 
    name: opa-ext-authz-grpc
  rules:
  - {}

So my questions are:

1. Is it correct to specify the webserver pod via matchLabels in the AuthorizationPolicy?
   --> If I do so, the problem is, that in the webserver pod the kube-probe failes
   These are the logs from the istio-sidecar inside my webserver pod
   {"decision_id":"5ee16484-1d8a-41bd-9df6-79ba4c21b156","input":{"attributes":{"destination":{"address":{"socketAddress":{"address":"10.244.0.33","portValue":8282}}},"metadataContext":{},"request":{"http":{"headers":{":authority":"10.244.0.33:8282",":method":"GET",":path":"/health?plugins",":scheme":"http","accept":"*/*","user-agent":"kube-probe/1.28","x-forwarded-proto":"http","x-request-id":"fc887185-a26b-94fc-83e1-c4222e99c28e"},"host":"10.244.0.33:8282","id":"11541058857952101185","method":"GET","path":"/health?plugins","protocol":"HTTP/1.1","scheme":"http"},"time":"2024-04-16T09:51:06.246514Z"},"routeMetadataContext":{},"source":{"address":{"socketAddress":{"address":"10.244.0.1","portValue":58708}}}},"parsed_body":null,"parsed_path":["health"],"parsed_query":{"plugins":[""]},"truncated_body":false,"version":{"encoding":"protojson","ext_authz":"v3"}},"labels":{"id":"c30da3a7-50e6-413a-b6b4-c9ee6fa221c9","version":"0.63.0-envoy-4"},"level":"info","metrics":{"timer_rego_query_eval_ns":712458,"timer_server_handler_ns":3173250},"msg":"Decision Log","path":"istio/authz/allow","result":false,"time":"2024-04-16T09:51:06Z","timestamp":"2024-04-16T09:51:06.251995803Z","type":"openpolicyagent.org/decision_logs"}
2. When I try to add the kube-probe for the webserver pod back into the AP through these lines, I get an error message from the opa-sidecar.
   I added this (like from the example of quick_start)

    allow {
        parsed_path[0] == "health"
        http_request.method == "GET"
    }

And the logs from the ops-sidecar:

error: initialization error: 2 errors occurred:
/policy/policy.rego:8: rego_unsafe_var_error: var parsed_path is unsafe
/policy/policy.rego:9: rego_unsafe_var_error: var http_request is unsafe

I understand that the last added code seems to be wrong, but it works with the quick_start example ...

[srenatus]

Is it correct to specify the webserver pod via matchLabels in the AuthorizationPolicy?

I'm afraid I have no idea. Perhaps @ashutosh-narkar can have a look when the sun goes up that side of the globe.

When I try to add the kube-probe for the webserver pod back into the AP through these lines, I get an error message from the opa-sidecar.

So in the quickstart examples, we also have imports,

    import input.attributes.request.http as http_request
    import input.parsed_path

these are what makes parsed_path and http_request defined in the snippet you have there.

[ashutosh-narkar]

Unfortunately, I have not found out exactly where to insert these lines (--component-log-level ext_authz:trace).

This would be used if you're using standalone Envoy. Istio may have some other config to expose this.

Is it correct to specify the webserver pod via matchLabels in the AuthorizationPolicy?

So the idea here is that whenever the server get a request, OPA is called? From the decision logs, it does not seem OPA is getting the actual traffic but the health check calls. OPA is running in the mesh along side Istio proxy. So how is the webserver supposed to reach OPA? I'd imagine you'll have to setup some other config to make that happen.

[AnnikaTestumgebung]

Thank you for your answers. I have several pods in the namespace default. All pods in this namespace are added to both istio and opa. This means that the webserver also communicates with the opa pod via the opa sidecar container.

I have now found out that the webserver pod can be partially selected via machtLabel as far as opa is concerned. I have to add this explicitly, for example:

default allow := false

      allow if {
        regex.match(`^/health`, input.attributes.request.http.path)
        input.attributes.request.http.method == "GET"
      }

Otherwise the kube-probe will fail for the webserver, but not for the other pods. All other rules seem to affect all other pods as well. I didn't really understand the behavior, but I assume that it more or less doesn't matter where the OPA pod gets its rules from. In my case from the webserver, but the rules are then queried by all other OPA sidecar containers via the OPA pod as the central location, so they affect all pods in the defined namespace.

At least that's what I assume ... But it does not explain why without the code above the webserver pod fails, but the other pods do not.