Deploying OPA in Kubernetes with a PersistentVolume and multiple Pods (or even a HPA)

[HenriBlacksmith]

Hello,

I am looking for information about deploying OPA in Kubernetes with a failure. (and update)-tolerant deployment.

For now we are using OPA in a deployment which looks like the official one: https://www.openpolicyagent.org/docs/latest/deployments/#kicking-the-tires

The issue is that we write data to OPA and evaluate policies which means that our OPA instance has dynamic data which must be shared between all OPA instances.

For now we use only one replica as documented but this is not failure-tolerant and adding more replicas cannot work as OPA does not seem to support a shared storage (with transactions or locks to allow a consistant update of the storage by multiple instances).

Is there a way to have a failure-tolerant deployment of OPA in Kubernetes (i.e. a shared storage and more than one Pod)?

[charlieegan3]

Generally we see users distributing data to OPA in an eventually consistent manner using the bundle service API. This is generally the most simple and robust option.

Is there a reason that eventually consistent updates for data won't work for you?

[HenriBlacksmith]

Hmm ok, I see, I will explore that option with my team

[HenriBlacksmith]

I mean, our goal is to rely on OPA only and not to create a dependency on another service to read data and we also need to be able to update data (so static data cannot be used either)

[anderseknert]

Can only +1 the recommendation of using a bundle server for this purpose. Running OPA in production and at scale almost always entails running one distributed component (OPA) and a centralized one (a bundle server or control plane) for management.

Things like policy and data updates are all accounted for in the bundle service API, and additionally you'll most likely want to use a centralized service for decision logging as well.

[ritazh]

I’m curious if you have tried Gatekeeper (multiple pod deployment of opa in k8s) and the external data feature for your data? https://open-policy-agent.github.io/gatekeeper/website/docs/externaldata

[HenriBlacksmith]

Is'nt Gatekeeper only to apply policies to Kubernetes resources?

My use-case is to use OPA for application permissions

[HenriBlacksmith]

Sorry for replying late, i seems that my team used a Bundle API in the past and we switched to a push mode (as opposed to a mode where OPA pulls the bundle) for performance/speed issues as there was a significant delay (a few seconds) between the bundle publication and its pull by OPA.

Another option I imagine is to push data to multiple instances synchronously but it does not look like the official way to go (that's what we do today but on a single instance/Pod)

[ashutosh-narkar]

Sorry for replying late, i seems that my team used a Bundle API in the past and we switched to a push mode (as opposed to a mode where OPA pulls the bundle) for performance/speed issues as there was a significant delay (a few seconds) between the bundle publication and its pull by OPA.

Specifically on this, have you tried a combination of Delta bundles with HTTP long polling? It may help with faster data propagation.

[Duanxinxin]

I also encountered the same problem, may I ask if there is any good solution in the end