High memory consumption for Operator when there are lots of pods

[ItalyPaleAle]

In what area(s)?

/area operator

What version of Dapr?

1.9.x
master

Actual Behavior

We are receiving user reports that the Dapr Operator service uses significant amounts of memory (700-800MB) when there are lots of pods in the K8s cluster. This happens even when there are no Dapr-enabled pods running.

Upon initial investigation, this seems to be due to the fact that the Dapr Operator implements a K8s reconciler that watches for all Deployment and StatefulSet objects, including those that don't use Dapr.

Per conversation with @yaron2, the reconciler is necessary so that Dapr can automatically provision K8s Service objects to allow app-to-app communication.

One possibility to reduce memory usage could be to make the reconciler only listen for objects with the Dapr annotation (or, if we cannot filter for annotations, we will need to add a label that is injected by the Injector). This will help reducing memory usage at least in the case when there are lots of pods without Dapr.

Release Note

RELEASE NOTE: FIX Dapr Operator uses lots of memory when there are lots of pods that aren't Dapr-enabled

[akhilac1]

kubernetes-sigs/controller-runtime#244
external-secrets/external-secrets#721

@ItalyPaleAle - seems like filtering by annotations is not supported and we need to look at labels

[ItalyPaleAle]

@akhilac1 are you suggesting disabling the cache? It may very well work, but I'm concerned with the impact on the K8s API server in terms of load.

@yaron2 wdyt?

[akhilac1]

@akhilac1 are you suggesting disabling the cache? It may very well work, but I'm concerned with the impact on the K8s API server in terms of load.

@yaron2 wdyt?

Oh no.. I was only suggesting that we focus on label based filtering. I missed submitting the updated comment. did so now

[ItalyPaleAle]

I see now.

If we can't filter by annotations, we can use the Dapr injector to add labels too, perhaps.

[ItalyPaleAle]

Also worth pointing out that if the Watchdog is enabled, the Operator needs to request the full list of all Pods too. That may add to memory usage.

[snoopy-dooog]

Hey, this is my first time trying to contribute to this repo, and am really not familiar with the codebase at all. However, you project sounds really cool and I'd like to start helping out. I have a WIP, which I'm not exactly sure how to test for and if it is correct, but happy to get some help from someone as I get more familiar with it :)

I was also trying to go over how to run my own dapr in a Kube cluster to test this out better but couldn't find much guidelines on it. Looking forward to get some help on this as well. I usually develop locally so would appreciate to be able to run it locally, but if I have to go for the codespaces, will do that as well.

Thanks in advance!

PR in progress: #5889

[yaron2]

No problem @snoopy-dooog we are glad you've decided to contribute to Dapr and appreciate any code contributions. To test Dapr locally, here's a cheat sheet:

1. Build the binaries with make build-linux
2. Copy docker/Dockerfile to dist/linux_amd64/release
3. Go to the release dir and build the Dapr control plane: docker build -t <your-repo>
4. Uninstall Dapr from your cluster
5. Go to charts/dapr/values.yaml and change global.registry and global.tag accordingly
6. Go to charts/dapr and run helm install dapr . --namespace dapr-system --create-namespace

There are also some make commands that automate this but I find them slower to get going with.

[yaron2]

#5889 remains to close this.

[filintod]

/assign

[filintod]

This was tackled in #5944