HPA and ArgoCD - scaling is disabled since the replica count of the target is zero

[KasparWe]

Hi there,

I've created a Deployment without setting replicas and created a ScaledObject for scaling the deployment based on number of messages in AWS SQS.
The replica count is set to 0 because the application should act event driven. So whenever a new message appears in the queue it will scale up/down.

The Problem
The HPA shows scaling is disabled since the replica count of the target is zero because the default value for replicas in the deployment is 0. => The ScaledObject will be in not active state

When setting replicas in the deployment to eg. to 1 the HPA will work as expected and the ScaledObject will show active state BUT ArgoCD`s AutoSync will fight till infinity against the difference from the deployment spec and terminate/create replicas .

Why the HPA requires the deployment to have a replica count set?

As a workaround I disabled the AutoSync for the Deployments which are scaled by the HPA but ArgoCD will show them as OutOfSync.

[zroubalik]

HPA doesn't properly support scaling to 0, but that's something that KEDA does. So once the Deployment is scaled to 0, KEDA will activate it to 1 replica, you shouldn't do that. So your original ask (scale 0<->N replicas based on AWS SQS) should work properly, without your interaction.

But there should be only one controller that manages the number of replicas in the Deployment, so in this case it is KEDA. If you have additional tool that tries to modify the replica number, it will result in conflicts. So in this case it is probably the ArgoCD AutoSync.

[ishworg]

As a workaround I disabled the AutoSync for the Deployments which are scaled by the HPA but ArgoCD will show them as OutOfSync.

All things considered, fwiw, disabling autosync in Argo is not a recommended approach (although different teams/orgs have differing deployment workflows). A better approach is to update Argo's application (declaratively in the YAML) - documented at https://argo-cd.readthedocs.io/en/stable/user-guide/diffing/#application-level-configuration

[ishworg]

Under https://argo-cd.readthedocs.io/en/stable/user-guide/diffing/#application-level-configuration, apps are deployed using a fixed number of replicas. After that, keda.sh's scaler will take over the replica's scale-in/scale-out activity and argo will completely ignore the JSON pointer that points to the deployment's replica count.

[JorTurFer]

In general, you shouldn't specify the replica count in the workload manifests when you work with autoscaling, because every time when you apply that manifest, the replica count will go to the fixed value in your manifest and then the autoscaling will put the desired replicas, but you could go from X to fixed value and the to X again, impacting in your users if the reduction is big

[ishworg]

@JorTurFer Sure. What you are describing is great but that's the basic minimum and general version of it. Most of the deployments are rollingUpdate and when you specify a fixed y for spec.replicas in your workload (for Deployment for e.g.,), the scheduler will ensure you have y replicas. However, rollingUpdate ensures all the health checks and surges are met before rolling out a new pod. Once this happens, your replica count is at y. The prior described scenario is where the responsibility of k8s scheduler ends. Now, comes the argo's own enforcement of y replica count (autosync+health enabled). This is where we need to tell argo to ignore y replica count (as specified in spec.replicas) and the above linked comment is what that achieves.

Now, during operational steady-state, keda.sh will take over the replica count completely and schedule replicas according to its own HPA policy driven by its own autoscaling policy (queue depth e.g.) using ScaledObject's min and max replicas.

With regards to:

X to fixed value and the to X again, impacting in your users if the reduction is big

Couple of things regarding service availability:

• Don't create one own's HPA (let Keda manage it)
• Peruse spec.advanced.restoreToOriginalReplicaCount set to false.
• Use a suitable stabilization window behaviour in ScaledObject.
• Use lifecycle hook in the container to manage availability https://github.com/kedacore/keda-docs/blob/main/content/docs/2.9/concepts/scaling-deployments.md#leverage-the-container-lifecycle

[dgrezza]

Is there a conclusion to managing KEDA with ArgoCD? since I got same issue

[JorTurFer]

You don't need to add any ignore. When you are using HPA (or any other scaling) you have to not set spec.replica or you will have problems because however you deploy (autosync, manual sync, manual apply, etc) you will override the HPA desired value for spec.replica with your set value.
Every time when you are using autoscaling, you have to keep spec.replica empty, just not set it in the yaml (this means that you shouldn't set 1 or 0, just keep it unset)

[NihalKA]

@JorTurFer Even though we removed spec.replica from a deployment the default deployment replica will be one according to kubernetes docs, Argocd is getting out of sync issue even this is removed. Adding https://argo-cd.readthedocs.io/en/stable/user-guide/diffing/#application-level-configuration is resolving the issue since it ignores replica count on checking sync status

[JorTurFer]

I do use ArgoCD in my daily basics, and I haven't ever needed to ignore the replicas value. If you don't set it in the workload, you don't need to ignore it. I'd suggest templating your manifests to double-check the yaml because I'm almost sure that the value is set at any point

[talha-naeem1]

Comment out replica parameter in your deployment yaml. and it will work. Thanks , this issue helped me actually.

  # replicas: 1

[NihalKA]

@JorTurFer Yes that worked. Thanks. There was replca: (without any values) coming from template which cause replicas:null. removed that and now out of sync issue is resolved.