What Is the Recommended Approach to Configuring the Cluster Autoscaler?

[ryehowell]

What are some recommended configurations for the EKS/Kubernetes Cluster Autoscaler? Some general recommendations around the following will be helpful:

• CAS with Spot Instances
• CAS with multiple Instance Types
• What Expander/Strategy is best for my environment

---

Tracked in ticket #110830

[lorelei-rupp-imprivata]

EKS 1.27
Terraform 1.5.7
Terragrunt 0.54.2
Cluster autoscaler 1.27.5

We use these grunt works modules:
gruntwork-io/terraform-aws-eks/tree/master/modules/eks-cluster-managed-workersref=v0.65.2
gruntwork-io/terraform-aws-eks.git//modules/eks-k8s-cluster-autoscaler?ref=v0.65.2

For our Dev env for example this is a snippet of Node Group Configuration:

One ASG per Node Group Per AZ
We do not use mixed instance policies in a given ASG

For example our SPOT ASG/NG config would look like this
We have spot generic workers for most our services, and then gpu and istio specific services go to dedicated workers

app-workers_spot:
	instance_types: [ "m5.xlarge", "m5a.xlarge", "m6i.xlarge", "m6id.xlarge", "m7i.xlarge" ]
	capacity_type: "SPOT"
istiogw-workers:
      instance_types: [ "m5.large" ]
      tags:
        "as-label/env": "ingressgw"
      taints:
        - key: "agent"
          value: "istio"
          effect: "NO_SCHEDULE"
gpu-workers:
      instance_types: [ "g4dn.xlarge"  ]
      taints:
        - key: "agent"
          value: "processor_gpu"
          effect: "NO_SCHEDULE" 
Search-workers:
      instance_types: [ "m5.large", "m5a.large", "m6i.large", "m6a.large", "m6id.large", "m7i.large", "m7a.large" ]
      taints:
        - key: "agent"
          value: "search"
          effect: "NO_SCHEDULE"

For our env in us-east-2, we would end up with three ASG/Node Group one for us-east-1a, us-east-1b, us-east-1c for each of these node groups

Our Cluster Autoscaler config we set
scaling_strategy to “priority”
expander_priorities to
{ 10 = [".*"]
50 = [".spot."] }

We do not set any other config on the grunt works module today, so the rest of CA config is defaults in the upstream helm chart/and/or grunt works default settings

We have seen issues in our lower ends that use SPOT where we used to have m4.xlarge in the list as the first item in the instance types list. We would see issues where we would lose all our SPOT instances and then CA would try to scale up and in the ASG Activity in AWS Console you would see like 6 instances just sitting in “Pending:WAIT” for HOURS
By chance we ended up removing the m4.xlarge type and then we got instances almost immediately. We figured perhaps AWS just doesn’t have the capacity there anymore, but this happened quite a bit over the last few weeks till we figured this out.

We are looking though to understand if our Cluster Autoscaler config is good and if there are any updates we should be making to it, especially around the scaling strategy. I am not sure we fully understand when to use the different types. Also there are lots of other configuration options and we are interested to know if we should update to any of those. Especially when it comes to our production env, where we do NOT use SPOT instances, and we set scaling_strategy to “least-waste” which is the default.

[mateimicu]

Hi,

Scaling EKS is a challenging task. It is generally best to enable logging and observability to investigate any scheduling/scaling hiccups. For Cluster Autoscaler, you can make sure to have the verbosity level set to 4

You can also consider setting up alarms on Instances stuck in the Pending state or pods that failed to be scheduled.

There are multiple places for configuration you can look at:

Worker Pools

• asg_default_spot_allocation_strategy and AWS docs can affect how often you get interrupted for SPOT instances. Also, in the context of Cluster Autoscaler, it can reduce the chances of making a scaling decision for a SPOT instance type that is no longer available by the time it is scheduled.

• Check AWS Quotas Make sure your AWS Quotas allow for the number of instance types you are planning to use

• Use multiple instance types for SPOT instances if possible because the availability can change.

• Use multiple Availability Zones (even all) for SPOT instances. This gives you a better chance of catching available instances

Cluster Autoscaler

• balance-similar-node-groups can affect the scaling of one AZ compared to another. Ideally, this is set to true on production workloads for greater availability.

• Overprovisioning for production workloads is something you may consider. This is usually achieved with Priority Preemption

• scaling_strategy and expander_priorities: To use priority based expansion you need to have scaling_strategy set to priority or to a combination of expanders. E.g. priority,least-waste. See Expander Docs.

• expander_priorities In the example provided

  { 10 = [".*"]
  50 = [".spot."] }
  

  ".spot." may be a bug as the values for priorities is a regex and ".spot." will match exactly the string .spot.. You can use ".*spot.*" to match all scaling group ID's that contain spot`. See docs about expander priority for more info

More granularity (Karpenter)

The Karpenter offers better configurations for more granular control over the scaling process. It can handle interruption events and provide new capacity quickly. It also integrates with AWS better.

But it is a newer project compared to Cluster Autoscaler.

[lorelei-rupp-imprivata]

Thanks! We will take a look at these things!