Kubernetes Autoscaling Empty Queue / Busy Worker #5023
Replies: 2 comments
-
If I understand what you are proposing it would have to be at the queue level. Autoscaling happens at the deployment level, the deployment is the collection of workers for any given queue. From the docs:
So if the stat on queue depth returns less desired replicas, but the stat on busy workers returns more desired workers (or the same count of desired workers), then we would get the result that you are aiming for. At least that is my understanding from the docs |
Beta Was this translation helpful? Give feedback.
-
To dig up an old discussion, is there a recommended approach for this scenario? Currently we have a wonky work-around where every 30 seconds we create a redis list of length
Then in the k8s
|
Beta Was this translation helpful? Give feedback.
-
Question: Is it possible to get the "sidekiq.busy" stat tagged with worker and/or queue?
Context: We are considering using Kubernetes autoscaling for some of our sidekiq workers. We have the
sidekiq.in_queue
stat that we can use for detecting if there are jobs waiting. However we have workers we would like to scale down to 0 if there are no jobs in the queue.The issue we anticipate: if the queue is empty and we only base autoscaling on queue depth, then when the last job is picked up, kubernetes will scale down the worker, even if the job is still running. For short-running jobs that would probably be fine as sidekiq has some time to finish before k8s kills the worker. However, the specific worker in question sometimes processes jobs that run for 30+ minutes.
Potential Solution: We can configure the autoscaler to use both queue depth and "busy count" to keep enough workers around to finish the last job. In order to do that however, we need to be able to target the "busy" stat at either the worker or queue level. Looking at the Stat code, this seems doable, but I'm not sure what the drawbacks/limitations are.
Do you think this is the right approach?
Beta Was this translation helpful? Give feedback.
All reactions