Sidekiq "Scale to Zero" workers

[bradgessler]

Context: picking up a Twitter thread from https://twitter.com/getajobmike/status/1574896874674614273 that I started with @mperham

I've been working on "Scale to Zero"/auto-scaling background workers at Fly.io. During my initial research & experimentation phase I was able to get a Rails process to boot a VM, run a job, and shut down, but I discovered I'd need to re-invent a lot of the functionality Sidekiq (and other Rails worker libraries) have to handle job failures, retries, job status, etc. There's more at https://fly.io/ruby-dispatch/rails-background-jobs-with-fly-machines/, particularly around the "how", that goes into some of the details.

My question for the Sidekiq community, and maintainers, if there's interest in "Scale to Zero" workers in Sidekiq? The benefits are:

• Cost-savings - When there's no jobs, there's no servers to to pay for.
• Scale up - On the other side of the equation, if a bunch of jobs land on the queue, SIdekiq can rapidly scale up a bunch of VMs to handle spikes of jobs.

What makes Sidekiq particularly interesting and exciting for this problem now is Embedding, introduced in 7.0. I'm imagine a world where an embedded Sidekiq worker is handling the launching of Fly Machines when a job comes in, keeping them around for however long they're needed to process jobs on the queue, and then shut them down when they're no longer needed. What's great about this approach is the heavy lifting is done on other machines, which means the app server won't have to break much of a sweat.

If there is interest, there's a lot of questions around implementation. Its probably not worth getting into the specifics now, but most of the questions would be around what an adapter API would look like between Sidekiq::Job and the process that monitors those job threads. There would be some discussion too around how to make the Sidekiq::Manager exit after a certain amount of time has passed without processing jobs, which is the mechanism that would scale down workers on the Machines. Ideally this is implemented in a way that can plug into different processes like threads, VMs, docker containers, etc. (not just a proprietary integration).

I'm not familiar with how the Sidekiq community operates, so forgive me if I come across as aloof in my approach. I'm assuming most of the terminology I used above is 80% accurate and I'm not familiar with how features/enhancements make its way into main. I assume they go through Mike 😄.

[mperham]

Shouldn't cost-conscious users just embed Sidekiq into puma until they are comfortable and have the budget for running a worker 24/7?

I don't know exactly how Rails Autoscale works but I imagine it starts a thread running within the web process which checks queue latency and size within Redis every N seconds. It then issues a scale up or scale down command based on those metrics.

The Sidekiq API has all of the various APIs you need to monitor queues programmically for this type of logic with the caveat that it's not stateful -- it won't tell you if the queue size has been 0 for the last 30 seconds, you'd need to implement any stateful logic.

[bradgessler]

Shouldn't cost-conscious users just embed Sidekiq into puma until they are comfortable and have the budget for running a worker 24/7?

Yes, but I'm not thinking about folks who have tiny budgets, like $5/mo for a hobby website. I'm thinking about larger production websites that don't get a lot of traffic during off hours, but see lots of activity during the day. It would help out ops teams if they didn't have to worry so much about capacity planning and auto-scaling with a worker scheduler that was capable of spawning its own machines to process jobs.

In this case I see the cost savings applying to ops teams and devs of all sizes. Here's how:

1. The $5/mo folks would be happy spending $4.98/mo 😆
2. Smaller "real" Rails apps that are spending $100's/mo on hosting would save a few bucks and be happy about that, but I think they'd be more happy that they don't have to worry about capacity planning as much. These size teams don't necessarily even have the expertise to know they have a capacity problem.
3. Larger and medium size Rails shops would benefit by not having to look as closely at capacity planning their worker queues if a scheduler like this makes sense, which is great because they could focus on putting out other fires in their stack before they prioritize tuning worker queue capacity. At some point they can't escape capacity planning, but it's valuable buying these teams time.

The Sidekiq API has all of the various APIs you need to monitor queues programmically for this type of logic with the caveat that it's not stateful -- it won't tell you if the queue size has been 0 for the last 30 seconds, you'd need to implement any stateful logic.

Thanks! Good to know. If I'm understanding the terminology and docs correctly, I could have the process keep tabs on how many jobs it runs by incrementing a value somewhere in memory to track how many jobs were run within the past N seconds. Each second the process could add all of these values together and if it is zero, it would exit (and terminate the Machine).

[bradgessler]

Yeah! If I take a crack at it I'll create a fork of Sidekiq and open a PR for more concrete discussion on the implementation. Does that seem like a reasonable next step to you?

[mperham]

I think you'd be best served by creating an extension that works with Sidekiq as is. People will want to run stock Sidekiq, not a fork.

[jpcamara]

@bradgessler you may want to look at the implementation of how hirefire checks for job counts and latency here: https://github.com/hirefire/hirefire-resource/blob/master/lib/hirefire/macro/sidekiq.rb

Hirefire is similar to rails autoscale (but similarly only for heroku) and that's the code they use to monitor sidekiq usage

[seanders]

@bradgessler Hey just wondering. Did you make any progress on some sort of middleware/plugin for this behavior?

[bradgessler]

Nope! Haven’t even started on it. I’ll post updates on here if I take a crack at it. If somebody beats me to getting started, I’d be happy to advise along the way.

[nateberkopec]

When a job comes in, the Rails ActiveJob processor spins up a Fly Machine to process it.

This feels backwards to me. Shouldn't you create a Fly Machine when the number of jobs in a queue is > 0? Your approach of creating a machine per job is likely to get incredibly expensive for anyone enqueueing more than a few jobs. Plus, the point of a queue is that you put things in it and they can sit there and wait to be processed. Spawning up a Fly Machine for every job basically makes the queue completely unnecessary because a new machine is always available to process the job. Queues have no purpose if availability of the downstream processor is 100%.

[bradgessler]

Yeah, I wouldn't do that. What you quoted was a very rudimentary and basic "one machine per job" ActiveJob Fly Machine adapter. Towards the end of the article (https://fly.io/ruby-dispatch/rails-background-jobs-with-fly-machines/#activejob-might-not-be-the-right-place-to-solve-these-problems) you'll see me concluding that creating a new Fly Machine per job probably isn't a great idea for small jobs that require lower latency.

What I realize is a more sensible approach is to spin up a Fly Machine when the FIRST job comes rolling in. That machine would be running a sidekiq process that connects to the queue to start processing jobs. When the queue is empty for a certain period of time, that Machine would then shut down.

Queues have no purpose if availability of the downstream processor is 100%.

They're still useful for situations where a job fails and needs to be retried.