Sidekiq large batches and avoiding thread starvation #6128
Unanswered
RobsonKing
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Some times we need to enqueue a large batch of jobs (example: update x on every account)
When the queue gets back up, other jobs can get delayed. What are the strategies people use to mitigate this?
Option 1: create a new queue for lower priority work. Downside, extra cost to run additional queues
Option 2: I was hoping there was some way to use Batches from sidekiq pro to limit the number of threads a batch could use. Example: enqueue this whole batch of jobs, but only use a maximum of 3 threads so we leave threads for other jobs.
Option 3: make one job enqueue the next one. Downside, much more complicated to setup.
Anyone have any recommendations?
Beta Was this translation helpful? Give feedback.
All reactions