Questions / Best Practices / *Absolute* rate limiter?

[pvogel1967]

Description

Hi, I'm embarking on a project to move some of our processing into queued tasks rather than doing them synchronously with an API request and I'm looking for some best practices or at least a discussion of tradeoffs to consider as I design our approach.

First of all -- # of queues vs. named jobs within a queue? We have a variety of events that occur within our system that require some additional, somewhat time consuming processing that can be a problem when lots of those events happen in a short period of time. Should we think about each event as a separate queue or a different named job within a limited set of queues? What's a "reasonable" number of queues to have overall in a system?

What are the key stressors to redis CPU? We're running redis in cluster mode I assume you are managing key names to avoid any cross-slot issues? Would different queues hit different shards or do you use such a limited # of keys that the workload tends to be heavily concentrated on one shard?

Next -- I've seen the documentation and understand the O(n) vs. O(1) difference between using priorities on a job vs. having all jobs run at the same priority. Is it reasonable to just think about having, say, 3 separate queues with different rate limiting (and potentially different #'s of workers) and have named jobs in each of those queues?

Finally, I see the rate limiter, but it seems fairly, um, "limited" :-). I'd like to be able to say "There should never be more than "n" of these jobs being processed at any given time. When a job finishes if there are more jobs on the queue go ahead and start the next one". Is there a way to use the rate limiter options to enforce this kind of behavior?

Minimal, Working Test code to reproduce the issue.

(An easy to reproduce test case will dramatically decrease the resolution time.)

Bull version

3.22.1

Additional information

For context, I've done a trivial approach to moving things into background processing using redis queues and pub/sub to let containers know there's new work in a queue and an atomic lua script to maintain a count of how many tasks of a given type are currently running, if I get notification that there's a new job in the queue but I'm at capacity for that number of tasks, I don't start additional work and I just rely on the existing processing to get around to the new job since it pops new jobs off the queue until it's empty once it starts work. I'm just looking at bull as a more robust/full-featured pattern that's easier to utilize as additional opportunities for background processing present themselves.

[manast]

See my answers below

First of all -- # of queues vs. named jobs within a queue? We have a variety of events that occur within our system that require some additional, somewhat time consuming processing that can be a problem when lots of those events happen in a short period of time. Should we think about each event as a separate queue or a different named job within a limited set of queues? What's a "reasonable" number of queues to have overall in a system?

named jobs are no different from standard jobs, so from a design point of view it does not make any difference, its just a convenient feature. Normally you should have a manageable number of queues though.

What are the key stressors to redis CPU? We're running redis in cluster mode I assume you are managing key names to avoid any cross-slot issues? Would different queues hit different shards or do you use such a limited # of keys that the workload tends to be heavily concentrated on one shard?

In general CPU for the redis instance should not very high, typical max at 20%, but this can increase if you have a lot of delayed jobs (in the 100k order of magnitude). All keys belonging to a queue must be in the same redis instance, so there is no advantage in user cluster. However some users have experienced issues with clusters, so I recommend you to test it works in your particular setup and use this naming for your queues: https://github.com/OptimalBits/bull/blob/develop/PATTERNS.md#redis-cluster

Next -- I've seen the documentation and understand the O(n) vs. O(1) difference between using priorities on a job vs. having all jobs run at the same priority. Is it reasonable to just think about having, say, 3 separate queues with different rate limiting (and potentially different #'s of workers) and have named jobs in each of those queues?

You can, but named or not, the jobs in a given queue will executed in order, see my answer above.

Finally, I see the rate limiter, but it seems fairly, um, "limited" :-). I'd like to be able to say "There should never be more than "n" of these jobs being processed at any given time. When a job finishes if there are more jobs on the queue go ahead and start the next one". Is there a way to use the rate limiter options to enforce this kind of behavior?

The concurrency factor can only be specified at the queue level per worker. So the amount of jobs processed in parallel would be concurrency * num_workers.