[Question] What exactly does ratelimit and groupKey, doesis it related to adding of data into queue or processing data?

[mai1x9]

Hi there, I have a conceptual question regarding ratelimit option,

const imageProcessQueue = new Queue('imageprocess', {
  redis: {
    host: process.env.REDIS_HOST,
    port: process.env.REDIS_PORT
  },
  limiter: {
    max: 5000, // limit 5000 jobs per 1 second
    duration: 1000,
    groupKey: "orgid"
  }
});

With above configurations, it means limit 5000 jobs per second. My question is what exactly it ratelimits. Does it ensures that not more than 5000 jobs per second. are added into the queue when imageProcessQueue.add() is called or
does it mean do not process more than 5000 jobs per second when imageProcessQueue.process() is called. ?

From documentation,

interface RateLimiter {
  max: number; // Max number of jobs processed
  duration: number; // per duration in milliseconds
  bounceBack?: boolean = false; // When jobs get rate limited, they stay in the waiting queue and are not moved to the delayed queue
  groupKey?: string; // allows grouping of jobs with the specified key from the data object passed to the Queue#add (ex. "network.handle")
}

It was mentioned groupKey, does it means it will only ratelimit those data points when added into queue have groupKey. Does grouping adds performance issues as it may create a new queue to hold grouped jobs ?

let's say if I ratelimit based on some ProductId, it means all data points in the queue having same ProductId would be grouped and ratelimited based on the unique ProductId value, or will it be ratelimited cumulatively irresepctive of ProductId value, It will just use ProductId as a word in grouping.

[manast]

It limits the processor. You can learn about rate limiting in detail here (it is for BullMQ but the same principles apply): https://blog.taskforce.sh/rate-limit-recipes-in-nodejs-using-bullmq/

[mai1x9]

@manast thank you for the amazing blog. It did clear out lot of doubts.

I have 2 question, after refering to blog,

1. What is exactly difference between Creating worker vs using queue.process() both does the same thing. Is there any significant difference where worker is more advantageous than queue.process()?

2. It was mentioned for cpu bound tasks use multiple workers,

worker1 = new Worker('ratelimit', ....)
worker2 = new Worker('ratelimit', ....)
worker3 = new Worker('ratelimit', ....)
worker4 = new Worker('ratelimit', ....)

Does it mean all 4 workers defines above runs on 4 different cores (parallel processing).

[manast]

I recommend you to use BullMQ instead of Bull if you are new to Bull, as Bull is just in maintain-mode, whereas BullMQ is where all the active development is taking place.