perf: Speed up performance of queue.clean when called with a limit

[pxpeterxu]

Motivation

I was trying to clean a large queue with millions of entries. Originally, I ran:

import Queue from 'bull';
const queue = new Queue(...);
queue.clean(0, 'completed');

But this blocked Redis from saving any other records with the error message:

BUSY Redis is busy running a script. You can only call SCRIPT KILL or SHUTDOWN NOSAVE

I switched this to something more like:

import Queue from 'bull';
const queue = new Queue(...);

async function run() {
  let count = null;
  while (count == null || count >= 100) {
    count = (await queue.clean(0, 'completed', 100)).length;
  }
}

But each loop of 100 still took a really long time. Digging into the script, regardless of how many we try to delete, we're listing all the keys on every run of .clean()

bull/lib/commands/cleanJobsInSet-3.lua

Line 23 in 019d612

local jobIds = rcall(command, KEYS[1], 0, -1)

Changes

• Change the cleanJobsInSet script so that:
  • If we're trying to clean n entries, only fetch n keys at a time
  • (Note that we might have to loop through this if the first n entries don't have all the timestamps we need, but this should generally still be an improvement)
• Cleanup: make cleanJobsInSet's variables be all named for slightly easier readability

Testing

Tested in production with my queue -- this sped it up a lot! Also ran it on empty queues to make sure that we ended properly

Also added an automated test for some pagination edge cases

Note for reviewers

It's a lot easier to see changes with "Hide whitespace changes" turned on: https://github.com/OptimalBits/bull/pull/2205/files?diff=split&w=1

[manast]

@pxpeterxu thanks, looks good to me.

@roggervalf do we need to port this improvement to BullMQ?

[manast]

🎉 This PR is included in version 4.1.2 🎉

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot 📦🚀