queue middlewares

[dbrab]

Description

I have several small servers with 1gb ram, currently the queues are processed evenly, but the amount of memory used by each queue is constant but variable over time, there are times when 1 single server has 90% of the memory occupied, while the rest is 40 or 50%, because the queues are processed equally, I read the documentation and I can't find a place where I can set the limit of memory / disk / cpu usage or by x condition, for this I first thought of a simple setInterval , to check memory, disk, cpu or whatever variable you need every X minutes and pause the queues and/or resume when the amount is valid.

audioQueue.pause()
setInterval(async ()=>{
    if(await isAvailableMemory()){
        audioQueue.resume();
    }else{
        audioQueue.pause()
    }
},1000);

It works for me, I feel that something is missing, then try to create middlewares based on express (I tried based on koa, but it seems that it would need a little more work)

audioQueue.use(async function(next){
    if(!(await isAvailableMemory())){
        return false;
    }
    next();
})

queue.js

Queue.prototype.use=function(cb){
  this.middlewares=this.middlewares??[];
  this.middlewares.push(cb);
  return this;
}
Queue.prototype.execute=async function(){
  let prevIndex=-1;
  const reject=false;
  const runner= async (index)=>{
    if(index===prevIndex){
      throw new Error('next() called multiple times');
    }
    prevIndex=index;

    const middleware=this.middlewares[index];

    if(middleware){
       const result= await middleware(()=>{
        return runner(index+1);
      });
      if(result!=undefined){
        throw new Error('middleware return');
      }
     return result;
    }
  }
  try{
    return Promise.resolve(runner(0));
  }catch(e){
    return Promise.reject(e);
  }
}

Queue.prototype.processJobs = function(index, resolve, job) {
    this.execute()
    .then((ok)=>{
    process.nextTick(()=>{
      this._processJobOnNextTick(processJobs, index, resolve, job);
    });
    }).catch(e=>{
    })
};

Currently both solutions work well for me, I lean more towards the middleware solution, because it could be scaled/improved for different types of solutions, such as sessions or rate limiter, which is not the goal of bull, but it would be a good complement.

I would like to know if this topic has been discussed, because I did not find anything in the issues, I would love this feature.