No jobs get to consumers in production environment.

[Roquentin97]

Hello!

I'm developing a NestJs service heavily utilizing bull. For quite a while everything was fine. But recently, I got a problem in all hosted environments - no jobs reach consumers. I monitor flow with a bull-monitor. Every job from every queue gets to the active state, hangs there for ~30 seconds, and gets failed with a message job stalled more than allowable limit.

The jobs are processed in sandboxed processes.

const queues = [
  BullModule.registerQueueAsync({
    name: PROCESS_NEW_EVENT,
    useFactory: async () => ({
      defaultJobOptions: JOBS_CONFIG,
      processors: [
        {
          path: join(__dirname, 'jobs/handle-new-event.processor.js'),
          concurrency: DEFAULT_CONCURRENCY,
        },
      ],
    }),
  }),
];

@Module({
  imports: [
    AnotherModule,
    ...queues,
  ],
  providers: [Repository, Service],
  controllers: [Controller],
  exports: `[Repository,` ...queues],
})
export class EventModule {}

If jobs/handle-new-event.processor.js the following code

export default async function processNewEvent(job) {
  console.log('got job');
  await executor(job);
  console.log('job processed');
}

Then got job is never printed.

I thought it might be a problem with a Redis server. But everything is fine when I connect to the same server from the local environment.

I would appreciate any suggestions and directions.

[manast]

Either the workers are losing connection while processing the jobs, or the jobs are CPU intensive and hanging NodeJS Event loop for more than 30 seconds.

[Roquentin97]

Thank you for your reply!

I doubt the high CPU utilization is the case, as each job normally completes in a couple of seconds at most, and the service is basically idling until I fix the problem. And if it's the case, why the first log is never printed?

[manast]

Sorry, I don't know. Without a test case that proves the issue, I do not have a lot to work with. Btw, if you write a test it must be without nest-specific code as that is a code base owned by another team.

[Roquentin97]

Looks like the workers losing connection was the cause. Thank you!