Increase lock duration and paginate in getBsdsIdentifiers

[benoitguigal]

D'après https://github.com/OptimalBits/bull?tab=readme-ov-file#important-notes

The queue aims for an "at least once" working strategy. This means that in some situations, a job could be processed more than once. This mostly happens when a worker fails to keep a lock for a given job during the total duration of the processing.

When a worker is processing a job it will keep the job "locked" so other workers can't process it.

It's important to understand how locking works to prevent your jobs from losing their lock - becoming stalled - and being restarted as a result. Locking is implemented internally by creating a lock for lockDuration on interval lockRenewTime (which is usually half lockDuration). If lockDuration elapses before the lock can be renewed, the job will be considered stalled and is automatically restarted; it will be double processed. This can happen when:

1 - The Node process running your job processor unexpectedly terminates.
2- Your job processor was too CPU-intensive and stalled the Node event loop, and as a result, Bull couldn't renew the job lock (see OptimalBits/bull#488 for how we might better detect this). You can fix this by breaking your job processor into smaller parts so that no single part can block the Node event loop. Alternatively, you can pass a larger value for the lockDuration setting (with the tradeoff being that it will take longer to recognize a real stalled job).

As such, you should always listen for the stalled event and log this to your error monitoring system, as this means your jobs are likely getting double-processed.

As a safeguard so problematic jobs won't get restarted indefinitely (e.g. if the job processor always crashes its Node process), jobs will be recovered from a stalled state a maximum of maxStalledCount times (default: 1).

Dans l'indexation de cette nuit, on voit que l'event loop a été bloquée à plusieurs moment lors de la récupération des identifiants.

Ce blocage a certainement empêché le lock de se renouveler et le job master a été marqué comme stalled. Le comportement par défaut dans ce cas là est de réessayer de run le job ( maxStalledCount=1) et la réindexation globale a donc eu lieu deux fois.

Cette PR inclut plusieurs corrections pour éviter que ça se reproduise :

• Augmente légèrement la valeur de lockDuration (de 30s à 2 minutes).
• Ajout d'une pagination par curseur dans getBsdIdentifiers.
• Set maxStalledCount=0 pour éviter que le job soit exécuté deux fois s'il se retrouve quand mêmestalled pour une raison ou une autre.
• Log l'événement en cas de job stalled

---

• Mettre à jour la documentation
• Mettre à jour le change log
• Documenter les manipulations à faire lors de la mise en production (sur le ticket Favro de release)
• S'assurer que la numérotation des nouvelles migrations est bien cohérente
• Informer le data engineer de tout changement de schéma DB

---

• Ticket Favro

[sonarcloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarCloud