Exponential backoff strategy for job retries

[strawhat5]

Is your feature request related to a problem? Please describe.
Currently whenever a job fails or gets timed out, it is retried immediately. This can result in issues such as #5063 where a job which was getting timed out (due to an application bug), was getting retried immediately and infinitely. It resulted in excessive generation of partition logs resulting in impact on the broker itself. In any and all cases, any application (worker) bug should not impact the availability or durability of the broker.

Describe the solution you'd like
We would like to have a config-based-solution where we can set the backoff retry strategy for a task while defining the task in BPMN itself. If not possible in BPMN, we should be able to define it programmatically in the job worker code. Also, we should be able to configure the max retries in case of timeouts. Currently in case of job timeout, it is retried infinitely impacting the broker.

Having such a solution will minimize the impact on the broker, and eventually on the worker.

Additional context
This is probably linked to #164, but covers configuration of retries in case of failures and timeouts plus a backoff retry strategy. Hazelcast clients have a similar backoff connection retry strategy implemented.

[npepinpe]

What's the use case for job time outs? Time outs are used in case the worker which grabbed the job crashed, so Zeebe knows to make it available again for other workers. It makes sense there to retry immediately, no? I get the idea for failures however - at the same time, when a job is marked as failed (meaning it ran out of retries), it cannot be retried immediately unless the user resolves the related incident. The onus is on the worker to correctly set the amount of retries a job still has when failing it (which you can simply take the current amount of retries and decrement it by one).

[strawhat5]

Hey Nicolas let me try to elaborate the use case for both failures and timeouts. Edit: Just to clarify when I am talking about retrying a failure, I am talking about retrying a fail command, before the Incident is raised.

1. Failures: Yes after an Incident is raised, the job remains in the failed state until user intervention. That is a good feature. What I am talking about is adding a custom backoff strategy for retries of fail commands. Suppose a job has 3 retries before raising an Incident, all the retries are tried immediately. It may happen that there was a transient error on client side, like a DB connection failure, which would be resolved automatically in few minutes. But the job worker will fail or throw an exception for each retry, if it is retried in a matter of seconds. And then user would have to intervene manually to resolve the incident. This can be avoided if we have a way to configure a retry strategy for each type of task. There we can specify whether to retry a job immediately, or in case of transient failures, have a custom retry strategy like trying it again after 5 minutes or 1 hour. That will give more leverage to the user to retry a task, without having to manually intervene for all (transient) failures. I think this will automatically be covered once I can set a retry timeout for failed tasks #164 is implemented.
2. Timeouts: We have already experienced the catastrophic impact of continuous timeouts on Zeebe broker, as detailed in Excessive number of log segments generated on one partition #5063. Timeouts can happen if the job was not completed before its deadline. In that case it will be assigned to another worker thread, which is again a great feature. But, currently it does not have a failsafe mechanism. So in cases where client is facing a connection error: like an external microservice API call taking longer than the specified timeout, or a DB connection failure (e.g. Job has a timeout of 20 seconds while JDBC manager waits for 30 seconds to get a connection before throwing an exception). In all such cases the job will be timed out before completing its processing. These cases when running on a large scale system will soon go out of hand and cause a flooding of log generation on raft-partitions. As described in Excessive number of log segments generated on one partition #5063, we have seen this error filling up over 150GB of raft-partition logs per broker:

Expected to time out activated job with key '6755399445248597', but it must be activated firstŽ¨deadlineÏs‹éH‡¦worker±logistics§retries¤type±logisticscustomHeaders€©variablesÄ€¬errorMessageÚÅjavax.persistence.PersistenceException: org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection

Yes we can avoid such issues by configuring the timeouts appropriately on client side. But in a large scale system, we cannot expect all clients to follow the standards and conventions. It makes more sense to have a failsafe mechanism in the broker, to avoid any client mishaps from impacting the availability of the broker. For these reasons I propose having a configurable retry strategy, where a user/client can configure the max number of retries along with a configurable retry interval, in case of both failures (job fail commands) and timeouts.

[npepinpe]

Sorry, but it's still unclear to me what's the use case for time outs 😅 If the job has a timeout of 20 seconds, and we have a failsafe that adds another 10 seconds time out, and the JDBC job takes 30 seconds and errors out, I'm not sure I see any benefits to this. It's just really increasing the time out in the end, no? It sounds a little like you don't trust your users/clients to configure their workloads properly? Zeebe isn't really built to handle malicious/abusive users (e.g. no multi tenancy), so in that sense I'm not sure this feature would particularly help.

[strawhat5]

Ah, my bad for not explaining it properly. 🙂

Let me take an example to help explain better. There are a fixed number of retries configured for every job (by default 3). So, here 3 retries act as failsafe in case of any exception happening on worker. Whatever happens inside the worker, any and all jobs assigned to that worker will be failed/completed within 3 retries. This is a great feature and helps both the worker and broker keep their (un)necessary load in check.

Now, coming to timeouts: There are no fixed number of retries in case of timeouts. Impact: A job getting timed-out will be reassigned infinitely until it gets completed. So there is no failsafe here. Now any timeout happening inside the worker, is also impacting the broker - because it is creating that many events and that much data.

So, what I am proposing here is to have a fixed number of retries for timeouts as well. So a job getting timed-out is not retried infinitely, rather an Incident is raised after 3 timeouts or so (user-configurable) - just like fail commands.

[marcelocquadros]

Nicolas, If I understood correctly, the feature requested is something similar RetryPolicy used in uber-cadence.
In the resilience4j framework the retry feature provides the same. Basically, if some task fails you will be able to retry it in 30s, then 60s, then 120s... until the service becomes healthy or the number of retries has exceeded.

[npepinpe]

I think this was partially implemented in #5626 and #5627. However, some parts are still missing, namely #5629. I will close this issue in favor of #5529. Let me know if I misunderstood the request.