Clients can set an optional backoff time to the failed tasks.

[aivinog1]

Is your feature request related to a problem? Please describe.
This is the client's part of #164. The idea is that a client (Java/Go/Zbctl) can specify an optional backoff time when sending a JobFailed command. If a backoff time is set then the job will be available after the backoff time. Otherwise, the job will be available immediately (current behavior).

Describe the solution you'd like
Clients can set an optional backoff time.

Describe alternatives you've considered
We could split this task into three for different clients.

[tfnick]

this feature is useful

[aivinog1]

Hey @saig0, @npepinpe! I have a question about implementation. So, when I try to use:

CLIENT_RULE
        .getClient()
        .newFailCommand(jobKey)
        .retries(1)
        .retryBackoff(backoffTimeout)
        .requestTimeout(Duration.ofDays(1))
        .send()
        .join();

in this test, the synchronous call became locked for the whole backoffTimeout duration. I think that this happens because we are expecting some response (JobRecord with a fail) and we will wait for it. I don't think that this is the right behavior but maybe I'm wrong 🤔

[npepinpe]

👀 I wasn't really involved so far, so I'd have to look into what this means on Monday.

[B-hamza]

Oh, that will be cool to set a backoff from the client on failing jobs, is there any update on this issue ? Many thanks.

[saig0]

@aivinog1 do you want to continue working on the topic?
I would like to see the feature in version 8.1 (Q3 2022). If you don't want or can't work on it then I would assign it to someone else.

[aivinog1]

Hey @saig0!
Sure, I'm happy to work on it soon :) But can you help me with this question? :)

[saig0]

@aivinog1 I'm sorry for overseeing your question 🙈

Do you have your changes on a branch? Please share a link. I can't reproduce the behavior otherwise.

In general, the request waits until the response is received (when using .send().join()). But the processor should also send the response if a backoff is set.

[aivinog1]

@saig0 Here is the test that is failing: aivinog1@78f67ef#diff-ef4fccf13ed0e32bc54cf53556fcbeb151f2c6636988f33bc7b254e0c81a4aceR78

[aivinog1]

@saig0 I think that I found the problem. So I'll need to do flush on the responseWriter somehow. I've tried to add responseWriter.flush(); like this, and it works but breaks other cases 🤔 :

17:05:17.324 [Broker-0-StreamProcessor-1] [Broker-0-zb-actors-1] ERROR io.camunda.zeebe.processor - Expected to execute side effects for record 'LoggedEvent [type=0, version=0, streamId=1, position=5, key=-1, timestamp=1652522717321, sourceEventPosition=-1] RecordMetadata{recordType=COMMAND, intentValue=255, intent=CREATE, requestStreamId=1, requestId=1, protocolVersion=3, valueType=PROCESS_INSTANCE_CREATION, rejectionType=NULL_VAL, rejectionReason=, brokerVersion=8.1.0}' successfully, but exception was thrown.
java.lang.NullPointerException: null
	at java.util.Objects.requireNonNull(Objects.java:208) ~[?:?]
	at io.camunda.zeebe.broker.transport.commandapi.CommandResponseWriterImpl.tryWriteResponse(CommandResponseWriterImpl.java:99) ~[classes/:?]
	at io.camunda.zeebe.engine.processing.streamprocessor.writers.TypedResponseWriterImpl.flush(TypedResponseWriterImpl.java:114) ~[classes/:?]
	at io.camunda.zeebe.util.retry.ActorRetryMechanism.run(ActorRetryMechanism.java:36) ~[classes/:?]
	at io.camunda.zeebe.util.retry.AbortableRetryStrategy.run(AbortableRetryStrategy.java:44) ~[classes/:?]
	at io.camunda.zeebe.util.sched.ActorJob.invoke(ActorJob.java:79) ~[classes/:?]
	at io.camunda.zeebe.util.sched.ActorJob.execute(ActorJob.java:44) ~[classes/:?]
	at io.camunda.zeebe.util.sched.ActorTask.execute(ActorTask.java:122) ~[classes/:?]
	at io.camunda.zeebe.util.sched.ActorThread.executeCurrentTask(ActorThread.java:103) ~[classes/:?]
	at io.camunda.zeebe.util.sched.ActorThread.doWork(ActorThread.java:83) ~[classes/:?]
	at io.camunda.zeebe.util.sched.ActorThread.run(ActorThread.java:195) ~[classes/:?]

Also, I think that it works without retryBackoff because retries do so fast so in the end we quickly activate jobs and flush them.

[saig0]

@aivinog1 I see. Thank you for sharing 👍

I was able to fix the behavior. See my commit here: a7195f7