A huge rejection reason causes an overflow in the record metadata

[saig0]

Describe the bug
When I deploy a big workflow that contains many violations (e.g. no job type, invalid expression) then I see the following failure in the log:

io.zeebe.util.retry.EndlessRetryStrategy - Catched exception class java.lang.IllegalArgumentException 
  with message invalid offset: -17568, will retry... 

As a result, the record and no further records are exported. After a restart of the broker, the same failure occurs. The failure is logged until the broker is stopped.

This bug was reported in the forum: https://forum.zeebe.io/t/error-io-zeebe-util-retry-endlessretrystrategy-invalid-offset-17568/2023/

Analysis

On the stream, we see that the deployment is rejected. It has a huge rejection reason that lists all the violations.

The underlying problem is that the rejection reason is too long. The rejection reason is part of the RecordMetada. The length of the metadata is decoded in the SBE record as a short. If the length of the metadata is greater than 32767 bytes then the deserialization of the record fails.

To Reproduce

1. Deploy a big workflow that contains many violations (for example: solaris_firmware_update.bpmn.txt)

Or run the following test case:

Unit Test

   // given
    final var builder = Bpmn.createExecutableProcess("p").startEvent();
    IntStream.range(0, 400).forEach(i -> builder.serviceTask("task" + i));
    final var workflow = builder.done();

    // when
    final var deployment = engine.deployment().withXmlResource(workflow).expectRejection().deploy();

    // then
    assertThat(deployment.getRecordType()).isEqualTo(RecordType.COMMAND_REJECTION);

Expected behavior
The deployment is rejected and the rejection is exported successfully.

Log/Stacktrace

[Broker-0-Exporter-1] [Broker-0-zb-fs-workers-1] ERROR io.zeebe.util.retry.EndlessRetryStrategy - Catched exception class java.lang.IllegalArgumentException with message invalid offset: -17568, will retry...

Full Stacktrace

2021-02-12 11:24:47.521 [Broker-0-Exporter-1] [Broker-0-zb-fs-workers-1] ERROR io.zeebe.util.retry.EndlessRetryStrategy - Catched exception class java.lang.IllegalArgumentException with message invalid offset: -17568, will retry...
java.lang.IllegalArgumentException: invalid offset: -17568
        at org.agrona.concurrent.UnsafeBuffer.boundsCheckWrap(UnsafeBuffer.java:1702) ~[agrona-1.8.0.jar:1.8.0]
        at org.agrona.concurrent.UnsafeBuffer.wrap(UnsafeBuffer.java:256) ~[agrona-1.8.0.jar:1.8.0]
        at io.zeebe.msgpack.spec.MsgPackReader.wrap(MsgPackReader.java:49) ~[zeebe-msgpack-core-0.25.3.jar:0.25.3]
        at io.zeebe.msgpack.UnpackedObject.wrap(UnpackedObject.java:29) ~[zeebe-msgpack-value-0.25.3.jar:0.25.3]
        at io.zeebe.logstreams.impl.log.LoggedEventImpl.readValue(LoggedEventImpl.java:135) ~[zeebe-logstreams-0.25.3.jar:0.25.3]
        at io.zeebe.engine.processing.streamprocessor.RecordValues.readRecordValue(RecordValues.java:35) ~[zeebe-workflow-engine-0.25.3.jar:0.25.3]
        at io.zeebe.broker.exporter.stream.ExporterDirector$RecordExporter.wrap(ExporterDirector.java:328) ~[zeebe-broker-0.25.3.jar:0.25.3]
        at io.zeebe.broker.exporter.stream.ExporterDirector.lambda$exportEvent$6(ExporterDirector.java:253) ~[zeebe-broker-0.25.3.jar:0.25.3]
        at io.zeebe.util.retry.ActorRetryMechanism.run(ActorRetryMechanism.java:36) ~[zeebe-util-0.25.3.jar:0.25.3]
        at io.zeebe.util.retry.EndlessRetryStrategy.run(EndlessRetryStrategy.java:50) ~[zeebe-util-0.25.3.jar:0.25.3]
        at io.zeebe.util.sched.ActorJob.invoke(ActorJob.java:73) [zeebe-util-0.25.3.jar:0.25.3]
        at io.zeebe.util.sched.ActorJob.execute(ActorJob.java:39) [zeebe-util-0.25.3.jar:0.25.3]
        at io.zeebe.util.sched.ActorTask.execute(ActorTask.java:122) [zeebe-util-0.25.3.jar:0.25.3]
        at io.zeebe.util.sched.ActorThread.executeCurrentTask(ActorThread.java:94) [zeebe-util-0.25.3.jar:0.25.3]
        at io.zeebe.util.sched.ActorThread.doWork(ActorThread.java:78) [zeebe-util-0.25.3.jar:0.25.3]
        at io.zeebe.util.sched.ActorThread.run(ActorThread.java:191) [zeebe-util-0.25.3.jar:0.25.3]

Environment:

• OS: Ubuntu
• Zeebe Version: 0.25.3, 0.26.1
• Configuration: Hazelcast exporter

[npepinpe]

Let's do it before 1.0 as it may require protocol changes which are always harder to keep backwards compatible.

[npepinpe]

Oops, looks like we didn't get there. But I imagine this should be do-able in a backwards compatible way? We should check that 🤔

[Zelldon]

Similar observed symptom we had with the web-modeler #11284 I think both can be solved with the same approach, you can read in the linked issue