RuntimeException: Cannot handle event with version newer than what is implemented by broker (4 > 3)

[remcowesterhoud]

Describe the bug

Since Aug 1st this exception has been occurring steadily, and still is in the latest SNAPSHOT version of Zeebe.

Likely caused by the protocol version change in #9926

To Reproduce

Expected behavior

Log/Stacktrace

https://console.cloud.google.com/errors/detail/CMeVr4q4s7Pz8wE;service=zeebe;time=P7D?project=camunda-cloud-240911

Full Stacktrace

 java.lang.RuntimeException: java.lang.RuntimeException: Cannot handle event with version newer than what is implemented by broker (4 > 3)
	at io.camunda.zeebe.streamprocessor.ReplayStateMachine.lambda$replayNextEvent$4(ReplayStateMachine.java:169) ~[zeebe-workflow-engine-8.1.0-SNAPSHOT.jar:8.1.0-SNAPSHOT]
	at io.camunda.zeebe.scheduler.future.FutureContinuationRunnable.run(FutureContinuationRunnable.java:33) ~[zeebe-scheduler-8.1.0-SNAPSHOT.jar:8.1.0-SNAPSHOT]
	at io.camunda.zeebe.scheduler.ActorJob.invoke(ActorJob.java:94) ~[zeebe-scheduler-8.1.0-SNAPSHOT.jar:8.1.0-SNAPSHOT]
	at io.camunda.zeebe.scheduler.ActorJob.execute(ActorJob.java:45) ~[zeebe-scheduler-8.1.0-SNAPSHOT.jar:8.1.0-SNAPSHOT]
	at io.camunda.zeebe.scheduler.ActorTask.execute(ActorTask.java:119) ~[zeebe-scheduler-8.1.0-SNAPSHOT.jar:8.1.0-SNAPSHOT]
	at io.camunda.zeebe.scheduler.ActorThread.executeCurrentTask(ActorThread.java:106) ~[zeebe-scheduler-8.1.0-SNAPSHOT.jar:8.1.0-SNAPSHOT]
	at io.camunda.zeebe.scheduler.ActorThread.doWork(ActorThread.java:87) ~[zeebe-scheduler-8.1.0-SNAPSHOT.jar:8.1.0-SNAPSHOT]
	at io.camunda.zeebe.scheduler.ActorThread.run(ActorThread.java:198) ~[zeebe-scheduler-8.1.0-SNAPSHOT.jar:8.1.0-SNAPSHOT]
Caused by: java.lang.RuntimeException: Cannot handle event with version newer than what is implemented by broker (4 > 3)
	at io.camunda.zeebe.streamprocessor.RecordProtocolVersionFilter.applies(RecordProtocolVersionFilter.java:19) ~[zeebe-workflow-engine-8.1.0-SNAPSHOT.jar:8.1.0-SNAPSHOT]
	at io.camunda.zeebe.streamprocessor.MetadataFilter.lambda$and$0(MetadataFilter.java:28) ~[zeebe-workflow-engine-8.1.0-SNAPSHOT.jar:8.1.0-SNAPSHOT]
	at io.camunda.zeebe.streamprocessor.MetadataEventFilter.applies(MetadataEventFilter.java:25) ~[zeebe-workflow-engine-8.1.0-SNAPSHOT.jar:8.1.0-SNAPSHOT]
	at io.camunda.zeebe.streamprocessor.ReplayStateMachine.replayEvent(ReplayStateMachine.java:224) ~[zeebe-workflow-engine-8.1.0-SNAPSHOT.jar:8.1.0-SNAPSHOT]
	at java.util.Iterator.forEachRemaining(Unknown Source) ~[?:?]
	at io.camunda.zeebe.streamprocessor.ReplayStateMachine.lambda$tryToReplayBatch$5(ReplayStateMachine.java:210) ~[zeebe-workflow-engine-8.1.0-SNAPSHOT.jar:8.1.0-SNAPSHOT]
	at io.camunda.zeebe.db.impl.rocksdb.transaction.ZeebeTransaction.run(ZeebeTransaction.java:84) ~[zeebe-db-8.1.0-SNAPSHOT.jar:8.1.0-SNAPSHOT]
	at io.camunda.zeebe.streamprocessor.ReplayStateMachine.tryToReplayBatch(ReplayStateMachine.java:208) ~[zeebe-workflow-engine-8.1.0-SNAPSHOT.jar:8.1.0-SNAPSHOT]
	at io.camunda.zeebe.streamprocessor.ReplayStateMachine.lambda$replayNextEvent$3(ReplayStateMachine.java:165) ~[zeebe-workflow-engine-8.1.0-SNAPSHOT.jar:8.1.0-SNAPSHOT]
	at io.camunda.zeebe.scheduler.retry.ActorRetryMechanism.run(ActorRetryMechanism.java:36) ~[zeebe-scheduler-8.1.0-SNAPSHOT.jar:8.1.0-SNAPSHOT]
	at io.camunda.zeebe.scheduler.retry.RecoverableRetryStrategy.run(RecoverableRetryStrategy.java:48) ~[zeebe-scheduler-8.1.0-SNAPSHOT.jar:8.1.0-SNAPSHOT]
	at io.camunda.zeebe.scheduler.ActorJob.invoke(ActorJob.java:92) ~[zeebe-scheduler-8.1.0-SNAPSHOT.jar:8.1.0-SNAPSHOT]

Environment:

• OS: Cloud
• Zeebe Version: zeebe:SNAPSHOT

[deepthidevaki]

#7449