Leader recovery fails because because the logstream does not contain events from snapshot

[lenaschoenburg]

Describe the bug

A transition to leader fails during recovery because the event with the snapshot position 1998448 was not found in the log stream.

Additionally, the error handling initiates a teardown of the stream processor which fails because closing the replay state machine causes an NPE.

To Reproduce
Unclear.

Expected behavior

As far as I understand, for leaders this is an unexpected exception that shouldn't happen.

Log/Stacktrace

Full Stacktrace

2022-06-19 14:55:10.617  INFO [zeebe-wrapper,,] 7 --- [      Thread-10] c.s.t.s.z.w.zeebe.ZeebeBrokerProcess     : java.lang.IllegalStateException: Expected to find event with the snapshot position 1998448 in log stream, but nothing was found. Failed to recover 'Broker-1-StreamProcessor-2'.
2022-06-19 14:55:10.617  INFO [zeebe-wrapper,,] 7 --- [      Thread-10] c.s.t.s.z.w.zeebe.ZeebeBrokerProcess     : 	at io.camunda.zeebe.engine.processing.streamprocessor.StreamProcessor.recoverFromSnapshot(StreamProcessor.java:287) ~[zeebe-workflow-engine-1.3.8.jar:1.3.8]
2022-06-19 14:55:10.617  INFO [zeebe-wrapper,,] 7 --- [      Thread-10] c.s.t.s.z.w.zeebe.ZeebeBrokerProcess     : 	at io.camunda.zeebe.engine.processing.streamprocessor.StreamProcessor.onActorStarted(StreamProcessor.java:124) ~[zeebe-workflow-engine-1.3.8.jar:1.3.8]
2022-06-19 14:55:10.617  INFO [zeebe-wrapper,,] 7 --- [      Thread-10] c.s.t.s.z.w.zeebe.ZeebeBrokerProcess     : 	at io.camunda.zeebe.util.sched.ActorJob.invoke(ActorJob.java:79) [zeebe-util-1.3.8.jar:1.3.8]
2022-06-19 14:55:10.617  INFO [zeebe-wrapper,,] 7 --- [      Thread-10] c.s.t.s.z.w.zeebe.ZeebeBrokerProcess     : 	at io.camunda.zeebe.util.sched.ActorJob.execute(ActorJob.java:44) [zeebe-util-1.3.8.jar:1.3.8]
2022-06-19 14:55:10.617  INFO [zeebe-wrapper,,] 7 --- [      Thread-10] c.s.t.s.z.w.zeebe.ZeebeBrokerProcess     : 	at io.camunda.zeebe.util.sched.ActorTask.execute(ActorTask.java:122) [zeebe-util-1.3.8.jar:1.3.8]
2022-06-19 14:55:10.617  INFO [zeebe-wrapper,,] 7 --- [      Thread-10] c.s.t.s.z.w.zeebe.ZeebeBrokerProcess     : 	at io.camunda.zeebe.util.sched.ActorThread.executeCurrentTask(ActorThread.java:97) [zeebe-util-1.3.8.jar:1.3.8]
2022-06-19 14:55:10.617  INFO [zeebe-wrapper,,] 7 --- [      Thread-10] c.s.t.s.z.w.zeebe.ZeebeBrokerProcess     : 	at io.camunda.zeebe.util.sched.ActorThread.doWork(ActorThread.java:80) [zeebe-util-1.3.8.jar:1.3.8]
2022-06-19 14:55:10.617  INFO [zeebe-wrapper,,] 7 --- [      Thread-10] c.s.t.s.z.w.zeebe.ZeebeBrokerProcess     : 	at io.camunda.zeebe.util.sched.ActorThread.run(ActorThread.java:189) [zeebe-util-1.3.8.jar:1.3.8]

Environment:

• Zeebe Version: 1.3.8

[lenaschoenburg]

Closed as user error in response to an availability issue caused by #9820