Unexpected error in resolving last processed position.

[rodrigo-lourenco-lopes]

Describe the bug

Relates to: #9949

Snapshot director is unable to retrieve the last processed position because the stream processor is closed and in the reported error cases there was a version mismatch.

Error group.

To Reproduce

As soon as the stream processor is closed the snapshot director will close.

Expected behavior

If we detect that the stream processor needs to be closed then we also should close the partition a related services.

Log/Stacktrace

Full Stacktrace

java.lang.RuntimeException: Actor is closed
	at io.camunda.zeebe.scheduler.ActorJob.failFuture(ActorJob.java:170) ~[zeebe-scheduler-8.1.0.jar:8.1.0]
	at io.camunda.zeebe.scheduler.ClosedQueue.offer(ClosedQueue.java:84) ~[zeebe-scheduler-8.1.0.jar:8.1.0]
	at io.camunda.zeebe.scheduler.ClosedQueue.offer(ClosedQueue.java:15) ~[zeebe-scheduler-8.1.0.jar:8.1.0]
	at io.camunda.zeebe.scheduler.ActorTask.submit(ActorTask.java:99) ~[zeebe-scheduler-8.1.0.jar:8.1.0]
	at io.camunda.zeebe.scheduler.ActorControl.call(ActorControl.java:108) ~[zeebe-scheduler-8.1.0.jar:8.1.0]
	at io.camunda.zeebe.streamprocessor.StreamProcessor.getLastProcessedPositionAsync(StreamProcessor.java:418) ~[zeebe-workflow-engine-8.1.0.jar:8.1.0]
	at io.camunda.zeebe.broker.system.partitions.impl.AsyncSnapshotDirector.prepareTakingSnapshot(AsyncSnapshotDirector.java:230) ~[zeebe-broker-8.1.0.jar:8.1.0]
	at io.camunda.zeebe.broker.system.partitions.impl.AsyncSnapshotDirector.scheduleSnapshotOnRate(AsyncSnapshotDirector.java:191) ~[zeebe-broker-8.1.0.jar:8.1.0]
	at io.camunda.zeebe.scheduler.ActorJob.invoke(ActorJob.java:94) ~[zeebe-scheduler-8.1.0.jar:8.1.0]
	at io.camunda.zeebe.scheduler.ActorJob.execute(ActorJob.java:45) ~[zeebe-scheduler-8.1.0.jar:8.1.0]
	at io.camunda.zeebe.scheduler.ActorTask.execute(ActorTask.java:119) ~[zeebe-scheduler-8.1.0.jar:8.1.0]
	at io.camunda.zeebe.scheduler.ActorThread.executeCurrentTask(ActorThread.java:106) ~[zeebe-scheduler-8.1.0.jar:8.1.0]
	at io.camunda.zeebe.scheduler.ActorThread.doWork(ActorThread.java:87) ~[zeebe-scheduler-8.1.0.jar:8.1.0]
	at io.camunda.zeebe.scheduler.ActorThread.run(ActorThread.java:198) ~[zeebe-scheduler-8.1.0.jar:8.1.0]```

</p>
</details>

**Environment:**
- OS: <!-- [e.g. Linux] -->
- Zeebe Version: 8.1.0 and 8.3.0 snapshots
- Configuration: <!-- [e.g. exporters etc.] -->

[megglos]

ZDP-Triage:

• mostly noise when stream processor is already unhealthy
• would require to react on health changes to also shutdown the snapshot director
• putting as later for now as it is noise

[nicpuppa]

Happened again a prod cluster zeebe:8.2.12.

Logs here