Improve coordination of Raft and Zeebe role transitions

[romansmirnov]

Description

This issue is an outcome of the discussions in #8717. In a nutshell, this issue is about improving the coordination of Raft and Zeebe role transitions that happens asynchronously. Currently, if the Raft layer transitions to FOLLOWER then the corresponding transition to FOLLOWERin the Zeebe layer happens asynchronously. That current approach causes different issues, things to consider, and seems to overly complicate the communication between these layers, for instance:

• Despite the current approach trying to keep the transition concerns separate, there are still cases in which the Zeebe layer directly access the Raft layer to do some checks, for example, when installing the Log Storage to ensure that there wasn't a leader change in the meantime:
  https://github.com/camunda-cloud/zeebe/blob/1081002029552e091779869fc419efc400993f7b/broker/src/main/java/io/camunda/zeebe/broker/system/partitions/impl/steps/LogStoragePartitionTransitionStep.java#L113-L128
• Ensure to throw the right exceptions for recoverable cases.
• Since the current implementation is a result of a refactoring, some of the assumptions do not apply anymore. For example, in ZeebePartition#handleRecoverableFailure() it assumes that context.getCurrentRole() returns the role that the Zeebe layer tried to transition to. But actually context.getCurrentRole() returns the role it was trying to transition from:
  https://github.com/camunda-cloud/zeebe/blob/1081002029552e091779869fc419efc400993f7b/broker/src/main/java/io/camunda/zeebe/broker/system/partitions/ZeebePartition.java#L341-L364
• Recently, different issues are arising:
  • Raft transitions to INACTIVE in case of a failure during the installation of the services: When installing services fails during a transition, the partition does not recover and becomes inactive #8717
  • Missed transition in the Zeebe layer causing the usage of an "old" snapshot: NPE during replay #8830
  • Partition without leader where all brokers are listed as followers #8978
  • ...

To Do

• In the first place, this issue is about to reconsider the current approach and come up with a concept to implement

[romansmirnov]

@npepinpe, according to your last comment in #8717, I raised this issue. Feel free to adjust the description as necessary.

[npepinpe]

I would like to dedicate some time to this in Q2. I'm still working on how to formulate more precisely what I want, but what you wrote is a good start. I think there are more issues even than those you mentioned.

[menski]

As already mentioned this is a topic we should try to put on the roadmap for Q2, I removed it from the project board for now. @npepinpe please bring it to our roadmap planning.

[romansmirnov]

Added another related issue #8978