fix: notify new SnapshotReplicationListeners about missed replications

[lenaschoenburg]

Description

We are using SnapshotReplicationListeners to transition to inactive when a snapshot replication starts. Snapshot replication can start before any listeners have registered, which means that the listener will only be notified about snapshot replication finishing, triggering a transition to follower without first transitioning to inactive.

This PR introduces a new flag in the RaftContext to keep track of ongoing snapshot replications so that newly registered listeners can be notified about immediately.

Related issues

closes #8830

[romansmirnov]

@oleschoenburg, thanks for the PR. As already discussed, there is still at least one scenario in which the ZeebePartition transitions too late into the INACTIVE role.

Basically, when during the Broker startup procedure a snapshot is received, then this PR ensures that three transitions are happening in an expected order (and thus, there is always happening a transition to INACTIVE):

1. Initial transition to FOLLOWER
2. On snapshot replication start, transition to INACTIVE
3. On snapshot replication completed, transition to FOLLOWER

Now, what could happen is that the second transition to INACTIVE happens too late, and in the meantime, the log got compacted already, for instance:

1. First, by the initial transition to FOLLOWER the latest snapshot is installed and a stream processor is started in replay mode.
2. In the meantime, the last snapshot chunk is received which will cause a log compaction (i.e., the entire log gets deleted if the follower is lagging behind a lot):
   https://github.com/camunda-cloud/zeebe/blob/b470b4e8e9f821775d2237ce8e91b28d070ab586/atomix/cluster/src/main/java/io/atomix/raft/roles/PassiveRole.java#L764-L776
3. After a complete snapshot replication, the Raft starts receiving append requests. With these requests, new entries are appended to the log.
4. The stream processor receives a log stream reader and starts reading from the log to replay events to the runtime database.

At this stage, the installed snapshot during the first transition is not in-sync with the logstream anymore. Meaning, the stream processor might replay events that expect a certain element to be present in the (runtime) database which must not be the case. As a result, the stream processor potentially might fail with a NullPointerException.

With this PR, this might get resolved on its own. In this situation, it is sure that the transition to INACTIVE (and afterward again to FOLLOWER) will happen in any case. With that, the ZeebePartition will recover from that failure and the installed snapshot is in-sync with the logstream again.

However, as Zeebe is able to recover from that state on its own, I think, we can accept that scenario. @npepinpe, do you agree?

Please note: In theory, when the installed snapshot and the logstream are out of sync, and before an NPE is thrown and the transition to INACTIVE begins, it could happen that Zeebe takes its own snapshot for that given partition which is newer than the replicated snapshot (so that the replicated snapshot gets deleted eventually). As a consequence, with the next transition to FOLLOWER it will recover from that "new" taken snapshot and continue with replaying the events from the logstream. This would result in a potentially corrupted snapshot that does not contain the expected state. But given the fact, that the first snapshot is taken at least after one minute after starting the ZeebePartition, it is very likely that the transition to INACTIVE and then back to FOLLOWER happened in the meantime. So, I guess this scenario is not likely to happen.

[romansmirnov]

🚀

[npepinpe]

I'm not 100% sure it's acceptable, if I understood correctly. Is there a chance the follower will persist its incorrect state? After all, when we take a snapshot, we simply take the runtime and save that, without really checking if that runtime was tied to a particular term or anything. So if we'd applied the wrong things, is there a chance we could snapshot it, before we transition to inactive? I don't think so, but it's not 100% sure to me.

[romansmirnov]

@npepinpe,

So if we'd applied the wrong things, is there a chance we could snapshot it, before we transition to inactive? I don't think so, but it's not 100% sure to me.

No, it won't snapshot it. Unless the transition to INACTIVE happens after the firstTimeSnapshot, only then it could potentially snapshot it:

https://github.com/camunda-cloud/zeebe/blob/34c4014405165b5c77c8cb72223fe8a5b59d0cac/broker/src/main/java/io/camunda/zeebe/broker/system/partitions/impl/AsyncSnapshotDirector.java#L104-L106

But how likely is it? To my understanding, this is not very likely and should not happen. Otherwise, I would agree that snapshotting wrong things is not acceptable.

Assuming, we can be sure that the wrong things/state won't be snapshotted, do you agree that we can accept a potential NPE and/or apply wrong things to the runtime database? To me, this would be a transient situation that gets resolved by a transition to INACTIVE and subsequently to FOLLOWER. By transitioning to FOLLOWER, it will recover from the latest snapshot and replay the events correctly. WDYT?

Should we maybe create a separate issue for further discussion, if necessary?

[lenaschoenburg]

bors r+

[ghost]

Build succeeded:

• Run smoke tests on macos-latest
• continuous-integration/jenkins/branch
• Run smoke tests on ubuntu-latest
• Run smoke tests on windows-2022

[github-actions]

Successfully created backport PR #8853 for stable/1.2.

[github-actions]

Successfully created backport PR #8854 for stable/1.3.

[github-actions]

Successfully created backport PR #8855 for release-1.4.0-alpha2.