Could not take snapshot on followers because the position doesn't exist

[deepthidevaki]

Describe the bug

Found the following error in a cluster

java.lang.IllegalStateException: Failed to take snapshot. Expected to find an indexed entry for determined snapshot position 243718629 (processedPosition = 243718629, exportedPosition=243718630), but found no matching indexed entry which contains this position.

On further investigation, this happened because the follower received a new snapshot from the leader, and it reset the log. After that it did not receive any new events because of network issues. Then later the follower tries to take a snapshot, but since the log is empty it cannot find the index of the position. In this case this is not actually an error because nothing has been processed/exported yet. So ideally this error case should be handled differently on follower and leader.

The impact of this is nothing. The follower will eventually takes the snapshot when it starts receiving the new events.

Expected behavior

• If possible, not log this as an error on the follower. But this is most likely an error (inconsistent state) on the leader.
• It would be good if the follower does not attempt to take a snapshot if its log is not up-to-date with the processed and exported positions in the latest snapshot.

Log/Stacktrace

https://console.cloud.google.com/errors/CJu-z5WFm_fKVw?service=zeebe-broker&version=release-1-2-0&time=P7D&project=zeebe-io

Environment:

• Zeebe Version: release candidate 1.2.0

[npepinpe]

Assumption is it's noise, and an expected case on the follower which we should handle as it's not an error - but it should be harmless, and will recover by itself. Although it might be difficult to differentiate between the cases where it's expected, and where it isn't.

EDIT: one option could be to log as warning on the follower, as it will most likely recover, but should be treated as an error if it occurs many times.

[npepinpe]

Still happening as of today (last occurred on 22nd of June). To avoid worrying users for a non-issue, let's fix it.