Follower cannot receive snapshot because "chunk received out of order"

[deepthidevaki]

Describe the bug

 io.atomix.raft.roles.LeaderAppender - RaftServer{raft-partition-partition-2} - Failed to send InstallRequest{currentTerm=191, leader=2, index=6790469, term=188, version=1, chunkId=HeapByteBuffer{position=0, remaining=10, limit=10, capacity=10, mark=java.nio.HeapByteBuffer[pos=0 lim=10 cap=10], hash=1628710428}, nextChunkId=HeapByteBuffer{position=0, remaining=10, limit=10, capacity=10, mark=java.nio.HeapByteBuffer[pos=0 lim=10 cap=10], hash=1777397309}, data=HeapByteBuffer{position=0, remaining=8821580, limit=8821580, capacity=8821580, mark=java.nio.HeapByteBuffer[pos=0 lim=8821580 cap=8821580], hash=2050789216}, initial=true, complete=false} to member 1, with RaftError{type=ILLEGAL_MEMBER_STATE, message=Request chunk is was received out of order}. Restart sending snapshot.

Broker 1 is not ready because Partition-2 has not caught up with the leader. The leader is Broker 2. Initially there are lot of timeouts which leads to frequent leader election, and broker 2 becomes leader again and again. This happens for a few minutes. After that there are no timeouts. No more leader election. Broker 2 stays as the leader without any interruption.

Now Broker 2 is trying to replicate the snapshot to Broker 1. But it is caught in a loop with error - "message=Request chunk is was received out of order}. Restart sending snapshot." It is expected to be able to recover from this error, but it did not happen. Leader keeps restart sending the snapshot, but follower rejects with the same error response. No new snapshot is taken on the leader. So leader tries replicating the same snapshot.

When the follower was restarted it was able to come out of this error loop, and was thus able to complete the start up.

To Reproduce

(Trying to write a reproducer test for it).

Expected behavior

Brokers can recover from this error without a restart.

Related Support Case:- SUPPORT-14296

Version: 8.0.4 (and previous versions)

[Zelldon]

@deepthidevaki can you add a version and config to the issue please?