RandomizedRaftTest.livenessTestWithNoSnapshot fails because member is ACTIVE not READY

[oleschoenburg]

org.opentest4j.AssertionFailedError: 
 expected: READY
 but was: ACTIVE
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
	at io.atomix.raft.ControllableRaftContexts.lambda$assertAllMembersAreReady$20(ControllableRaftContexts.java:516)
	at java.base/java.util.HashMap$Values.forEach(HashMap.java:1065)
	at io.atomix.raft.ControllableRaftContexts.assertAllMembersAreReady(ControllableRaftContexts.java:516)
	at io.atomix.raft.RandomizedRaftTest.livenessTest(RandomizedRaftTest.java:226)
	at io.atomix.raft.RandomizedRaftTest.livenessTestWithNoSnapshot(RandomizedRaftTest.java:141)

surefire-reports.zip

[oleschoenburg]

Here's what jqwik has to say:

                              |-------------------jqwik-------------------
tries = 3                     | # of calls to property
checks = 3                    | # of not rejected calls
generation = RANDOMIZED       | parameters are randomly generated
after-failure = PREVIOUS_SEED | use the previous seed
when-fixed-seed = ALLOW       | fixing the random seed is allowed
edge-cases#mode = NONE        | edge cases are not explicitly generated
seed = -7949969560383210802   | random seed to reproduce generated values

I can't reproduce it with those seeds.

[deepthidevaki]

Was able to reproduce it with seed = "6434342535110401459" on main at commit bb335699f503a467a58f354c9631305a57f39e4d. @oleschoenburg was also able to reproduce it with the same seed. No idea why the seed from the CI does not reproduce failure.

[deepthidevaki]

This is what happened:

1. Node 1 is the leader
2. Leader sends an AppendRequest to node 2, before sending it updates RaftMemberContext#inFlightAppendCount to 1.
3. Node 1 steps down
4. Node 1 becomes leader again
5. Node 1 resets the member context of 2. So inFlightAppendCount is reset to 0
6. Node 1 sends a new append request to node 2, inFlightAppendCount is set to 1
7. The first request times out. On handling the response it decrements the inFlightAppendCount to 0. BUG! This request was sent when this node was leader in the previous term. So the response should not be handled in this term.
8. The second request times out. On handling the response it decrements the inFlightAppendCount to -1.
9. Next time, when the leader attempts to send an append request RaftMemberContext#canAppend returns false because inFlightAppendCount != 0. As a result leader will never sent a heartbeat or new append request to node 2. Thus node 2 can never become ready.