fix: don't replicate snapshot if member already has the latest snapshot #9824
Conversation
lenaschoenburg
force-pushed
the
9820-fix-snapshot-replication-loop
branch
from
13a3a62
to
df0d417
Compare
There was a problem hiding this comment.
👍
If I understand correctly, the leader will keep track of each member's position to know which entry to send to them next. When we've replicated a snapshot at X, then we reset the member's context to X+1 (which means pointing at X such that the next entry is X+1). If there's no X, then we set the member's current index to 0? It seems a little strange to me, but I guess we really cannot set it to X 😄 Or would it be "OK" for the context to have X, as we treat the snapshot as that first entry? 💭 I wonder if it would make sense to have a different view in this context.
At any rate, just thinking out loud. Good catch!
|
Yep, that's summarized well I think 👍
Either way seems like they require a lot of changes in |
|
bors r+ |
9824: fix: don't replicate snapshot if member already has the latest snapshot r=oleschoenburg a=oleschoenburg ## Description This fixes an issue where a leader becomes stuck in a snapshot replication loop if the leader's log starts right after the snapshot index. After successfully installing a snapshot, the leader attempts to seek back to the log entry before the snapshot which will fail if the log starts after the snapshot. When that happens, the member context has no current entry until the next append request is send so we must not use the current entry of the member to decide between sending a snapshot or not. Instead, we can bail out and decide not to send a snapshot when the member has the latest snapshot. ## Related issues <!-- Which issues are closed by this PR or are related --> closes #9820 Co-authored-by: Ole Schönburg <ole.schoenburg@gmail.com>
|
Build failed: |
|
bors retry |
9824: fix: don't replicate snapshot if member already has the latest snapshot r=oleschoenburg a=oleschoenburg ## Description This fixes an issue where a leader becomes stuck in a snapshot replication loop if the leader's log starts right after the snapshot index. After successfully installing a snapshot, the leader attempts to seek back to the log entry before the snapshot which will fail if the log starts after the snapshot. When that happens, the member context has no current entry until the next append request is send so we must not use the current entry of the member to decide between sending a snapshot or not. Instead, we can bail out and decide not to send a snapshot when the member has the latest snapshot. ## Related issues <!-- Which issues are closed by this PR or are related --> closes #9820 Co-authored-by: Ole Schönburg <ole.schoenburg@gmail.com>
|
Build failed: |
|
bors retry |
|
Build succeeded: |
|
Successfully created backport PR #9830 for |
|
Successfully created backport PR #9831 for |
|
@oleschoenburg I think this should've been added to the release notes. |
Description
This fixes an issue where a leader becomes stuck in a snapshot replication loop if the leader's log starts right after the snapshot index. After successfully installing a snapshot, the leader attempts to seek back to the log entry before the snapshot which will fail if the log starts after the snapshot. When that happens, the member context has no current entry until the next append request is send so we must not use the current entry of the member to decide between sending a snapshot or not. Instead, we can bail out and decide not to send a snapshot when the member has the latest snapshot.
Related issues
closes #9820
Definition of Done
Not all items need to be done depending on the issue and the pull request.
Code changes:
backport stable/1.3) to the PR, in case that fails you need to create backports manually.Testing:
Documentation:
Please refer to our review guidelines.