doc: add textual description of the raft.Apply process #511
Conversation
huikang
force-pushed
the
doc-textual-description-apply
branch
from
cbb2678
to
22cdaea
Compare
docs/apply.md
Outdated
| 12. The actual place applying the commited log entries is in the main loop of `runFSM()`. | ||
|
|
||
| 13. After the log entries that contains the client req are applied to the fsm, the fsm | ||
| module will set the reponses to the client request (`req.future.respond(nil)`). |
There was a problem hiding this comment.
Just to close the loop on the whole process it might be worth noting that the future returned by raft.Apply should now be unblocked and calls to Error or Response should return the data at this point.
| 8. If all went well, the follower responds success (`resp.Success = true`) to the | ||
| `appendEntries` RPC call. |
There was a problem hiding this comment.
I am not exactly sure where in this flow but do you think it would be good to point out the stepping nature of the processLogs within the append entries RPC.
One RPC will contain new logs entries but the LeaderCommitIndex will lag behind those log indexes. For example the following flow is likely
AppenEntries(Log: 1..5, LeaderCommitIndex: 0)
AppendEntries(Log: 6..8, LeaderCommitIndex: 4)
AppendEntries(Log: 9, LeaderCommitIndex: 8)
AppendEntries(Log: , LeaderCommitIndex: 9)
Sometimes append entries rpcs wont contain new logs but only be bumping the commit index. Only if a server is far enough behind the majority of all servers will it ever see logs with a leader commit index which is greater or equal to the logs index within the rpc.
This subtlety is important because when the first AppendEntries RPC from my example above returns, the follower will not have queued up any applies to the FSM.
Due to this behavior its possible that a very small window of time exists when all followers have committed the log to disk, the write has been realized in the FSM of the leader but the followers have not yet applied the log to their FSM.
Paul's replies in this issue are relevant: #508
There was a problem hiding this comment.
@mkeeler , thanks for your suggestions. Totally agreed that it is worth to point out the details. Will send a new commit to update.
There was a problem hiding this comment.
Hi, @mkeeler , could you please take another look at the updated PR when you have a chance? Thanks.
docs/apply.md
Outdated
| Apply is the primary operation provided by raft. | ||
|
|
||
| Apply is the primary operation provided by raft. A client calls `raft.Apply` to apply | ||
| a command to the FSM. If no error is returned, the applied command is commited to a |
There was a problem hiding this comment.
Would it be useful to make a distinction between committed and applied here?
committed == durably stored to disk
applied == applied to the fsm.
I am thinking maybe its worthwhile to define both terms so that the reader knows the two are not interchangeable.
Co-authored-by: Matt Keeler <mkeeler@users.noreply.github.com>
huikang
force-pushed
the
doc-textual-description-apply
branch
from
9fa3509
to
b86d3df
Compare
Co-authored-by: Matt Keeler <mkeeler@users.noreply.github.com>
|
I deployed a consul cluster of 3 server nodes.I stoped all servers and start again, its possible that a very small window of time exists when consul returns stale data. The reason is that consul restores from the snapshot and needs to re-apply the committed but unapplied logs to fsm. In this case, the server may return stale data. |
|
If you perform a consistent query in Consul it should ensure all old logs have been applied to the FSM before reading/returning the data. ( The way that is implemented is that the Raft leader (where the RPC for the query will be forwarded) will perform a Raft |
No description provided.