simulate hanging disk i/o (theory: raft does not trigger leader election)

[stapelberg]

On 2022-April-19, one of the RobustIRC servers showed what looked like symptoms of a hanging storage controller.

The server was still reachable over the network, but any reads/writes would just hang. (I observed this with an interactive shell session I still had open.)

Interestingly enough, it seems like Raft triggers a leader election when the server is unreachable on the network, but doesn’t detect when the leader can’t make progress.

We should try to simulate this and see if there’s anything we can do to improve the current failure mode.

[stapelberg]

Can reproduce, reported upstream at hashicorp/raft#503