Endless loop in deleting

[Zelldon]

We can have an endless loop in deleting:

�[36mzeebe_broker2 |�[0m 2021-10-27 14:14:18.668 [] [raft-server-2-raft-partition-partition-5] DEBUG
�[36mzeebe_broker2 |�[0m       io.camunda.zeebe.journal.file.SegmentedJournal - No segments can be deleted with index < 49499482 (first log index: 49463401)
�[36mzeebe_broker2 |�[0m 2021-10-27 14:14:18.669 [] [raft-server-2-raft-partition-partition-5] DEBUG
�[36mzeebe_broker2 |�[0m       io.camunda.zeebe.journal.file.SegmentedJournal - No segments can be deleted with index < 49499482 (first log index: 49463401)
�[36mzeebe_broker2 |�[0m 2021-10-27 14:14:18.670 [] [raft-server-2-raft-partition-partition-5] DEBUG
�[36mzeebe_broker2 |�[0m       io.camunda.zeebe.journal.file.SegmentedJournal - No segments can be deleted with index < 49499482 (first log index: 49463401)
�[36mzeebe_broker2 |�[0m 2021-10-27 14:14:18.671 [] [raft-server-2-raft-partition-partition-5] DEBUG
�[36mzeebe_broker2 |�[0m       io.camunda.zeebe.journal.file.SegmentedJournal - No segments can be deleted with index < 49499482 (first log index: 49463401)
�[36mzeebe_broker2 |�[0m 2021-10-27 14:14:18.672 [] [raft-server-2-raft-partition-partition-5] DEBUG
�[36mzeebe_broker2 |�[0m       io.camunda.zeebe.journal.file.SegmentedJournal - No segments can be deleted with index < 49499482 (first log index: 49463401)
�[36mzeebe_broker2 |�[0m 2021-10-27 14:14:18.672 [] [raft-server-2-raft-partition-partition-5] DEBUG
�[36mzeebe_broker2 |�[0m       io.camunda.zeebe.journal.file.SegmentedJournal - No segments can be deleted with index < 49499482 (first log index: 49463401)
�[36mzeebe_broker2 |�[0m 2021-10-27 14:14:18.673 [] [raft-server-2-raft-partition-partition-5] DEBUG
�[36mzeebe_broker2 |�[0m       io.camunda.zeebe.journal.file.SegmentedJournal - No segments can be deleted with index < 49499482 (first log index: 49463401)
�[36mzeebe_broker2 |�[0m 2021-10-27 14:14:18.675 [] [raft-server-2-raft-partition-partition-5] DEBUG
�[36mzeebe_broker2 |�[0m       io.camunda.zeebe.journal.file.SegmentedJournal - No segments can be deleted with index < 49499482 (first log index: 49463401)

Originally posted by @Zelldon in #8099 (comment)

[Zelldon]

@deepthidevaki found the cause #8099 (comment) 🙇

[npepinpe]

What's the impact on the cluster? Do we have any idea? Would it recover?

[Zelldon]

Hard to say. I mean it is stuck on OOD, with the endless loop it blocks useful resources to do snapshots on other partitions I can imagine. Restart might help to get out of the loop, but I think in general the cluster would need manual help, like increasing disk, delete data or something. In general it should not end in OOD in the first place and of course it should not end in an endless loop try to clean up data, where is not possible.

[deepthidevaki]

It is not an "endless loop" in which the thread is stuck for ever in this operation. Whenever follower receives an AppendRequest and it tries to write to the journal, it received OOD exception, and then it tries to "compact", which does nothing. So it is mostly just a wasteful work, and annoying logs. It will recover if the broker recovers from OOD.

[Zelldon]

Thanks @deepthidevaki for clarifying this, haven't checked the code before.

[npepinpe]

Let's remove the compact behavior here, as the follower can only compact on a new snapshot anyway.

[Zelldon]

Never occurred again I will close this for now