Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ensure RaftStore lock files are created and updated atomically #10681

Closed
deepthidevaki opened this issue Oct 12, 2022 · 0 comments · Fixed by #10683
Closed

Ensure RaftStore lock files are created and updated atomically #10681

deepthidevaki opened this issue Oct 12, 2022 · 0 comments · Fixed by #10683
Assignees
Labels
kind/bug Categorizes an issue or PR as a bug severity/high Marks a bug as having a noticeable impact on the user with no known workaround version:8.1.1 Marks an issue as being completely or in parts released in 8.1.1 version:8.2.0-alpha1 Marks an issue as being completely or in parts released in 8.2.0-alpha1 version:8.2.0 Marks an issue as being completely or in parts released in 8.2.0

Comments

@deepthidevaki
Copy link
Contributor

Describe the bug

We observed that on startup the lock file is empty, and this failed the startup of the partition with "Failed to acquire lock".
Possible reasons:

  1. Pod crashed after creating the lock file, but before writing the lock info
  2. Lock file was created and updated, but it was not flushed to disk before the pod restarted.

Workaround in this case is to delete the lock file and restart the pod. This is risky, because lock file exists to ensure that the directories are not shared by multiple pods. If we workaround it, we have to manually check if everything is configured correctly.

Expected behavior
The lock is not empty. Either lock file exists and have contents, or lock file does not exist.

Environment:

  • Zeebe Version: 8.1.0
@deepthidevaki deepthidevaki added kind/bug Categorizes an issue or PR as a bug severity/critical Marks a stop-the-world bug, with a high impact and no existing workaround labels Oct 12, 2022
@deepthidevaki deepthidevaki self-assigned this Oct 12, 2022
@deepthidevaki deepthidevaki added severity/high Marks a bug as having a noticeable impact on the user with no known workaround and removed severity/critical Marks a stop-the-world bug, with a high impact and no existing workaround labels Oct 12, 2022
zeebe-bors-camunda bot added a commit that referenced this issue Oct 13, 2022
10704: [Backport stable/8.1] Ensure raft storage lock file is update atomically r=deepthidevaki a=backport-action

# Description
Backport of #10683 to `stable/8.1`.

closes #10681

Co-authored-by: Deepthi Devaki Akkoorath <deepthidevaki@gmail.com>
zeebe-bors-camunda bot added a commit that referenced this issue Oct 13, 2022
10703: [Backport stable/8.0] Ensure raft storage lock file is update atomically r=deepthidevaki a=backport-action

# Description
Backport of #10683 to `stable/8.0`.

closes #10681

Co-authored-by: Deepthi Devaki Akkoorath <deepthidevaki@gmail.com>
@korthout korthout added the version:8.1.1 Marks an issue as being completely or in parts released in 8.1.1 label Oct 13, 2022
@korthout korthout added version:8.2.0-alpha1 Marks an issue as being completely or in parts released in 8.2.0-alpha1 release/8.0.8 labels Nov 1, 2022
@npepinpe npepinpe added the version:8.2.0 Marks an issue as being completely or in parts released in 8.2.0 label Apr 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Categorizes an issue or PR as a bug severity/high Marks a bug as having a noticeable impact on the user with no known workaround version:8.1.1 Marks an issue as being completely or in parts released in 8.1.1 version:8.2.0-alpha1 Marks an issue as being completely or in parts released in 8.2.0-alpha1 version:8.2.0 Marks an issue as being completely or in parts released in 8.2.0
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants