Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Align k8s pods auto-restart with CP members removal and promotion #24912

Open
arodionov opened this issue Jun 28, 2023 · 2 comments
Open

Align k8s pods auto-restart with CP members removal and promotion #24912

arodionov opened this issue Jun 28, 2023 · 2 comments
Labels
Module: CP Subsystem Module: Kubernetes Source: Community PR or issue was opened by a community user Source: Internal PR or issue was opened by an employee Team: Core Type: Enhancement Type: Feature Request

Comments

@arodionov
Copy link
Contributor

arodionov commented Jun 28, 2023 •
edited

In the case of running the CP Subsystem on k8s, during podes (auto) restart:

  • if CP Persistence is enabled: will be tolerated correctly
  • if CP Persistence is disabled: restarted CP member should be removed from the CP subsystems (manually, or after the timeout, or via graceful shut-down) and then promoted manually to CP member again (since it changes the CP UUID).

In addition to usability issues, such behaviour makes the CP Subsystem on k8s unreliable if a majority of pods with CP members will be (auto) restarted.

Other problematic points #24903 (comment)

Internal JIRA ticket
@arodionov
Copy link
Contributor Author

arodionov commented Jun 28, 2023 •
edited

Community request #24897 (comment)

@arodionov arodionov added Source: Internal PR or issue was opened by an employee Source: Community PR or issue was opened by a community user Type: Feature Request labels Jun 28, 2023
@arodionov arodionov mentioned this issue Aug 17, 2023
Open
@lprimak
Copy link
Contributor

lprimak commented Aug 17, 2023

See #24897 description for many different instances of the instability issues.

If you run https://github.com/flowlogix/hazelcast-issues on 3 terminals, (stop/restart, use Ctrl-z) in about 10 minutes you will see all those issues in action.

This is how I found all these out.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Module: CP Subsystem Module: Kubernetes Source: Community PR or issue was opened by a community user Source: Internal PR or issue was opened by an employee Team: Core Type: Enhancement Type: Feature Request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@lprimak @arodionov