Tests for CP Membership restart issues

[lprimak]

Demos CP member restart and rejoin issues
Relates to #24897

[devOpsHazelcast]

Can one of the admins verify this patch?

[devOpsHazelcast]

Can one of the admins verify this patch?

[devOpsHazelcast]

Can one of the admins verify this patch?

[devOpsHazelcast]

Can one of the admins verify this patch?

[arodionov]

If a cluster lost the majority of its members it will be blocked and should be recovered manually https://docs.hazelcast.com/hazelcast/5.3/cp-subsystem/management#handling-a-lost-majority

[lprimak]

@arodionov I think it's premature to close this for the following reasons:

• There is no event sent by Hazelcast when members shut down normally and majority is lost, thus there is no concrete way to find out when the "unsafe state" occurs.
• There is no way (even manually) to recover the cluster to a working state unless at east 3 members exist and are functioning.
• "Unrecoverable state" is dubious at best
• 100% CPU usage is seen under certain cicrumstances
• Requiring manual recovery is also dubious.

I would suggest reopening this PR.

[arodionov]

@lprimak thanks for your points!

Regarding,

There is no event sent by Hazelcast when members shut down normally and majority is lost, thus there is no concrete way to find out when the "unsafe state" occurs.

there is a CP Group Availability Listeners https://docs.hazelcast.com/hazelcast/5.3/cp-subsystem/management#cp-group-availability-listeners

Other points, I'll copy to #24912

[lprimak]

Thanks @arodionov All of this is already described in #24897

there is a CP Group Availability Listeners https://docs.hazelcast.com/hazelcast/5.3/cp-subsystem/management#cp-group-availability-listeners

Just want to reiterate that those listeners are not called when members are shut down properly, only when they die / freeze unexpectedly. This is why the above cannot relied upon currently.

If you run https://github.com/flowlogix/hazelcast-issues on 3 terminals, (stop/restart, use Ctrl-z) in about 10 minutes you will see all those issues in action.

[devOpsHazelcast]

PR closed by Hazelcast automation as no activity (>6 months). Please reopen with comments, if necessary. Thank you for using Hazelcast and your valuable contributions

[lprimak]

Please reopen. Still valid