Map (IMap) Corrupting/Losing data when there is competition at task startup

[vinicius-colutti]

Describe the bug

When there is competition at task startup on Java HazelcastInstance, to form the server cluster, the Map are corrupting, as if the information/positions were removed from the map.

I got this situation, in AWS ECS environment, using ECS discovery to create the cluster of Hazelcast instances.

And then, I was able to replicate this situation in my local environment, using the simple Hazelcast instances.

Below, I will explain how to replicate the problem.

Expected behavior

Not corrupt the map, when one or more tasks are initializing in the same moment.

To Reproduce

I disposed a sample application to test this problem: https://github.com/vinicius-colutti/hazelcast-map-trade-off

Lang, framework and versions

Lang: Java17
Framework: spring-boot-starter-web 3.1.10
Hazelcast: 5.4.0

What does this application do?

1. This application create a simple Hazelcast Instance as a bean

2. Have a controller to get full map and populate the map

3. When the application started, I get the map to check size and show the keys-values

To Reproduce the problem

1. Cluster three tasks in a Hazelcast cluster (differentiating only the HTTP and TCP port on which each node will run)

2. Then, I populate the map with 10 objects, calling the Hazelcast port 5701 node (or any other)

3. I check if all members have the complete map by calling the HTTP request /getMap

4. So I stop two cluster nodes
5. Then I start both nodes again, pretty much at the same time
6. We can already see at startup that the map is no longer complete. We can also see that when I make HTTP requests, the map already responds with another size.
No matter which node I make requests to, they will all return the same size and the same objects.

7. If we repeat the process of stopping the nodes and starting them again, practically at the same time, we can observe the map becoming corrupted again, and in the test carried out the map had only two objects.

Additional context

With ReplicatedMap, this problem not occurred.

Regarding Map, I didn't find any information in the documentation saying that this is an expected or common situation.