Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Member OOME, after calling map size sequentially on 4000 maps [HZ-4555] #26282

Open
Danny-Hazelcast opened this issue Mar 7, 2024 · 2 comments

Comments

@Danny-Hazelcast
Copy link
Contributor

High Map Count, High partition count

4000 maps loaded with 1 key value

4000 maps cleared.

on checking each map size it zero,
calling map size from a client on each map sequentially in the end the member OOMES

Member OOME, after calling map size sequentially on 4000 maps

https://jenkins.hazelcast.com/view/stable/job/high-map-count/2/console

/home/ec2-user/workspace/workspace/high-map-count/5.4.0-SNAPSHOT/2024_02_27-15_24_51/high-map-count

output/HZ/HzMember3HZ/HzMember3HZ.hprof

@Danny-Hazelcast
Copy link
Contributor Author

23:59:17 Installing core
23:59:17 installing /home/ec2-user/.m2/repository/com/hazelcast/hazelcast-enterprise/5.4.0-SNAPSHOT/hazelcast-enterprise-5.4.0-SNAPSHOT.jar
23:59:17 installing /home/ec2-user/.m2/repository/com/hazelcast/hazelcast-sql/5.4.0-SNAPSHOT/hazelcast-sql-5.4.0-SNAPSHOT.jar
23:59:17 HzMember3HZ Started 10.0.0.29 2413@ip-10-0-0-29.ec2.internal /home/ec2-user/hz-root/hz-lib/5.4.0-SNAPSHOT 
23:59:17 HzMember2HZ Started 10.0.0.184 2415@ip-10-0-0-184.ec2.internal /home/ec2-user/hz-root/hz-lib/5.4.0-SNAPSHOT 
23:59:17 HzMember1HZ Started 10.0.0.31 2414@ip-10-0-0-31.ec2.internal /home/ec2-user/hz-root/hz-lib/5.4.0-SNAPSHOT 
23:59:17 HzMember4HZ Started 10.0.0.130 2418@ip-10-0-0-130.ec2.internal /home/ec2-user/hz-root/hz-lib/5.4.0-SNAPSHOT 
23:59:17 HzClient3HZ Started 10.0.0.29 2710@ip-10-0-0-29.ec2.internal /home/ec2-user/hz-root/hz-lib/5.4.0-SNAPSHOT 
23:59:17 HzClient1HZ Started 10.0.0.31 2706@ip-10-0-0-31.ec2.internal /home/ec2-user/hz-root/hz-lib/5.4.0-SNAPSHOT 
23:59:17 HzClient2HZ Started 10.0.0.184 2709@ip-10-0-0-184.ec2.internal /home/ec2-user/hz-root/hz-lib/5.4.0-SNAPSHOT 
23:59:17 HzClient4HZ Started 10.0.0.130 2717@ip-10-0-0-130.ec2.internal /home/ec2-user/hz-root/hz-lib/5.4.0-SNAPSHOT 
23:59:17 info hzcmd.info.Memory driver=Member threads=1 duration=0 bench=Roll interval=0
23:59:17 info Done 0 sec
23:59:17 load_mapBak1HD hzcmd.map.member.Load driver=Member threads=1 duration=0 bench=Roll interval=0 mapCount=4000 keyDomain=1 name=mapBak1HD
23:59:17 load Done 2 sec
23:59:17 clear_mapBak1HD hzcmd.distributed.Clear driver=Client1 threads=1 duration=0 bench=Roll interval=0
23:59:17 clr Done 8 sec
23:59:17 size0_mapBak1HD hzcmd.distributed.Size driver=Client1 threads=1 duration=0 bench=Roll interval=0 size=0
23:59:17 fail HzClient1HZ size0_mapBak1HD hzcmd.distributed.Size threadId=0 com.hazelcast.core.OperationTimeoutException: ClientInvocation{clientMessage = ClientMessage{connection=null, length=41, operation=Map.Size, isRetryable=true, correlationId=4894, messageType=12a00, isEvent=false, isfragmented=false}, objectName = mapBak1HD1847, target = random, sentConnection = null} timed out because exception occurred after client invocation timeout 120000 ms. Current time: 2024-02-27 15:59:17.032. Start time: 2024-02-27 15:28:38.574. Total elapsed time: 1838459 ms.  
23:59:43 output/HZ/HzClient1HZ/exception.txt
23:59:43 output/HZ/HzMember3HZ/HzMember3HZ.hprof
23:59:43 output/HZ/HzMember3HZ/HzMember3HZ.oome

@github-actions github-actions bot changed the title Member OOME, after calling map size sequentially on 4000 maps Member OOME, after calling map size sequentially on 4000 maps [HZ-4555] Mar 7, 2024
Copy link
Contributor

github-actions bot commented Mar 7, 2024

Internal Jira issue: HZ-4555

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant