JVM crash on heavy load

[jknair]

using netty master @ c40b0d2 . We are seeing recurring JVM crashes on heavy loads. Similar issue
#1117 . We saw this using both epoll and nio eventloops.

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fa85505224c, pid=40786, tid=140359960991488
#
# JRE version: Java(TM) SE Runtime Environment (8.0_20-b26) (build 1.8.0_20-b26)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.20-b23 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# v  ~StubRoutines::jint_disjoint_arraycopy
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.sun.com/bugreport/crash.jsp
#

---------------  T H R E A D  ---------------

Current thread (0x00007fa78401b000):  JavaThread "nioEventLoopGroup-2-40" daemon [_thread_in_Java, id=40889, stack(0x00007fa8198d9000,0x00007fa8199da000)]

siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x00007fa5a2360c10

Registers:
RAX=0x00007fa5a23bfe14, RBX=0x00000007b7f52f68, RCX=0x000000000002ff75, RDX=0xffffffffffff41c6
RSP=0x00007fa8199d8120, RBP=0x00007fa8199d8120, RSI=0x00000007b9312d40, RDI=0x00007fa5a23bfe08
R8 =0x0000000000000000, R9 =0x0000000000000000, R10=0x00007fa855052a40, R11=0x000000000004022c
R12=0x0000000000000000, R13=0x00000000013bfde4, R14=0x00007fa5a23bfe14, R15=0x00007fa78401b000
RIP=0x00007fa85505224c, EFLAGS=0x0000000000010296, CSGSFS=0x0000000000000033, ERR=0x0000000000000004
  TRAPNO=0x000000000000000e

Top of Stack: (sp=0x00007fa8199d8120)
0x00007fa8199d8120:   0000000000000000 00007fa855e30fde
0x00007fa8199d8130:   0000000700c5bf98 0000000000000000
0x00007fa8199d8140:   00000007b7f52f68 00007fa8013bfdd4
0x00007fa8199d8150:   00000000013bfdd4 0000000000000000
0x00007fa8199d8160:   00000007b2cb03d0 00007fa8552a4e47
0x00007fa8199d8170:   00000000013bfdd4 00007fa8563c4df0
0x00007fa8199d8180:   0000000700c5bf98 00000007b7f52f68
0x00007fa8199d8190:   00000007b2cb03d0 efc7e860efc7e3ea
0x00007fa8199d81a0:   000000077e3f1f50 00000007c01d3818
0x00007fa8199d81b0:   00000007c020e290 013bfdd4f0423c21
0x00007fa8199d81c0:   00000000013bfdd4 000000078211e108
0x00007fa8199d81d0:   00000007c0072028 00000007c000ee10
0x00007fa8199d81e0:   00000007b7f52f68 0000000000000000
0x00007fa8199d81f0:   00007fa8199d8220 000000006a6af87d
0x00007fa8199d8200:   00000007013bfdd4 00007fa869f1b68f
0x00007fa8199d8210:   00000006c65d52c0 00000000354a6480
0x00007fa8199d8220:   00000007b1246378 00007fa855d8ba69
0x00007fa8199d8230:   000000000db4cbc6 00007fa855a26164
0x00007fa8199d8240:   0000000100000001 00000006c3238bc0
0x00007fa8199d8250:   00000007b2cb03d0 00007fa855b7de08
0x00007fa8199d8260:   00000006fe5814d0 00007fa8dfcb029a
0x00007fa8199d8270:   00000006c469abe8 00000000efd73d1f
0x00007fa8199d8280:   00000000efd73d1f 0000000000000000
0x00007fa8199d8290:   00007fa8199d82e0 00007fa8558b6e0b
0x00007fa8199d82a0:   000000077eb9e8f8 00000007b1243b68
0x00007fa8199d82b0:   00000000dfcb029a 00007fa856372db8
0x00007fa8199d82c0:   00000007b2cb0370 00007fa85600dc18
0x00007fa8199d82d0:   0000000000000000 00000007b12464c8
0x00007fa8199d82e0:   00000007b2cb0370 00000006c65d4760
0x00007fa8199d82f0:   0000000100000002 000000077eb9ea38
0x00007fa8199d8300:   000000077eb9ea38 000000077eb9e988
0x00007fa8199d8310:   0000000000000001 00007fa86a083103

Instructions: (pc=0x00007fa85505224c)
0x00007fa85505222c:   f7 c1 01 00 00 00 74 06 8b 47 08 89 46 08 48 33
0x00007fa85505223c:   c0 c9 c3 90 c5 fa 6f 44 d7 c8 c5 fa 7f 44 d6 c8
0x00007fa85505224c:   c5 fa 6f 4c d7 d8 c5 fa 7f 4c d6 d8 c5 fa 6f 54
0x00007fa85505225c:   d7 e8 c5 fa 7f 54 d6 e8 c5 fa 6f 5c d7 f8 c5 fa

Register to memory mapping:

RAX=0x00007fa5a23bfe14 is an unknown value
RBX=0x00000007b7f52f68 is an oop
[B
 - klass: {type array byte}
 - length: 20708820
RCX=0x000000000002ff75 is an unknown value
RDX=0xffffffffffff41c6 is an unknown value
RSP=0x00007fa8199d8120 is pointing into the stack for thread: 0x00007fa78401b000
RBP=0x00007fa8199d8120 is pointing into the stack for thread: 0x00007fa78401b000
RSI=0x00000007b9312d40 is an oop

[error occurred during error reporting (printing register info), id 0xb]

Stack: [0x00007fa8198d9000,0x00007fa8199da000],  sp=0x00007fa8199d8120,  free space=1020k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
v  ~StubRoutines::jint_disjoint_arraycopy
J 3958 C2 io.netty.buffer.PooledUnsafeDirectByteBuf.getBytes(I[BII)Lio/netty/buffer/ByteBuf; (81 bytes) @ 0x00007fa855e30fde [0x00007fa855e30f00+0xde]

System info :

java version "1.8.0_20"
Java(TM) SE Runtime Environment (build 1.8.0_20-b26)
Java HotSpot(TM) 64-Bit Server VM (build 25.20-b23, mixed mode)

Linux testbox # 1 SMP Debian 3.14.15-2 (2014-08-09) x86_64 GNU/Linux

Any workaround how to avoid this ?

[jknair]

one more thing whenever it runs on native epoll the error would be at

# Problematic frame:
# v  ~StubRoutines::jlong_disjoint_arraycopy

[normanmaurer]

@jknair could you try to run with -Dio.netty.noUnsafe=true and let me know if you still can reproduce the crash ?

[normanmaurer]

@jknair ok I may have an idea... stay tuned.

[jknair]

hey ran the app disabling unsafe and used nio event loops and no crashes in the last 12 hours

[jknair]

@normanmaurer keeping unsafe turned off we are hitting FULL GC's !!!

[normanmaurer]

@jknair let me close this and please re-open if you still encounter the problem with latest code