Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HeapHogLoadTest_5m_0_FAILED Segmentation error vmState=0x00000000 #19456

Open
JasonFengJ9 opened this issue May 6, 2024 · 17 comments
Open

HeapHogLoadTest_5m_0_FAILED Segmentation error vmState=0x00000000 #19456

JasonFengJ9 opened this issue May 6, 2024 · 17 comments
Labels
segfault Issues that describe segfaults / JVM crashes test failure

Comments

@JasonFengJ9
Copy link
Member

JasonFengJ9 commented May 6, 2024 •
edited

Failure link

From an internal build(macaarch64rt8):

openjdk version "22.0.1-beta" 2024-04-16
IBM Semeru Runtime Open Edition 22.0.1+8-202405041602 (build 22.0.1-beta+8-202405041602)
Eclipse OpenJ9 VM 22.0.1+8-202405041602 (build master-2c34fcf0fb, JRE 22 Mac OS X aarch64-64-Bit 20240504_59 (JIT enabled, AOT enabled)
OpenJ9   - 2c34fcf0fb
OMR      - acd6f7e0b
JCL      - 7b1ea2cb9 based on jdk-22.0.1+8)

Rerun in Grinder - Change TARGET to run only the failed test targets.

Optional info

Failure output (captured from console output)

[2024-05-05T04:45:48.550Z] variation: Mode110
[2024-05-05T04:45:48.550Z] JVM_OPTIONS:  -Xjit -Xgcpolicy:gencon -Xnocompressedrefs 

[2024-05-05T04:48:11.849Z] HHLT 00:48:10.589 - Completed 46.7%. Number of tests started=341621 (+47856)
[2024-05-05T04:48:23.425Z] HHLT stderr Unhandled exception
[2024-05-05T04:48:23.425Z] HHLT stderr Type=Segmentation error vmState=0x00000000
[2024-05-05T04:48:23.425Z] HHLT stderr J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000002
[2024-05-05T04:48:23.425Z] HHLT stderr Handler1=0000000100A9DC98 Handler2=000000010094CFE0 InaccessibleAddress=0000000099669956
[2024-05-05T04:48:23.425Z] HHLT stderr x0=000000013C964500 x1=00000001705C27C8 x2=0000000150133C00 x3=0000000150133C00
[2024-05-05T04:48:23.425Z] HHLT stderr x4=00000001705C27D8 x5=00000001705C27C0 x6=00000001705C27B8 x7=000000036ACE7C60
[2024-05-05T04:48:23.425Z] HHLT stderr x8=000000014169B100 x9=000000013C964500 x10=0000000099669966 x11=00000001705C27C8
[2024-05-05T04:48:23.425Z] HHLT stderr x12=00000001705C27D8 x13=00000001705C27B0 x14=0000000100AE98FC x15=0000000000000010
[2024-05-05T04:48:23.425Z] HHLT stderr x16=00000001705C2820 x17=00000001EA494DB0 x18=00000001705C2301 x19=0000000040000000
[2024-05-05T04:48:23.425Z] HHLT stderr x20=0000000099669966 x21=00000001705C27B0 x22=0000000110D3ED1C x23=00000001705C27C0
[2024-05-05T04:48:23.425Z] HHLT stderr x24=00000001705C27B0 x25=00000001705C27C8 x26=00000001705C27C0 x27=00000001705C27B8
[2024-05-05T04:48:23.425Z] HHLT stderr x28=0000000100BC6D84 x29(FP)=00000001705C27A0 x30(LR)=0000000100AE937C x31(SP)=00000001705C22C0
[2024-05-05T04:48:23.425Z] HHLT stderr PC=0000000100AFE318 SP=00000001705C22C0
[2024-05-05T04:48:23.425Z] HHLT stderr v0 00000001340a0309 (f: 873071360.000000, d: 2.553350e-314)
[2024-05-05T04:48:23.425Z] HHLT stderr v1 000000013680bfb8 (f: 914407360.000000, d: 2.573773e-314)
[2024-05-05T04:48:23.425Z] HHLT stderr v2 0000000040000000 (f: 1073741824.000000, d: 5.304989e-315)
[2024-05-05T04:48:23.425Z] HHLT stderr v3 3f51d8e2cbc49e05 (f: 3418660352.000000, d: 1.089307e-03)
[2024-05-05T04:48:23.425Z] HHLT stderr v4 bf9411c4195bf130 (f: 425455936.000000, d: -1.959902e-02)
[2024-05-05T04:48:23.425Z] HHLT stderr v5 3f51d8e2cbc49e1c (f: 3418660352.000000, d: 1.089307e-03)
[2024-05-05T04:48:23.425Z] HHLT stderr v6 4018000000000000 (f: 0.000000, d: 6.000000e+00)
[2024-05-05T04:48:23.425Z] HHLT stderr v7 000e1781000e1781 (f: 923521.000000, d: 1.959708e-308)
[2024-05-05T04:48:23.425Z] HHLT stderr v8 000003c10000745f (f: 29791.000000, d: 2.039238e-311)
[2024-05-05T04:48:23.425Z] HHLT stderr v9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-05-05T04:48:23.425Z] HHLT stderr v10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-05-05T04:48:23.425Z] HHLT stderr v11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-05-05T04:48:23.425Z] HHLT stderr v12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-05-05T04:48:23.425Z] HHLT stderr v13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-05-05T04:48:23.425Z] HHLT stderr v14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-05-05T04:48:23.425Z] HHLT stderr v15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-05-05T04:48:23.425Z] HHLT stderr v16 bfd0000000000000 (f: 0.000000, d: -2.500000e-01)
[2024-05-05T04:48:23.425Z] HHLT stderr v17 3fd54f257ba2fd55 (f: 2074279296.000000, d: 3.329557e-01)
[2024-05-05T04:48:23.425Z] HHLT stderr v18 3f58ba9f20072c0c (f: 537340928.000000, d: 1.509338e-03)
[2024-05-05T04:48:23.425Z] HHLT stderr v19 3fe62e42fefa39ef (f: 4277811712.000000, d: 6.931472e-01)
[2024-05-05T04:48:23.425Z] HHLT stderr v20 c1150fb645a1cac1 (f: 1168231168.000000, d: -3.450696e+05)
[2024-05-05T04:48:23.425Z] HHLT stderr v21 c1150fb600000000 (f: 0.000000, d: -3.450695e+05)
[2024-05-05T04:48:23.425Z] HHLT stderr v22 bc90000000000000 (f: 0.000000, d: -5.551115e-17)
[2024-05-05T04:48:23.425Z] HHLT stderr v23 bca8000000000000 (f: 0.000000, d: -1.665335e-16)
[2024-05-05T04:48:23.425Z] HHLT stderr v24 b948000000000000 (f: 0.000000, d: -9.244464e-33)
[2024-05-05T04:48:23.425Z] HHLT stderr v25 bc90000000000000 (f: 0.000000, d: -5.551115e-17)
[2024-05-05T04:48:23.425Z] HHLT stderr v26 bfb16872b03ff0cf (f: 2956980480.000000, d: -6.800000e-02)
[2024-05-05T04:48:23.425Z] HHLT stderr v27 401c125cd987c2bb (f: 3649553152.000000, d: 7.017932e+00)
[2024-05-05T04:48:23.425Z] HHLT stderr v28 bff0000000000000 (f: 0.000000, d: -1.000000e+00)
[2024-05-05T04:48:23.425Z] HHLT stderr v29 bff0000000000000 (f: 0.000000, d: -1.000000e+00)
[2024-05-05T04:48:23.425Z] HHLT stderr v30 3ff5555555555555 (f: 1431655808.000000, d: 1.333333e+00)
[2024-05-05T04:48:23.425Z] HHLT stderr v31 4000000000000000 (f: 0.000000, d: 2.000000e+00)
[2024-05-05T04:48:23.425Z] HHLT stderr Module=/Users/jenkins/workspace/Test_openjdk22_j9_extended.system_aarch64_mac_testList_0/jdkbinary/j2sdk-image/Contents/Home/lib/default/libj9vm29.dylib
[2024-05-05T04:48:23.425Z] HHLT stderr Module_base_address=0000000100A78000 Symbol=_ZN26VM_BytecodeInterpreterFull3runEP10J9VMThread
[2024-05-05T04:48:23.425Z] HHLT stderr Symbol_address=0000000100AE9388
[2024-05-05T04:48:23.425Z] HHLT stderr Target=2_90_20240504_59 (Mac OS X 13.0)
[2024-05-05T04:48:23.425Z] HHLT stderr CPU=aarch64 (8 logical CPUs) (0x400000000 RAM)
[2024-05-05T04:48:23.425Z] HHLT stderr ----------- Stack Backtrace -----------
[2024-05-05T04:48:23.425Z] HHLT stderr ---------------------------------------
[2024-05-05T04:48:23.425Z] HHLT stderr JVMDUMP039I Processing dump event "gpf", detail "" at 2024/05/05 00:48:21 - please wait.

[2024-05-05T04:48:47.158Z] HeapHogLoadTest_5m_0_FAILED

50x internal grinder - 2/50 failed at macaarch64rt5

There was a similar failure:
@JasonFengJ9 JasonFengJ9 added test failure segfault Issues that describe segfaults / JVM crashes labels May 6, 2024
@pshipton
Copy link
Member

pshipton commented May 7, 2024

@knn-k can you pls take the first look at the core.

@pshipton
Copy link
Member

pshipton commented May 7, 2024

@babsingh fyi in case it's related to #14713

@knn-k
Copy link
Contributor

knn-k commented May 8, 2024

It is a SEGV caused by an instruction trying to load from an address pointed by x10, which holds 0x99669966 eyecatcher value.

   8630c: 08 17 40 f9  	ldr	x8, [x24, #40]
   86310: 14 01 40 f9  	ldr	x20, [x8]
   86314: ea 03 14 aa  	mov	x10, x20
   86318: 4b 0d 9f b8  	ldrsw	x11, [x10, #-16]! <- SEGV here, x10=x20=0x99669966
   8631c: 4a 01 0b 8b  	add	x10, x10, x11
   86320: 0b 03 40 f9  	ldr	x11, [x24]
   86324: 6b 15 4c f9  	ldr	x11, [x11, #6184]
   86328: 4c 01 40 79  	ldrh	w12, [x10]
   8632c: 8d 69 6a 38  	ldrb	w13, [x12, x10]
   86330: bf 6d 01 71  	cmp	w13, #91
   86334: a0 01 00 54  	b.eq	0x86368 <__ZN26VM_BytecodeInterpreterFull3runEP10J9VMThread+0x14fe0>

@knn-k knn-k mentioned this issue May 8, 2024
Merged
@knn-k
Copy link
Contributor

knn-k commented May 8, 2024

x20 is loaded from [x8], and jdmpview shows x8 is a pointer to a J9Class MethodTypeForm as shown below.

> !j9class 0x14169b100
J9Class at 0x14169b100 {
  Fields for J9Class:
	0x0: UDATA eyecatcher = 0x0000000099669966 (2573637990)
	0x8: struct J9ROMClass* romClass = !j9romclass 0x0000000121429CA8
	0x10: void** superclasses = !j9x 0x000000014169AFE0
	0x18: UDATA classDepthAndFlags = 0x00000000020E0001 (34471937)
	0x20: U32 classDepthWithFlags = 0x00000000 (0)
	0x24: U32 classFlags = 0x00000000 (0)
	0x28: struct J9ClassLoader* classLoader = !j9classloader 0x000000013C088668
	0x30: struct J9Object* classObject = !j9object 0x00000002800087A0 // java/lang/Class
	0x38: volatile UDATA initializeStatus = 0x0000000000000001 (1)
	0x40: struct J9Method* ramMethods = !j9method 0x000000014169B318 // java/lang/invoke/MethodTypeForm.erasedType()Ljava/lang/invoke/MethodType;
	0x48: UDATA* ramStatics = !j9x 0x000000014169B518
	0x50: struct J9Class* arrayClass = !j9class 0x0000000000000000
	0x58: UDATA totalInstanceSize = 0x0000000000000030 (48)
	0x60: struct J9ITable* lastITable = !j9itable 0x0000000100BC7878
	0x68: UDATA* instanceDescription = !j9x 0x000000000000003D
	0x70: UDATA* instanceLeafDescription = !j9x 0x0000000000000001
	0x78: UDATA instanceHotFieldDescription = 0x0000000000000004 (4)
	0x80: UDATA selfReferencingField1 = 0x0000000000000000 (0)
	0x88: UDATA selfReferencingField2 = 0x0000000000000000 (0)
	0x90: struct J9Method* initializerCache = !j9method 0x0000000000000000
	0x98: UDATA romableAotITable = 0x000000010209C000 (4329160704)
	0xa0: UDATA packageID = 0x0000000121391F01 (4852358913)
	0xa8: struct J9Module* module = !j9module 0x000000013C089868
	0xb0: struct J9Class* subclassTraversalLink = !j9class 0x000000014169AA00 // java/lang/invoke/LambdaForm
	0xb8: struct J9Class* subclassTraversalReverseLink = !j9class 0x000000014169C100 // jdk/internal/vm/Continuation
	0xc0: void** iTable = !j9x 0x0000000000000000
	0xc8: UDATA castClassCache = 0x00000001416BC701 (5392549633)
	0xd0: void** jniIDs = !j9x 0x0000000000000000
	0xd8: UDATA lockOffset = 0x0000000000000008 (8)
	0xe0: U32 paddingForGLRCounters = 0x00000000 (0)
	0xe4: U16 reservedCounter = 0x0000 (0)
	0xe6: U16 cancelCounter = 0x0000 (0)
	0xe8: UDATA newInstanceCount = 0x00000000000003E8 (1000)
	0xf0: IDATA backfillOffset = 0x0000000000000038 (56)
	0xf8: struct J9Class* replacedClass = !j9class 0x0000000000000000
	0x100: UDATA finalizeLinkOffset = 0x0000000000000000 (0)
	0x108: struct J9Class* nextClassInSegment = !j9class 0x000000014169AA00 // java/lang/invoke/LambdaForm
	0x110: UDATA* ramConstantPool = !j9x 0x000000014169ACA0
	0x118: struct J9Object** callSites = !j9x 0x0000000000000000
	0x120: struct J9Object** invokeCache = !j9x 0x0000000000000000
	0x128: struct J9Object** varHandleMethodTypes = !j9x 0x0000000000000000
	0x130: struct J9VMCustomSpinOptions* customSpinOption = !j9vmcustomspinoptions 0x0000000000000000
	0x138: void** staticSplitMethodTable = !j9x 0x0000000000000000
	0x140: void** specialSplitMethodTable = !j9x 0x0000000000000000
	0x148: struct J9JITExceptionTable* jitMetaDataList = !j9jitexceptiontable 0x0000000000000000
	0x150: struct J9Class* gcLink = !j9class 0x0000000000000000
	0x158: struct J9Class* hostClass = !j9class 0x000000014169B100 // java/lang/invoke/MethodTypeForm
	0x160: struct J9Class* nestHost = !j9class 0x0000000000000000
	0x168: struct J9FlattenedClassCache* flattenedClassCache = !j9flattenedclasscache 0x0000000000000000
	0x170: struct J9ClassHotFieldsInfo* hotFieldsInfo = !j9classhotfieldsinfo 0x0000000000000000
	0x178: struct J9MemberNameListNode* memberNames = !j9membernamelistnode 0x0000000000000000
}
Class name: java/lang/invoke/MethodTypeForm
To view static fields, use !j9statics 0x000000014169B100
To view instance shape, use !j9classshape 0x000000014169B100

@knn-k
Copy link
Contributor

knn-k commented May 9, 2024

Many registers in the register dump have the addresses of a specific region.
0x1705C27A0-0x1705C27D8 in x1, x4, x5, x6, x11, x12, x13, x21, x23, x24, x25, x26, x27, x29

The memory dump looks like a table of addresses.

  • The x8 value for MethodTypeForm above was loaded from 0x1705c27d8.
  • 0x13c019e20 at 0x1705c27b0 is j9javavm.
  • 0x110d3ed1c at 0x1705c27c8 is a jitReturnAddress in crashInfo. It is in the address range of JITed invokeVirtual().
> hexdump 0x1705C27A0 80

1705c27a0: 30285c70 01000000 7c93ae00 01000000  |0(\p....|.......|
1705c27b0: 209e013c 01000000 0045963c 01000000  | ..<.....E.<....|
1705c27c0: a0000a34 01000000 1cedd310 01000000  |...4............|
1705c27d0: 00000000 00000000 00b16941 01000000  |..........iA....|
1705c27e0: 01000000 00000000 10000000 00000000  |................|

> !whatis 0x013c019e20
Found 0x000000013C019E20 as !j9javavm 0x13c019e20
Match found
> !whatis 0x0110d3ed1c
Found 0x0000000110D3ED1C as !void 0x110d3ed1c: !j9javavm 0x13c019e20->j9ras->crashInfo->failingThread->linkNext->jitReturnAddress
Match found

> info jitm
...
        start=0x110d3ec68  end=0x110d3edd0   java/lang/invoke/LambdaForm$DMH/0x000000003c99c820::invokeVirtual(Ljava/lang/Object;Ljava/lang/Object;I)F
...

@babsingh
Copy link
Contributor

babsingh commented May 9, 2024 •
edited

@knn-k DDR cmds to find the location of the crash in the interpreter:

> !gpinfo
Failing Thread: !j9vmthread 0x13c95ef00
Failing Thread ID: 0x14797b8a (343505802)
....

> !stackslots 0x13c95ef00
<13c95ef00> *** BEGIN STACK WALK, flags = 00400001 walkThread = 5,311,426,304 ***
<13c95ef00> 	ITERATE_O_SLOTS
<13c95ef00> 	RECORD_BYTECODE_PC_OFFSET
<13c95ef00> Initial values: walkSP = 0x0000000134810CB0, PC = 0x000000013622C451, literals = 0x0000000136819588, A0 = 0x0000000134810D88, j2iFrame = 0x0000000134810D78, decomp = 0x0000000000000000
<13c95ef00> J2I frame: bp = 0x0000000134810D78, sp = 0x0000000134810CB0, pc = 0x000000013622C451, cp = 0x00000001368194C0, arg0EA = 0x0000000134810D88, flags = 0x0000000010000000
<13c95ef00> 	Method: java/lang/invoke/LambdaForm$NFI/0x000000003622c320.invoke_LI_F(Ljava/lang/invoke/MethodHandle;[Ljava/lang/Object;)Ljava/lang/Object; !j9method 0x0000000136819588
<13c95ef00> 	Bytecode index = 13 <--- Bytecode being executed
<13c95ef00> 	Using local mapper
<13c95ef00> 	Locals starting at 0x0000000134810D88 for 0x0000000000000002 slots
<13c95ef00> 		I-Slot: a0[0x0000000134810D88] = 0x00000002805D0080
<13c95ef00> 		I-Slot: a1[0x0000000134810D80] = 0x000000036C3169F8
May 08, 2024 6:23:59 PM com.ibm.j9ddr.vm29.events.DefaultEventListener corruptData
WARNING: CorruptData encountered iterating o-slots. walkThread = 0x000000013C95EF00
com.ibm.j9ddr.CorruptDataException: Operand stack underflow in StackMap
	at com.ibm.j9ddr.vm29.j9.stackmap.StackMap$J9MappingStack.POP(StackMap.java:184)
	at com.ibm.j9ddr.vm29.j9.stackmap.StackMap$StackMap_V1.outputStackMap(StackMap.java:728)
	at com.ibm.j9ddr.vm29.j9.stackmap.StackMap$StackMap_V1.j9stackmap_StackBitsForPC(StackMap.java:271)
	at com.ibm.j9ddr.vm29.j9.stackmap.StackMap.j9stackmap_StackBitsForPC(StackMap.java:106)
	....

<13c95ef00> JIT frame: bp = 0x0000000134810DC8, pc = 0x0000000110521538, unwindSP = 0x0000000134810D80, cp = 0x000000013C138AC0, arg0EA = 0x0000000134810DE0, jitInfo = 0x0000000141A9A9E8
<13c95ef00> 	Method: java/lang/invoke/LambdaForm$DMH/0x000000003c09ac20.invokeStatic(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object; !j9method 0x000000013C138BA8
<13c95ef00> 	Bytecode index = 11, inlineDepth = 0, PC offset = 0x0000000000000110
<13c95ef00> 	stackMap=0x0000000141A9AB04, slots=I16(0x0003) parmBaseOffset=I16(0x0008), parmSlots=U16(0x0003), localBaseOffset=I16(0xFFD8)
<13c95ef00> 	Described JIT args starting at 0x0000000134810DD0 for U16(0x0003) slots
<13c95ef00> 		O-Slot: : a2[0x0000000134810DD0] = 0x000000036C3169F8
<13c95ef00> 		O-Slot: : a1[0x0000000134810DD8] = 0x00000002805D0080
<13c95ef00> 		O-Slot: : a0[0x0000000134810DE0] = 0x0000000280BEAD58
<13c95ef00> 	Described JIT temps starting at 0x0000000134810DA0 for IDATA(0x0000000000000005) slots
<13c95ef00> 		O-Slot: : t4[0x0000000134810DA0] = 0x0000000000000000
...

<13c95ef00> JIT inline frame: bp = 0x0000000134810E68, pc = 0x00000001109CA128, unwindSP = 0x0000000134810DD0, cp = 0x00000001501302C0, arg0EA = 0x0000000000000000, jitInfo = 0x000000016007D7E8
<13c95ef00> 	Method: java/lang/invoke/LambdaForm$NamedFunction.invokeWithArguments([Ljava/lang/Object;)Ljava/lang/Object; !j9method 0x0000000150130A10
<13c95ef00> 	Bytecode index = 21, inlineDepth = 1, PC offset = 0x00000001109C9DA7
<13c95ef00> JIT frame: bp = 0x0000000134810E68, pc = 0x00000001109CA128, unwindSP = 0x0000000134810DD0, cp = 0x0000000141698B00, arg0EA = 0x0000000134810E80, jitInfo = 0x000000016007D7E8
<13c95ef00> 	Method: java/lang/invoke/LambdaForm.interpretName(Ljava/lang/invoke/LambdaForm$Name;[Ljava/lang/Object;)Ljava/lang/Object; !j9method 0x000000014169A3F8
<13c95ef00> 	Bytecode index = 123, inlineDepth = 0, PC offset = 0x0000000000000338
<13c95ef00> 	stackMap=0x000000016007DAE4, slots=I16(0x0003) parmBaseOffset=I16(0x0008), parmSlots=U16(0x0003), localBaseOffset=I16(0xFFC8)
<13c95ef00> 	Described JIT args starting at 0x0000000134810E70 for U16(0x0003) slots
<13c95ef00> 		O-Slot: : a2[0x0000000134810E70] = 0x000000036C316998
<13c95ef00> 		O-Slot: : a1[0x0000000134810E78] = 0x0000000280BECFD0
<13c95ef00> 		O-Slot: : a0[0x0000000134810E80] = 0x0000000280BECEC8
<13c95ef00> 	Described JIT temps starting at 0x0000000134810E30 for IDATA(0x0000000000000007) slots
<13c95ef00> 		I-Slot: : t6[0x0000000134810E30] = 0x0000000000000001
...

> !j9method 0x0000000136819588
J9Method at 0x136819588 {
  Fields for J9Method:
	0x0: U8* bytecodes = !j9x 0x000000013622C444 // "*+2+2�"
	0x8: struct J9ConstantPool* constantPool = !j9constantpool 0x00000001368194C0 (flags = 0x0)
	0x10: void* methodRunAddress = !j9x 0x0000000000000006
	0x18: volatile void* extra = !j9x 0x0000000000000F93
}
Signature: java/lang/invoke/LambdaForm$NFI/0x000000003622c320.invoke_LI_F(Ljava/lang/invoke/MethodHandle;[Ljava/lang/Object;)Ljava/lang/Object; !bytecodes 0x0000000136819588
ROM Method: !j9rommethod 0x000000013622C430
Next Method: !j9method 0x00000001368195A8

> !bytecodes 0x0000000136819588
  Name: invoke_LI_F
  Signature: (Ljava/lang/invoke/MethodHandle;[Ljava/lang/Object;)Ljava/lang/Object;
  Access Flags (20080008): default static
  Internal Attribute Flags: @FrameIteratorSkip
  ...

    0 aload0
    ...
    7 checkcast 3 java/lang/Integer
   10 invokevirtual 4 java/lang/Integer.intValue()I
   13 invokevirtual 6 java/lang/invoke/MethodHandle.invokeBasic(Ljava/lang/Object;I)F <--- Bytecode being run
   16 invokestatic 8 java/lang/Float.valueOf(F)Ljava/lang/Float;
   19 return1

> !j9vmthread 0x13c95ef00
J9VMThread at 0x13c95ef00 {
  Fields for J9VMThread:
        ...
	0x20: UDATA* sp = !j9x 0x0000000134810CB0
	0x28: U8* pc = !j9x 0x000000013622C451 // "�"
	0x30: struct J9Method* literals = !j9method 0x0000000136819588 // java/lang/invoke/LambdaForm$NFI/0x000000003622c320.invoke_LI_F(Ljava/lang/invoke/MethodHandle;[Ljava/lang/Object;)Ljava/lang/Object;
	0x38: UDATA jitStackFrameFlags = 0x0000000000000000 (0)
	0x40: struct J9Object* jitException = !j9object 0x000000036BFDA050 // java/lang/ArrayIndexOutOfBoundsException
	0x48: struct J9Object* currentException = !j9object 0x0000000000000000
	0x50: UDATA* stackOverflowMark = !j9x 0xFFFFFFFFFFFFFFFF
	0x58: UDATA* stackOverflowMark2 = !j9x 0x000000013480CC28
        ...
	0xe8: UDATA tempSlot = 0xFFFFFFFFFFFFFFA8 (-88)
	0xf0: void* jitReturnAddress = !j9x 0x0000000110988410
	0xf8: void* floatTemp1 = !j9x 0x0000000000000000
	0x100: void* floatTemp2 = !j9x 0x00000000000CFB1C
	0x108: void* floatTemp3 = !j9x 0x0000000110C01C68
	0x110: void* floatTemp4 = !j9x 0x0000000000000000
	0x118: UDATA returnValue = 0x000000000000000D (13)
	0x120: UDATA returnValue2 = 0x0000000280618598 (10743809432)
        ...

@babsingh
Copy link
Contributor

babsingh commented May 9, 2024 •
edited

in case it's related to #14713

I don't see the symptoms from #14713 (comment). J9VMThread->literals is good in this crash. There are similarities in terms of tempSlot = -88 and invokebasic being run.

But, this crash looks like the reverse of #14713 (comment).

  • In this case, there is a J2I frame at the top, and we are running the invokebasic INL.
  • While running the invokebasic INL, J9VMThread->jitStackFrameFlags is 0. The fromJIT code-path won't be taken in invokebasic.
  • Due to the J2I frame, JIT has setup the arguments for invokebasic and the fromJIT code-path needs to be taken in invokebasic. Taking the interpreter path has probably led to the crash.
  • Also, DDR indicates corrupt data while processing the J2I frame. So, there is probably a bad O-slot in the J2I frame.

fyi @nbhuiyan @jdmpapin @0xdaryl @gacholio

@gacholio
Copy link
Contributor

If there is a J2I frame, then the jitStackFrameFlags have already been consumed. Is the invokebasic coming from the interpreted method that's running in the J2I frame?

@babsingh
Copy link
Contributor

babsingh commented May 10, 2024 •
edited

Is the invokebasic coming from the interpreted method that's running in the J2I frame?

Yes, bytecode 13 in the J2I frame is an invokebasic.

@gacholio
Copy link
Contributor

That contradicts your statement:

Due to the J2I frame, JIT has setup the arguments for [invokebasic](https://github.com/eclipse-openj9/openj9/blob/master/runtime/vm/BytecodeInterpreter.hpp#L9246) and the fromJIT code-path needs to be taken in [invokebasic](https://github.com/eclipse-openj9/openj9/blob/master/runtime/vm/BytecodeInterpreter.hpp#L9246). Taking the interpreter path has probably led to the crash.

as we are running in an interpreted method that was invoked by the JIT.

@nbhuiyan
Copy link
Member

@babsingh

Due to the J2I frame, JIT has setup the arguments for invokebasic

The JIT would not have set up the arguments for invokebasic. The tempSlot value that would have been set by the compiled code for invokebasic calls would be indicating the number of stack slots occupied by the args, and therefore would be a positive number.

@babsingh
Copy link
Contributor

babsingh commented May 10, 2024 •
edited

I see. So, we are taking the !fromJIT code-path in invokeBasic, which is correct since an interpreted method is being run.

For the interpreted method (invoke_LI_F) in the J2I frame, are the arguments on the stack being populated from the JIT?

<13c95ef00> J2I frame: bp = 0x0000000134810D78, sp = 0x0000000134810CB0, pc = 0x000000013622C451, cp = 0x00000001368194C0, arg0EA = 0x0000000134810D88, flags = 0x0000000010000000
<13c95ef00> 	Method: java/lang/invoke/LambdaForm$NFI/0x000000003622c320.invoke_LI_F(Ljava/lang/invoke/MethodHandle;[Ljava/lang/Object;)Ljava/lang/Object; !j9method 0x0000000136819588
<13c95ef00> 	Bytecode index = 13
<13c95ef00> 	Using local mapper
<13c95ef00> 	Locals starting at 0x0000000134810D88 for 0x0000000000000002 slots
<13c95ef00> 		I-Slot: a0[0x0000000134810D88] = 0x00000002805D0080
<13c95ef00> 		I-Slot: a1[0x0000000134810D80] = 0x000000036C3169F8

@nbhuiyan
Copy link
Member

are the arguments on the stack being populated from the JIT?

Given that the previous frame is a jitted method, yes.

@babsingh
Copy link
Contributor

babsingh commented May 10, 2024 •
edited

The crash happened on an amac: https://na.artifactory.swg-devops.com/artifactory/sys-rt-generic-local/hyc-runtimes-jenkins.swg-devops.com/Test_openjdk22_j9_extended.system_aarch64_mac_testList_0/10/system_test_output.tar.gz. I am unable to view the native stack in lldb. Can anyone see the line of code in lldb where the crash happened?

@knn-k
Copy link
Contributor

knn-k commented May 15, 2024

OpenJ9 for AArch64 macOS does not support native stack backtrace: Issue #15925.
It is not supported on x86 macOS, either.

@JasonFengJ9
Copy link
Member Author

JasonFengJ9 commented May 21, 2024 •
edited

JDK17 aarch64_mac(macaarch64rt1)

[2024-05-18T13:26:15.887Z] variation: Mode110
[2024-05-18T13:26:15.887Z] JVM_OPTIONS:  -Xjit -Xgcpolicy:gencon -Xnocompressedrefs 

[2024-05-18T13:26:18.068Z] java version "17.0.11" 2024-04-16
[2024-05-18T13:26:18.068Z] IBM Semeru Runtime Certified Edition 17.0.11.0-rc3 (build 17.0.11+9)
[2024-05-18T13:26:18.068Z] Eclipse OpenJ9 VM 17.0.11.0-rc3 (build v0.44.0-release-b0699311c7, JRE 17 Mac OS X aarch64-64-Bit 20240416_560 (JIT enabled, AOT enabled)
[2024-05-18T13:26:18.068Z] OpenJ9   - b0699311c7
[2024-05-18T13:26:18.068Z] OMR      - 254af5a04
[2024-05-18T13:26:18.068Z] JCL      - b522ac89460 based on jdk-17.0.11+9)

[2024-05-18T13:27:22.988Z] HHLT 09:27:18.694 - Completed 20.0%. Number of tests started=61161 (+13542)
[2024-05-18T13:27:37.922Z] HHLT stderr Unhandled exception
[2024-05-18T13:27:37.922Z] HHLT stderr Type=Segmentation error vmState=0x00000000
[2024-05-18T13:27:37.922Z] HHLT stderr J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000002
[2024-05-18T13:27:37.922Z] HHLT stderr Handler1=000000010113A82C Handler2=0000000100FF4DA4 InaccessibleAddress=0000000000000018
[2024-05-18T13:27:37.922Z] HHLT stderr x0=00000001501BEF00 x1=00000001780467C8 x2=0000000379028408 x3=00000001412B9200
[2024-05-18T13:27:37.922Z] HHLT stderr x4=00000001780467D8 x5=00000001780467C0 x6=00000001780467B8 x7=0000000178046820
[2024-05-18T13:27:37.922Z] HHLT stderr x8=0000000000000000 x9=0000000000000018 x10=00000001780467C0 x11=00000001780467C8
[2024-05-18T13:27:37.922Z] HHLT stderr x12=00000001780467D8 x13=00000001780467B0 x14=0000000101184030 x15=0000000000000010
[2024-05-18T13:27:37.922Z] HHLT stderr x16=0000000101183A9C x17=0000000000000000 x18=0000000178046D00 x19=0000000040000000
[2024-05-18T13:27:37.922Z] HHLT stderr x20=0000000140C68620 x21=00000001780467B0 x22=00000001103B07CC x23=00000001780467C0
[2024-05-18T13:27:37.922Z] HHLT stderr x24=00000001780467D8 x25=00000001780467C8 x26=00000001780467C0 x27=00000001780467B8
[2024-05-18T13:27:37.922Z] HHLT stderr x28=000000010125ED84 x29(FP)=00000001780467A0 x30(LR)=0000000101183A90 x31(SP)=0000000178046500
[2024-05-18T13:27:37.922Z] HHLT stderr PC=00000001011985A0 SP=0000000178046500
[2024-05-18T13:27:37.922Z] HHLT stderr v0 0000000135010ef9 (f: 889261824.000000, d: 2.561350e-314)
[2024-05-18T13:27:37.922Z] HHLT stderr v1 000000014803bb98 (f: 1208204160.000000, d: 2.718928e-314)
[2024-05-18T13:27:37.922Z] HHLT stderr v2 0000000040000000 (f: 1073741824.000000, d: 5.304989e-315)
[2024-05-18T13:27:37.922Z] HHLT stderr v3 c0e0000000000000 (f: 0.000000, d: -3.276800e+04)
[2024-05-18T13:27:37.922Z] HHLT stderr v4 bc6ce0f170000000 (f: 1879048192.000000, d: -1.252413e-17)
[2024-05-18T13:27:37.922Z] HHLT stderr v5 3df73cd68165e929 (f: 2170939648.000000, d: 3.381523e-10)
[2024-05-18T13:27:37.922Z] HHLT stderr v6 bd803323f5b53b23 (f: 4122295040.000000, d: -1.841700e-12)
[2024-05-18T13:27:37.922Z] HHLT stderr v7 000e1781000e1781 (f: 923521.000000, d: 1.959708e-308)
[2024-05-18T13:27:37.922Z] HHLT stderr v8 000000003f6b86d4 (f: 1064011456.000000, d: 5.256915e-315)
[2024-05-18T13:27:37.922Z] HHLT stderr v9 0000000041000000 (f: 1090519040.000000, d: 5.387880e-315)
[2024-05-18T13:27:37.922Z] HHLT stderr v10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-05-18T13:27:37.922Z] HHLT stderr v11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-05-18T13:27:37.922Z] HHLT stderr v12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-05-18T13:27:37.922Z] HHLT stderr v13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-05-18T13:27:37.922Z] HHLT stderr v14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-05-18T13:27:37.922Z] HHLT stderr v15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-05-18T13:27:37.922Z] HHLT stderr v16 bfd0000000000000 (f: 0.000000, d: -2.500000e-01)
[2024-05-18T13:27:37.922Z] HHLT stderr v17 3fd56847b4415555 (f: 3024180480.000000, d: 3.344898e-01)
[2024-05-18T13:27:37.922Z] HHLT stderr v18 bf72fd9fa58ecb18 (f: 2777598720.000000, d: -4.636406e-03)
[2024-05-18T13:27:37.922Z] HHLT stderr v19 3fe62e42fefa39ef (f: 4277811712.000000, d: 6.931472e-01)
[2024-05-18T13:27:37.922Z] HHLT stderr v20 7fefffffffffffff (f: 4294967296.000000, d: 1.797693e+308)
[2024-05-18T13:27:37.922Z] HHLT stderr v21 c1150fb645a1cac1 (f: 1168231168.000000, d: -3.450696e+05)
[2024-05-18T13:27:37.922Z] HHLT stderr v22 bfb16872b03ff0cf (f: 2956980480.000000, d: -6.800000e-02)
[2024-05-18T13:27:37.922Z] HHLT stderr v23 c1150fb600000000 (f: 0.000000, d: -3.450695e+05)
[2024-05-18T13:27:37.922Z] HHLT stderr v24 bc90000000000000 (f: 0.000000, d: -5.551115e-17)
[2024-05-18T13:27:37.922Z] HHLT stderr v25 bca8000000000000 (f: 0.000000, d: -1.665335e-16)
[2024-05-18T13:27:37.922Z] HHLT stderr v26 3ff0000000000000 (f: 0.000000, d: 1.000000e+00)
[2024-05-18T13:27:37.922Z] HHLT stderr v27 bfb16872b03ff0cf (f: 2956980480.000000, d: -6.800000e-02)
[2024-05-18T13:27:37.922Z] HHLT stderr v28 000000003e000000 (f: 1040187392.000000, d: 5.139209e-315)
[2024-05-18T13:27:37.922Z] HHLT stderr v29 000000003e000000 (f: 1040187392.000000, d: 5.139209e-315)
[2024-05-18T13:27:37.922Z] HHLT stderr v30 c0e0000000000000 (f: 0.000000, d: -3.276800e+04)
[2024-05-18T13:27:37.922Z] HHLT stderr v31 c0e0000000000000 (f: 0.000000, d: -3.276800e+04)

......

[2024-05-18T13:27:37.923Z] HHLT stderr Unhandled exception
[2024-05-18T13:27:37.923Z] HHLT stderr Type=Segmentation error vmState=0x00000000
[2024-05-18T13:27:37.923Z] HHLT stderr J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000002
[2024-05-18T13:27:37.923Z] HHLT stderr Handler1=000000010113A82C Handler2=0000000100FF4DA4 InaccessibleAddress=0000000000000016
[2024-05-18T13:27:37.923Z] HHLT stderr x0=00000001501CB700 x1=00000001780DE7C8 x2=000000037A040E50 x3=00000001412B9200
[2024-05-18T13:27:37.923Z] HHLT stderr x4=00000001780DE7D8 x5=00000001780DE7C0 x6=00000001780DE7B8 x7=00000001780DE820
[2024-05-18T13:27:37.923Z] HHLT stderr x8=000000015001A620 x9=000000000000000E x10=00000001780DE7C0 x11=00000001780DE7C8
[2024-05-18T13:27:37.923Z] HHLT stderr x12=00000001780DE7D8 x13=00000001780DE7B0 x14=0000000101184030 x15=0000000000000010
[2024-05-18T13:27:37.923Z] HHLT stderr x16=0000000101183A9C x17=0000000000000000 x18=00000001780DED00 x19=0000000040000000
[2024-05-18T13:27:37.923Z] HHLT stderr x20=0000000000000006 x21=00000001780DE7B0 x22=00000001103B07CC x23=00000001780DE7C0
[2024-05-18T13:27:37.923Z] HHLT stderr x24=00000001780DE7D8 x25=00000001780DE7C8 x26=00000001780DE7C0 x27=00000001780DE7B8
[2024-05-18T13:27:37.923Z] HHLT stderr x28=000000010125ED84 x29(FP)=00000001780DE7A0 x30(LR)=0000000101183A90 x31(SP)=00000001780DE500
[2024-05-18T13:27:37.923Z] HHLT stderr PC=0000000101198510 SP=00000001780DE500
[2024-05-18T13:27:37.923Z] HHLT stderr v0 000000014804b759 (f: 1208268672.000000, d: 2.718960e-314)
[2024-05-18T13:27:37.923Z] HHLT stderr v1 000000014803bb98 (f: 1208204160.000000, d: 2.718928e-314)
[2024-05-18T13:27:37.923Z] HHLT stderr v2 0000000040000000 (f: 1073741824.000000, d: 5.304989e-315)
[2024-05-18T13:27:37.923Z] HHLT stderr v3 7ff0000000000000 (f: 0.000000, d: inf)
[2024-05-18T13:27:37.923Z] HHLT stderr v4 00000000c0000000 (f: 3221225472.000000, d: 1.591497e-314)
[2024-05-18T13:27:37.923Z] HHLT stderr v5 be9e70430f68aab5 (f: 258517680.000000, d: -4.535693e-07)
[2024-05-18T13:27:37.923Z] HHLT stderr v6 40c0000000000000 (f: 0.000000, d: 8.192000e+03)
[2024-05-18T13:27:37.923Z] HHLT stderr v7 c0c0000000000000 (f: 0.000000, d: -8.192000e+03)
[2024-05-18T13:27:37.923Z] HHLT stderr v8 000000003f6b86d4 (f: 1064011456.000000, d: 5.256915e-315)
[2024-05-18T13:27:37.924Z] HHLT stderr v9 0000000041000000 (f: 1090519040.000000, d: 5.387880e-315)
[2024-05-18T13:27:37.924Z] HHLT stderr v10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-05-18T13:27:37.924Z] HHLT stderr v11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-05-18T13:27:37.924Z] HHLT stderr v12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-05-18T13:27:37.924Z] HHLT stderr v13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-05-18T13:27:37.924Z] HHLT stderr v14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-05-18T13:27:37.924Z] HHLT stderr v15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-05-18T13:27:37.924Z] HHLT stderr v16 bfd0000000000000 (f: 0.000000, d: -2.500000e-01)
[2024-05-18T13:27:37.924Z] HHLT stderr v17 3fd54a2ca0742f55 (f: 2691968768.000000, d: 3.326522e-01)
[2024-05-18T13:27:37.924Z] HHLT stderr v18 3f6649a4fe80e0b3 (f: 4269859072.000000, d: 2.720663e-03)
[2024-05-18T13:27:37.924Z] HHLT stderr v19 3fe62e42fefa39ef (f: 4277811712.000000, d: 6.931472e-01)
[2024-05-18T13:27:37.924Z] HHLT stderr v20 7fefffffffffffff (f: 4294967296.000000, d: 1.797693e+308)
[2024-05-18T13:27:37.924Z] HHLT stderr v21 c1150fb645a1cac1 (f: 1168231168.000000, d: -3.450696e+05)
[2024-05-18T13:27:37.924Z] HHLT stderr v22 bfb16872b03ff0cf (f: 2956980480.000000, d: -6.800000e-02)
[2024-05-18T13:27:37.924Z] HHLT stderr v23 c1150fb600000000 (f: 0.000000, d: -3.450695e+05)
[2024-05-18T13:27:37.924Z] HHLT stderr v24 bc90000000000000 (f: 0.000000, d: -5.551115e-17)
[2024-05-18T13:27:37.924Z] HHLT stderr v25 bca8000000000000 (f: 0.000000, d: -1.665335e-16)
[2024-05-18T13:27:37.924Z] HHLT stderr v26 3ff0000000000000 (f: 0.000000, d: 1.000000e+00)
[2024-05-18T13:27:37.924Z] HHLT stderr v27 00000000bf800000 (f: 3212836864.000000, d: 1.587352e-314)
[2024-05-18T13:27:37.924Z] HHLT stderr v28 000000003e000000 (f: 1040187392.000000, d: 5.139209e-315)
[2024-05-18T13:27:37.924Z] HHLT stderr v29 000000003e000000 (f: 1040187392.000000, d: 5.139209e-315)
[2024-05-18T13:27:37.924Z] HHLT stderr v30 7ff0000000000000 (f: 0.000000, d: inf)
[2024-05-18T13:27:37.924Z] HHLT stderr v31 7ff0000000000000 (f: 0.000000, d: inf)
[2024-05-18T13:27:37.924Z] HHLT stderr Module=/Users/jenkins/workspace/Test_openjdk17_j9_extended.system_aarch64_mac_testList_2/jdkbinary/j2sdk-image/Contents/Home/lib/default/libj9vm29.dylib
[2024-05-18T13:27:37.924Z] HHLT stderr Module_base_address=0000000101118000 Symbol=_ZN26VM_BytecodeInterpreterFull3runEP10J9VMThread
[2024-05-18T13:27:37.924Z] HHLT stderr Symbol_address=0000000101183A9C
[2024-05-18T13:27:37.924Z] HHLT stderr Target=2_90_20240416_560 (Mac OS X 12.6.9)
[2024-05-18T13:27:37.924Z] HHLT stderr CPU=aarch64 (8 logical CPUs) (0x400000000 RAM)
[2024-05-18T13:27:37.924Z] HHLT stderr ----------- Stack Backtrace -----------
[2024-05-18T13:27:37.924Z] HHLT stderr ---------------------------------------
[2024-05-18T13:27:37.924Z] HHLT stderr JVMDUMP039I Processing dump event "gpf", detail "" at 2024/05/18 09:27:36 - please wait.

[2024-05-18T13:29:23.366Z] HeapHogLoadTest_5m_0_FAILED

50x grinder - passed

@babsingh
Copy link
Contributor

babsingh commented May 27, 2024 •
edited

are the arguments on the stack being populated from the JIT?

Given that the previous frame is a jitted method, yes.

DDR is reporting corrupt data while iterating the O-slots of the J2I frame. Since the arguments on the stack are being populated by the JIT, @nbhuiyan can you please check if the JIT is supplying the correct arguments for the J2I frame?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
segfault Issues that describe segfaults / JVM crashes test failure
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@babsingh @JasonFengJ9 @gacholio @pshipton @nbhuiyan @knn-k