Running queries ocassionally causes ArcadeDB to get stuck at 100% CPU usage

[shasaur]

ArcadeDB Version: ArcadeDB Server v23.1.1 (build aa687e4/1673895089679/main)

JDK Version: OpenJDK (zsrc11.54.25-jdk11.0.14.1) [I think]

OS: Windows 11 Home (21H2)

Sometimes when I run a cypher query or perhaps just connect / disconnect from the database, the CPU usage increases to 100% and gets stuck there. It stays at 100% even when I am not running any queries.

I don't know what causes it to happen as sometimes I can do a lot of queries without this happening, and sometimes after the first few seconds of queries it gets stuck.

My guess would be that there is something accidentally jumps into a 'busy wait' mode where it gets stuck checking something in an infinite for loop. Would be really nice to have a fix for this.

[lvca]

Are you using the setting -Darcadedb.profile=high-performance or -Darcadedb.asyncOperationsQueueImpl=fast?

[lvca]

Or are you running o Docker? We have another issue about that.

[shasaur]

I'm not using any special setting (as far as I know), just running server.bat. So I just tested about half an hour of production usage, and it did not seem to happen. However, I think I closed it during a running transaction and it shot up to 70% and got stuck there.

Could it be happening when connections are closed during running transactions or something like that?

[shasaur]

Oh actually, the only thing is that I'm using the Postgres plug-in. And no, I'm not using Docker.

[lvca]

This was never found in production, AFAIK, so it could be the Postgres plugin. Is there any chance you can do a dump of the JVM when that happens? You can execute a jstack <pid>.

[shasaur]

Done! Here's the JVM dump and also the console output of the DB.

[shasaur]

Also just a note, now that (I think) it's happening only between connections, it's not super high priority for me as my use is based only on a single connection. Though I leave this prioritisation up to you.

[lvca]

Thanks for the dump! The issue seems to be in the postgres threads:

"ArcadeDB-postgres//0:0:0:0:0:0:0:1" #56 prio=5 os_prio=0 cpu=189828.13ms elapsed=239.99s tid=0x000002832c484000 nid=0x5d5c runnable  [0x000000e2f3fff000]
   java.lang.Thread.State: RUNNABLE
	at java.lang.Throwable.fillInStackTrace(java.base@11.0.14.1/Native Method)
	at java.lang.Throwable.fillInStackTrace(java.base@11.0.14.1/Throwable.java:787)
	- locked <0x0000000722b90c40> (a java.io.EOFException)
	at java.lang.Throwable.<init>(java.base@11.0.14.1/Throwable.java:255)
	at java.lang.Exception.<init>(java.base@11.0.14.1/Exception.java:54)
	at java.io.IOException.<init>(java.base@11.0.14.1/IOException.java:47)
	at java.io.EOFException.<init>(java.base@11.0.14.1/EOFException.java:50)
	at java.io.DataInputStream.readUnsignedByte(java.base@11.0.14.1/DataInputStream.java:295)
	at com.arcadedb.network.binary.ChannelBinary.readUnsignedByte(ChannelBinary.java:87)
	at com.arcadedb.postgres.PostgresNetworkExecutor.readNextByte(PostgresNetworkExecutor.java:994)
	at com.arcadedb.postgres.PostgresNetworkExecutor.readMessage(PostgresNetworkExecutor.java:953)
	at com.arcadedb.postgres.PostgresNetworkExecutor.run(PostgresNetworkExecutor.java:140)

The reading from the TCP/IP socket causes tons of exceptions. Checking on it.