Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

thread blocked in virtual thread envrionment with 1.4.14 version #767

Open
walkertest opened this issue Jan 12, 2024 · 4 comments
Open

thread blocked in virtual thread envrionment with 1.4.14 version #767

walkertest opened this issue Jan 12, 2024 · 4 comments

Comments

@walkertest
Copy link

walkertest commented Jan 12, 2024 •
edited

Env

  • Logback version :1.4.14
  • Springboot version: 3.2.1
  • My server framework with a reactor model which is like the opensource framework-tars :https://github.com/TarsCloud/TarsJava . I support the virtual thread model .
  • The stress request qps: 30_000/s
  • javaVersion:openjdk version "21" 2023-09-19/OpenJDK Runtime Environment (build 21+35-2513)/OpenJDK 64-Bit Server VM (build 21+35-2513, mixed mode, sharing)

Problem

  • My server thread was blocked by logger.error method. And then my program cannot handle new request.

the request and response

The thread stack

  • You can see my reactor thread is waiting for the logger.error method.

  • The full thread stack is in the added file.
    stack1.txt

  • the jcmd thread dump info（jcmd Thread.dump_to_file ）：
    thread_jcmdaa.txt
    thread_jcmdab.txt

"ts-reactor-0" #47 [693] prio=5 os_prio=0 cpu=195123.31ms elapsed=69926.73s tid=0x00007fbc7a0daf10 nid=693 waiting on condition  [0x00007fbc4cebf000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@21/Native Method)
	- parking to wait for  <0x0000000759b899e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@21/LockSupport.java:371)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@21/AbstractQueuedSynchronizer.java:519)
	at java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@21/ForkJoinPool.java:3780)
	at java.util.concurrent.ForkJoinPool.managedBlock(java.base@21/ForkJoinPool.java:3725)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@21/AbstractQueuedSynchronizer.java:1707)
	at java.util.concurrent.ArrayBlockingQueue.put(java.base@21/ArrayBlockingQueue.java:370)
	at ch.qos.logback.core.AsyncAppenderBase.putUninterruptibly(AsyncAppenderBase.java:186)
	at ch.qos.logback.core.AsyncAppenderBase.put(AsyncAppenderBase.java:177)
	at ch.qos.logback.core.AsyncAppenderBase.append(AsyncAppenderBase.java:166)
	at ch.qos.logback.core.UnsynchronizedAppenderBase.doAppend(UnsynchronizedAppenderBase.java:85)
	at ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:51)
	at ch.qos.logback.classic.Logger.appendLoopOnAppenders(Logger.java:272)
	at ch.qos.logback.classic.Logger.callAppenders(Logger.java:259)
	at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:426)
	at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:386)
	at ch.qos.logback.classic.Logger.error(Logger.java:539)
	at com.huya.taf.net.core.nio.TCPSession.handleVtServer(TCPSession.java:218)
	at com.huya.taf.net.core.nio.TCPSession.readRequest(TCPSession.java:177)
	at com.huya.taf.net.core.nio.TCPSession.read(TCPSession.java:140)
	at com.huya.taf.net.core.nio.TCPAcceptor.handleReadEvent(TCPAcceptor.java:111)
	at com.huya.taf.net.core.nio.Reactor.dispatchEvent(Reactor.java:197)
	at com.huya.taf.net.core.nio.Reactor.run(Reactor.java:114)

"ts-reactor-1" #48 [694] prio=5 os_prio=0 cpu=223469.95ms elapsed=69926.73s tid=0x00007fbc7a0eda90 nid=694 waiting on condition  [0x00007fbc4ce7e000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@21/Native Method)
	- parking to wait for  <0x0000000759b899e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@21/LockSupport.java:371)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@21/AbstractQueuedSynchronizer.java:519)
	at java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@21/ForkJoinPool.java:3780)
	at java.util.concurrent.ForkJoinPool.managedBlock(java.base@21/ForkJoinPool.java:3725)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@21/AbstractQueuedSynchronizer.java:1707)
	at java.util.concurrent.ArrayBlockingQueue.put(java.base@21/ArrayBlockingQueue.java:370)
	at ch.qos.logback.core.AsyncAppenderBase.putUninterruptibly(AsyncAppenderBase.java:186)
	at ch.qos.logback.core.AsyncAppenderBase.put(AsyncAppenderBase.java:177)
	at ch.qos.logback.core.AsyncAppenderBase.append(AsyncAppenderBase.java:166)
	at ch.qos.logback.core.UnsynchronizedAppenderBase.doAppend(UnsynchronizedAppenderBase.java:85)
	at ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:51)
	at ch.qos.logback.classic.Logger.appendLoopOnAppenders(Logger.java:272)
	at ch.qos.logback.classic.Logger.callAppenders(Logger.java:259)
	at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:426)
	at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:386)
	at ch.qos.logback.classic.Logger.error(Logger.java:539)
	at com.huya.taf.net.core.nio.Reactor.disConnectWithException(Reactor.java:144)
	at com.huya.taf.net.core.nio.Reactor.run(Reactor.java:117)

"ts-reactor-2" #49 [695] prio=5 os_prio=0 cpu=196075.69ms elapsed=69926.73s tid=0x00007fbc7a0eec00 nid=695 waiting on condition  [0x00007fbc4ce3d000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@21/Native Method)
	- parking to wait for  <0x0000000759b895b8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
	at java.util.concurrent.locks.LockSupport.park(java.base@21/LockSupport.java:221)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@21/AbstractQueuedSynchronizer.java:754)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@21/AbstractQueuedSynchronizer.java:990)
	at java.util.concurrent.locks.ReentrantLock$Sync.lock(java.base@21/ReentrantLock.java:153)
	at java.util.concurrent.locks.ReentrantLock.lock(java.base@21/ReentrantLock.java:322)
	at java.util.concurrent.ArrayBlockingQueue.remainingCapacity(java.base@21/ArrayBlockingQueue.java:485)
	at ch.qos.logback.core.AsyncAppenderBase.isQueueBelowDiscardingThreshold(AsyncAppenderBase.java:170)
	at ch.qos.logback.core.AsyncAppenderBase.append(AsyncAppenderBase.java:162)
	at ch.qos.logback.core.UnsynchronizedAppenderBase.doAppend(UnsynchronizedAppenderBase.java:85)
	at ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:51)
	at ch.qos.logback.classic.Logger.appendLoopOnAppenders(Logger.java:272)
	at ch.qos.logback.classic.Logger.callAppenders(Logger.java:259)
	at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:426)
	at ch.qos.logback.classic.Logger.filterAndLog_2(Logger.java:419)
	at ch.qos.logback.classic.Logger.info(Logger.java:592)
	at com.huya.taf.net.core.nio.TCPSession.handleVtServer(TCPSession.java:208)
	at com.huya.taf.net.core.nio.TCPSession.readRequest(TCPSession.java:177)
	at com.huya.taf.net.core.nio.TCPSession.read(TCPSession.java:140)
	at com.huya.taf.net.core.nio.TCPAcceptor.handleReadEvent(TCPAcceptor.java:111)
	at com.huya.taf.net.core.nio.Reactor.dispatchEvent(Reactor.java:197)
	at com.huya.taf.net.core.nio.Reactor.run(Reactor.java:114)

"ts-reactor-3" #50 [696] prio=5 os_prio=0 cpu=194924.27ms elapsed=69926.73s tid=0x00007fbc7a0f0020 nid=696 waiting on condition  [0x00007fbc4cdfc000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@21/Native Method)
	- parking to wait for  <0x0000000759b899e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@21/LockSupport.java:371)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@21/AbstractQueuedSynchronizer.java:519)
	at java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@21/ForkJoinPool.java:3780)
	at java.util.concurrent.ForkJoinPool.managedBlock(java.base@21/ForkJoinPool.java:3725)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@21/AbstractQueuedSynchronizer.java:1707)
	at java.util.concurrent.ArrayBlockingQueue.put(java.base@21/ArrayBlockingQueue.java:370)
	at ch.qos.logback.core.AsyncAppenderBase.putUninterruptibly(AsyncAppenderBase.java:186)
	at ch.qos.logback.core.AsyncAppenderBase.put(AsyncAppenderBase.java:177)
	at ch.qos.logback.core.AsyncAppenderBase.append(AsyncAppenderBase.java:166)
	at ch.qos.logback.core.UnsynchronizedAppenderBase.doAppend(UnsynchronizedAppenderBase.java:85)
	at ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:51)
	at ch.qos.logback.classic.Logger.appendLoopOnAppenders(Logger.java:272)
	at ch.qos.logback.classic.Logger.callAppenders(Logger.java:259)
	at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:426)
	at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:386)
	at ch.qos.logback.classic.Logger.error(Logger.java:539)
	at com.huya.taf.net.core.nio.TCPSession.handleVtServer(TCPSession.java:218)
	at com.huya.taf.net.core.nio.TCPSession.readRequest(TCPSession.java:177)
	at com.huya.taf.net.core.nio.TCPSession.read(TCPSession.java:140)
	at com.huya.taf.net.core.nio.TCPAcceptor.handleReadEvent(TCPAcceptor.java:111)
	at com.huya.taf.net.core.nio.Reactor.dispatchEvent(Reactor.java:197)
	at com.huya.taf.net.core.nio.Reactor.run(Reactor.java:114)

"ts-reactor-4" #51 [697] prio=5 os_prio=0 cpu=194829.49ms elapsed=69926.73s tid=0x00007fbc7a0f15b0 nid=697 waiting on condition  [0x00007fbc4cdbb000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@21/Native Method)
	- parking to wait for  <0x0000000759b899e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@21/LockSupport.java:371)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@21/AbstractQueuedSynchronizer.java:519)
	at java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@21/ForkJoinPool.java:3780)
	at java.util.concurrent.ForkJoinPool.managedBlock(java.base@21/ForkJoinPool.java:3725)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@21/AbstractQueuedSynchronizer.java:1707)
	at java.util.concurrent.ArrayBlockingQueue.put(java.base@21/ArrayBlockingQueue.java:370)
	at ch.qos.logback.core.AsyncAppenderBase.putUninterruptibly(AsyncAppenderBase.java:186)
	at ch.qos.logback.core.AsyncAppenderBase.put(AsyncAppenderBase.java:177)
	at ch.qos.logback.core.AsyncAppenderBase.append(AsyncAppenderBase.java:166)
	at ch.qos.logback.core.UnsynchronizedAppenderBase.doAppend(UnsynchronizedAppenderBase.java:85)
	at ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:51)
	at ch.qos.logback.classic.Logger.appendLoopOnAppenders(Logger.java:272)
	at ch.qos.logback.classic.Logger.callAppenders(Logger.java:259)
	at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:426)
	at ch.qos.logback.classic.Logger.filterAndLog_2(Logger.java:419)
	at ch.qos.logback.classic.Logger.info(Logger.java:592)
	at com.huya.taf.net.core.nio.TCPSession.handleVtServer(TCPSession.java:208)
	at com.huya.taf.net.core.nio.TCPSession.readRequest(TCPSession.java:177)
	at com.huya.taf.net.core.nio.TCPSession.read(TCPSession.java:140)
	at com.huya.taf.net.core.nio.TCPAcceptor.handleReadEvent(TCPAcceptor.java:111)
	at com.huya.taf.net.core.nio.Reactor.dispatchEvent(Reactor.java:197)
	at com.huya.taf.net.core.nio.Reactor.run(Reactor.java:114)
@walkertest
Copy link
Author

walkertest commented Jan 14, 2024 •
edited

jcmd Thread.dump_to_file info:
The dump file is bigger than 26M(the github issue limit), so i split the file to two files.

thread_jcmdaa.txt
thread_jcmdab.txt

@ArtemiyVorobev
Copy link

ArtemiyVorobev commented Feb 15, 2024 •
edited

On your project we have the same problem. Waiting for fix

@ArtemiyVorobev
Copy link

@walkertest Have you found a work around ?

@ceki
Copy link
Member

ceki commented Feb 15, 2024

@walkertest Can you please provide the logback configuration file?

@koszta5 koszta5 mentioned this issue Mar 4, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@ceki @walkertest @ArtemiyVorobev