Dropwizard 2.1.1 hangs on 1-core host intermittently

[yongzhao15]

5 days after we downgraded service hosts from 4-core to 1-core, we started to notice that dropwizard hangs intermittently. The symptom is service still accept web requests but doesn't process any. client will timeout on such host. so far it happened on 5 out of 100+ hosts.

based on locahost:/metrics and threads on the problem host

1. QueuedThreadPool.dw.size dropped from normally 5-7 to fixed size 2 and stuck there unless restarted.
2. QueuedThreadPool.dw.utilization went from 30-80% to 100%.

"org.eclipse.jetty.util.thread.QueuedThreadPool.dw.size" : {
      "value" : 2
    },
    "org.eclipse.jetty.util.thread.QueuedThreadPool.dw.utilization" : {
      "value" : 1.0
    },

based locahost:/threads, 2 remaining threads are acceptor and selector.

find ~/Downloads/thread.txt  | xargs egrep 'dw-[0-9]' -A9

"dw-70-acceptor-0@627f2120-application@67c6b166{SSL, (ssl, http/1.1)}{0.0.0.0:24443}" id=70 state=RUNNABLE (running in native)
    at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
    at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:424)
    at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:252)
    at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:388)
    at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:704)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
    at java.lang.Thread.run(Thread.java:750)
--
"dw-82961" id=82961 state=RUNNABLE (running in native)
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
      - locked java.util.Collections$UnmodifiableSet@5ec51094
      - locked sun.nio.ch.EPollSelectorImpl@5afa9f94
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
    at org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:183)

there's no worker threads in TIMED_WAITING state, eg on a healthy host.

curl localhost:19001/threads | egrep dw-[0-9]+ -A9

"dw-71-acceptor-0@6af94767-application@62c72490{SSL, (ssl, http/1.1)}{0.0.0.0:24443}" id=71 state=RUNNABLE (running in native)
    at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
    at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:424)
    at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:252)
    at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:388)
    at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:704)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
    at java.lang.Thread.run(Thread.java:750)

--
"dw-67336" id=67336 state=TIMED_WAITING
    - waiting on <0x003e2248> (a java.util.concurrent.SynchronousQueue$TransferStack)
    - locked <0x003e2248> (a java.util.concurrent.SynchronousQueue$TransferStack)
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
    at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
    at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
    at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:324)
    at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:399)
--
"dw-67392" id=67392 state=TIMED_WAITING
    - waiting on <0x003e2248> (a java.util.concurrent.SynchronousQueue$TransferStack)
    - locked <0x003e2248> (a java.util.concurrent.SynchronousQueue$TransferStack)
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
    at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
    at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
    at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:324)
    at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:399)
--
"dw-67446" id=67446 state=RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
      - locked java.util.Collections$UnmodifiableSet@6f55a532
      - locked sun.nio.ch.EPollSelectorImpl@6b6a3c00
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
    at org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:183)

we checked 4-core host: there're 2 selectors vs 1 selector on 1-core host.

Does 2 selectors provide some kind of redundancy? ie if one malfunctioned, the other can fill the gap. If so, could adding "selectorThreads: 2" to applicationConnectors configuration solve our problem?