QueuedThreadPool "free" threads

[sbordet]

Jetty version
9.4.x

Description
On large machines, the heuristics we have for number of acceptors, selectors, and reserved threads may "steal" a lot of threads from the thread pool, so that a metric such as (threads - idleThreads)/maxThreads may show large percentages even for a completely idle server.

For example, for a 112 cores machine we have

• 4 acceptors
• 56 selectors
• 112 reserved

for a total of 172 "stolen" threads.
With maxThreads=1024 or similar, that is a 17% "utilization" even if the server is idle.

Using busyThreads instead of (threads - idleThreads) takes into account reserved threads, so that figure would be only 60 "stolen" threads for a 6% "utilization".

I wonder if we should at exclude acceptors and selectors from the count (i.e. rather than taking them from the pool, allocate the threads outside the pool -- however this would make the thread budget component irrelevant now).

Excluding them would help people to answer the question: "should I increase maxThreads or I'm good?".
Users that monitor busyThreads/maxThreads will know on a 0-100% scale rather than always having a 6-17% "noise" even when the server is idle.

@gregw @joakime @lorban thoughts?

[sbordet]

OTOH, it's good to know how many threads are actually used, internally, by Jetty.

[gregw]

I think the (threads - idleThreads) is not the right number. It should be (threads - idleThreads + reservedThreads). While a reserved thread is not available directly to handle a request etc. they are available to replace a selector thread so that the selector thread can handle a request it has just accepted.

Other metrics to consider making available:

 readyThreads = idle + reserved;
 unusedThreads = max - threads
 availableThreads = readyThreads + unusedThreads
 unavailableThreads = threads - idle - reserved

[sbordet]

After initial discussion, we may expose the thread budget component, so that we will know how many threads are used internally by Jetty, but also we could use it to know how many are "stolen" and should not be accounted for.

[gregw]

@sbordet
We need to work on vocab, because "stolen" doesn't really capture the meaning hear. I can see the following levels of usage for threads in a QTP, each needs a short name: something like:

• inactive: not started but there is capacity to start another thread
• idle: started and waiting in the pool for a task
• reserved: taken from the pool but reserved to do a task
• budgeted: taken from the pool to do a budgeted task like selecting (could call it "server", but QTP used on client.. maybe allocated? )
• busy: handling a task (eg request). This is a little different to current busy definition

So we have some invariants and derived stats:

 max = inactive + idle + reserved + budgeted + busy
 threads = idle + reserved + budgeted + busy
 busy = threads - idle - reserved - budgeted 
 usable = max - budgeted
 utilised = busy / usable # 0.0 means idle, 1.0 means at capacity