Bound tokio worker thread count

[leftwo]

No description provided.

[pfmooney]

By default, the multi-thread tokio runtime will spawn a number of worker threads equal to the number of CPUs observed on the system. On a gimlet, that means north of 256 threads (128 CPUs * 2 runtimes), which is woefully excessive. The one runtime is used to run dropshot (and its associated bits), and can probably be limited to 4 workers. Determining the worker count for the propolis-specific runtime will require a bit more care, as it probably does need to be scaled with the performance expectations of the instance (more disks will drive more IOPS will require more workers).

[jclulow]

It would be good if we could also provide thread names for the tokio runtimes that get created so that we can tell them apart in, say, prstat. Short names would be best; e.g., rt-http and rt-vm.

[pfmooney]

CC: @leftwo @mkeeter

Is there any further input on where we should start in terms of minimums/maximums on the number of worker threads we spawn for an instance? I was going to start with a minimum of 8, scaling to a max of however many vCPUs an instance has. This is obviously not perfect as processing needs for Crucible presumably scale with the disk count (and workload), not the number of guest CPUs.

[mkeeter]

I don't have a strong opinion on this; 8 + vCPU count seems like a reasonable starting point.

However, be aware that heavy Crucible work (encryption / decryption) is actually done in the Rayon thread pool, not Tokio! The Rayon thread pool can be configured with ThreadPoolBuilder::build_global, so I think we should do this as well.

[pfmooney]

Because the global pool is somewhat challenging to initialize safely, I've filed oxidecomputer/crucible#1285 to cover that part of the problem. It will still be propolis' responsibility to bound the size of the tokio thread pool.