Feature: act as Make job server when executing locally

[cormacrelf]

If you are building a lot of rust crates locally, they are all going to spin up at least a bunch of LLVM threads, and in future (or with nightly) they will also spin up a bunch of threads for the rustc frontend. This is a bit of an issue because buck2 will saturate its own job queue's frontier (e.g. 8 concurrent builds with -j8) and then each of those rustc invocations will try to use as many threads as it can. You can often saturate N cores by passing -j with N/2 or so. Which means in the usual case of -jN, you are really scheduling about N^2 threads at once, which is surely not great.

There is an existing protocol for limiting the parallelism of individual build actions: the GNU Make jobserver: https://www.gnu.org/software/make/manual/html_node/Job-Slots.html. Rustc and I presume a number of other build tools are able to use the jobserver if you provide the environment variable (e.g. MAKEFLAGS=--jobserver-auth=...), and do the pipe IO / windows IPC serving. This is what tools like cargo do, and of course make itself. Ultimately all the participating processes share a single semaphore.

See here for where rustc acquires and releases jobserver tokens.

The jobserver crate has a type Client that also acts as a server. If you call Client::new when starting a build, pass in the buck2 CLI's -j flag, and use its methods to attach the environment variables to local execution processes, you are basically done.

https://docs.rs/jobserver/latest/jobserver/struct.Client.html#method.new

I am sure Meta will not care about this, given you scale your builds horizontally as parallel as you like, but it would be pretty helpful for OSS builds.