Compress each file in a ThreadPool

[rik]

fix #148

---

I've tested this on a 2020 M1 MacBook Air (4 big + 4 small cores). In milliseconds.

Executable	Files	Brotli	Sequential	Parallel	Win
CLI	4656	No	3190	1050	3.0x
CLI	4656	Yes	78800	17630	4.2x
CLI	783	No	674	299	2.2x
CLI	783	Yes	10740	3630	2.9x
collectstatic	4656	No	8860	6790	1.3x
collectstatic	4656	Yes	82410	25110	3.3x
collectstatic	783	No	1510	1370	1.1x
collectstatic	783	Yes	11370	5120	2.2x

Commands to test yourself

• CLI without Brotli

  time python -m whitenoise.compress --quiet --no-brotli staticfiles

• CLI with Brotli

  time python -m whitenoise.compress --quiet staticfiles

• collectstatic without Brotli

  rm -rf staticfiles/; pip uninstall brotli --yes; time ./manage.py collectstatic --no-input

• collectstatic with Brotli

  rm -rf staticfiles/; pip install brotli; time ./manage.py collectstatic --no-input

[rik]

A previous version of this patch used ProcessPoolExecutor but that resulted in a slight slowdown for some non-Brotli cases.

[rik]

@evansd @adamchainz Apologies for pinging you. May I get some feedback on this approach?

[stumpylog]

It would be great to have this merged. I've patched in a Docker build, where it was made the build much faster, but it would be best to have it built in instead.

[petrprikryl]

Some bench for my project. Tested on i9-11900 (8 cores, 2 threads per core).

Without ThreadPool:

1037 static files copied to '/app/staticfiles', 4137 post-processed.

real    1m37.004s
user    1m35.530s
sys     0m1.356s

With ThreadPool:

1037 static files copied to '/app/staticfiles', 4137 post-processed.

real    0m28.889s
user    3m4.132s
sys     0m2.164s

In CI it goes from 213s to 38s 🚀