Parallel Processing does not work on Windows

[Blendify]

Describe the bug
Using the -j auto command does not work on windows but it works on unix

To Reproduce
On a test project run:

sphinx-build -j auto -b html input output

Run that command under Windows and Unix.

Expected behavior
Parallel processing should work across all operating systems.

Not that you can tell that it works or not by the "Waiting for works" logging message (

sphinx/sphinx/builders/__init__.py

Line 466 in 4c1b8b8

logger.info(bold(__('waiting for workers...')))

) not appearing on windows, not to mention the also slower build times.

Your project
Any project works

Environment info

• OS: Windows 10
• Python version: 3.8.0
• Sphinx version: 3.2.1
• Sphinx extensions: none
• Extra tools: none

[Blendify]

Seems to be from https://github.com/sphinx-doc/sphinx/blob/4c1b8b8db6fa25afcab1a6a3634776009cfafddf/sphinx/util/parallel.py#L34 Is there a reason windows is not included? There is no code comment. It also seems it is not supported on macOS with recent python versions. The vary least the documentation needs to be updated to show these limitations.

…

On Mon, Oct 5, 2020, 17:00 Aaron Carlisle ***@***.***> wrote: *Describe the bug* Using the -j auto command does not work on windows but it works on unix *To Reproduce* On a test project run: sphinx-build -j auto -b html input output Run that command under Windows and Unix. *Expected behavior* Parallel processing should work across all operating systems. Not that you can tell that it works or not by the "Waiting for works" logging message ( https://github.com/sphinx-doc/sphinx/blob/4c1b8b8db6fa25afcab1a6a3634776009cfafddf/sphinx/builders/__init__.py#L466) not appearing on windows, not to mention the also slower build times. *Your project* Any project works *Environment info* - OS: Windows 10 - Python version: 3.8.0 - Sphinx version: 3.2.1 - Sphinx extensions: none - Extra tools: none — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#8296>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADT2422ICFXDQAFKSH6CQDLSJIXX7ANCNFSM4SFFWPXA> .

[Blendify]

Looked into it a bit more and it seems multiprocessing is a hairy topic across platforms.
Sphinx should instead look into the python threading module : https://docs.python.org/3/library/threading.html
This makes it a lot easier to execute processes across threads and share memory between them.

[xmo-odoo]

Looked into it a bit more and it seems multiprocessing is a hairy topic across platforms.
Sphinx should instead look into the python threading module : docs.python.org/3/library/threading.html
This makes it a lot easier to execute processes across threads and share memory between them.

Multithreading causes much trickier synchronisation issues, and while Sphinx would do some IO during reading and writing I expect most of its work is actual processing (parsing, transformations, etc...) as it pegs cores during compilation.

The GIL means it would get essentially no speedup from multithreading, as only one thread can execute Python code at a time.

[Blendify]

Yes looking more into this I agree, there will be too much overhead trying to split use multiple threads instead we should stick with what is currently done using multiprocessing but instead of using os.fork() we should use https://docs.python.org/3/library/multiprocessing.html

I haven't had a deep look but I don't think it would be that much work to use that library instead.

[Blendify]

I will do some tests to see if I could get this working if I have the free time.

[xmo-odoo]

FWIW you may want to use #6881 for further discussion.

It doesn't provide a ready-made solution (or even the embryo of one), but it was specifically created by Komiya-san to track the eventual reimplementation of parallel building.

Also sphinx does already use multiprocessing, it doesn't perform raw forks. However the way it uses multiprocessing currently implies forking, so you have your work cut out for you: the as Komiya-san notes in #9092 the entire feature needs to be removed and reimplemented, as the spawning model is a completely different beast than the forking one.

[Blendify]

Using #6881 instead