Fix top level multiprocessing.connection import and update yt

[hoodmane]

This updates yt. The new yt has a top level multiprocessing.connection import so we patch that so that it can be imported without raising an error.

• Add a CHANGELOG entry
• Add / update tests

[rth]

Yeah, we have been going a bit back and forth on this. The problem is if we add such an empty file, then it's hard to direct at import time that Python is built without multiprocessing. This is occasionally done by CPython itself (e.g. here). So the general logic for missing functionality so far is if it's not available, the _ prefixed module should fail to import.

Some packages have now come to rely on the current functionality (e.g. joblib/joblib#1256 added a fix specifically for Pyodide), so we cannot easily change this behavior without breaking working code. We should probably have a test for the current behavior with a comment explaining it. Unless we have a solution we are confident would address all the use-cases, and I'm not convinced this is really it.

In this case, IMO it would be better to make changes to the multiprocessing module so it's importable but not usable without _multiprocessing but we would need to talk to CPyton devs about it, to make it more permanent. Though maybe the easiest is for astropy to handle the case where multiprocessing fails to import.

[jobovy]

In this case, IMO it would be better to make changes to the multiprocessing module so it's importable but not usable without _multiprocessing but we would need to talk to CPyton devs about it, to make it more permanent. Though maybe the easiest is for astropy to handle the case where multiprocessing fails to import.

I think yt is the issue here, not astropy? I made the change to astropy to not fail from not being able to import _multiprocessing: astropy/astropy#12911. Although if import _multiprocessing worked, I think astropy would run without any pyodide-specific changes.

Arguably, depending on import _multiprocessing failing isn't a great way to detect that multi-processing is unavailable. Especially given that import multiprocessing works.

[rth]

I think yt is the issue here, not astropy

Yes, wrong astrophysics related package :)

Arguably, depending on import _multiprocessing failing isn't a great way to detect that multi-processing is unavailable. Especially given that import multiprocessing works.

Agree that the situation is not ideal. IMO this kind of thing should be specified by CPython as to what should happen when a stdlib functionality is not available. I opened a discuss.python.org thread about it some time ago, but there was no clear conclusion, as far as I remember.

[hoodmane]

Right. This also inconveniently breaks the tests. Let's just patch yt then.

[hoodmane]

@rth how about this?

[hoodmane]

Arguably, depending on import _multiprocessing failing isn't a great way to detect that multi-processing is unavailable. Especially given that import multiprocessing works.

It would be nice to have a setting multiprocessing.DISABLED or something like that.

[rth]

Sounds good to me, thanks!

It would be nice to have a setting multiprocessing.DISABLED or something like that.

Maybe, but we are not the only platform/build with these problems so we shouldn't have to invent new things either IMO. Similarly this kind of applies to many other optional stdlib modules, that may be excluded in the default Pyodide build. For instance, I just discovered there was a PEP 534 (deferred since 2016) which seems very relevant to this problem.