Fix top level import of loky package when _multiprocessing module is missing

[hoodmane]

I added a test which adds a dummy version of _multiprocessing to the path which just raises an ImportError and then tries to import loky. Then I added try blocks to catch all import errors from _multiprocessing.

Of course, it's not expected that loky will work very well when _multiprocessing is empty, but it is at least useful for it to import successfully.

See also joblib/joblib#1246 which adds the analogous test to joblib.

[rth]

Thanks for working on this @hoodmane !

Let's see what other people think, maybe there is a better solution to joblib/joblib#1246 that wouldn't involve wrapping module imports in try except blocks. For instance, saying that if multiprocessing is not available loky is never imported in joblib could also work (but there are likely lots of inter-dependecies there as well)

Otherwise, there is some irony in a feature request requesting to use loky without multiprocessing :)

[hoodmane]

There are a LOT of places that joblib imports from loky, but only a few places where loky imports from multiprocessing, so I think it's more natural to fix the loky imports. A particularly difficult aspect is that quite a few classes are imported from loky and then subclassed. We need a class of that name so that class SubBlah(SuperBlah) doesn't fail, but our plan is still for SubBlah to raise an error if ever instantiated.

there is some irony in a feature request requesting to use loky without multiprocessing

Yeah it is a bit silly.

[ogrisel]

Let's reconsider this once the current work on chunking and merging #304 is complete which hopefully should happen soon.

[ogrisel]

Shouldn't the __all__ list reflect the actual importable symbols?

[ogrisel]

Please add a comment to explain why:

Suggested change

	except ImportError:
	except ImportError:
	# On environments like pyodide that do not implement the _multiprocessing
	# module in the standard library, we still want loky to be importable without
	# failing, even if not really usable.

[ogrisel]

I would rather not merge this PR while we are in the process or dropping support for Python <3.7.

[hoodmane]

Thanks for the review @ogrisel!

I would rather not merge this PR while we are in the process or dropping support for Python <3.7.

Yeah I'll wait until #303 is done then there will surely be merge conflicts and simplifications to make here.

[ogrisel]

Is this really needed since joblib/joblib#1256 has been merged in joblib?

I assume that no-one will actually use loky on a platform without multiprocessing but it might help support other libraries that optionally use loky but not by default.

Let me know what you think, otherwise we can close.

[tomMoral]

I am in favor of closing as loky is really linked to multiprocessing, and therefor it would be strange to install it on a platform that does not have multiprocessing. It is not like joblib where we have a fallback option and it seems okay to ask downstream libraries to only install/import loky when it is actually usable.

But if you think it is worth it, I am fine with protecting the top level import.

[ogrisel]

I am in favor of closing as loky is really linked to multiprocessing, and therefor it would be strange to install it on a platform that does not have multiprocessing.

Some library that can run on pyiodide might have a dependency that is imported when importing the lib even though it is not used by default. But I am not sure if it is the case for any popular library or not, once joblib is taken care of.

[hoodmane]

Well I am happy with the fix in joblib. I think if we run into other packages that use loky we should also try to patch them to handle missing multiprocessing. This way they probably won't need to download a copy of loky at all.

Thanks @tomMoral and @ogrisel!