In the future will it be possible to run multiprossing, parallelism and GPU processes in Pyodide?

[sebastianherreramonterrosa]

I have the question if it is possible that at some point it will be possible to execute multiprocessing.Pool, concurrent.futures.ProcessPoolExecutor, pytorch or tensorflow.

What are the challenges or limitations for something like this to happen?

[ryanking13]

multiprocessing, concurrent.futures.ProcessPoolExecutor

I'm going to be negative on this one. Pyodide is subject to the constraints of the browser, and a single browser tab runs in a single process. So I don't think multiprocessing will possible even in the future.

pytorch or tensorflow.

For CPU-only mode, I think there is no "hard blocker". In fact, I've tried to build pytorch a few times.

The problem is that these packages have very complex build systems that make it difficult to build for WASM, so I think it would be quite tricky unless someone who knows the build systems of these packages spent a lot of time working on them.

For GPU support, as WebGPU is improving, so I think it will be possible in the long run, and actually tfjs has a WebGPU backend.

But that depends on the developers of tensorflow or PyTorch to implement a backend for WebGPU first, and there's not much we can do on the Pyodide-side.

parallelism

One more thing: SIMD operation is also partially supported in webassembly, and I've actually seen opencv-js use SIMD operations to optimize performance. So in the long run, I'm hoping that numpy and scipy can use simd operations to speed up their performance in WASM. Of course, this is only possible if BLAS and LAPACK libraries support WASM-compliant kernels.