Add Emscripten support

[hoodmane]

Resolves #1454. This is on top of #1462. Also close #1002.

We have trouble with the tests because currently python -m pytest is required (pytest by itself doesn't work as expected). Also python -m pip does not work, have to use pip. It's a work in progress...

TODO:

• Automate Python installation in certain CI environments?

Working:

• multidict -- (requires this fix to their config)
• numpy -- builds wheel but hangs 99% of the way through test suite
• scikitlearn
• scikit-image -- works on v0.19.3, doesn't work on v0.20 because of Meson
• mypy -- NonPlatformWheelError
• ultrajson -- one test failure due to fork
• ril -- rust compilation errors
• pydantic -- one test failure due to asyncio.run

[hoodmane]

@henryiii could you approve the github actions workflows?

[hoodmane]

@henryiii is the windows_x86 failure a flake?

failed to download pip version 20.0.2, pip download exit code 1
The conflict is caused by:
    The user requested pip==20.0.2
    The user requested (constraint) pip==20.0.2
To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict
WARNING: Skipping page https://pypi.org/simple/pip/ because the GET request got Content-Type: .The only supported Content-Type is text/html
ERROR: Cannot install pip==20.0.2 because these package versions have conflicting dependencies.
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies

[henryiii]

I'm curious to see if this fails CI. I know there's a bug which I've just fixed locally but would like to see if anything in CI catches it.

[rth]

Thanks @henryiii !

[henryiii]

Okay, "no space left on device" wasn't the failure I was expecting...

[henryiii]

@hoodmane, do you know why tests are failing to set up a venv?

FWIW, I'm testing it locally with awkward-array like this:

docker run -v $PWD:/cibuildwheel -v $PWD/../../scikit-hep/awkward:/awkward -w /awkward/awkward-cpp --rm -it python:3.11 bash
pip install /cibuildwheel
cibuildwheel --platform pyodide

And I'm getting:

FAILED: _ext.cpython-311-x86_64-linux-gnu.so
: && /tmp/tmp5ii37twm/c++ -fPIC -std=c++14 -std=c++14 -std=c++14 -std=c++14 -s DISABLE_EXCEPTION_CATCHING=0 -O3 -DNDEBUG  -flto -shared  -o _ext.cpython-311-x86_64-linux-gnu.so CMakeFiles/_ext.dir/src/python/_ext.cpp.o CMakeFiles/_ext.dir/src/python/content.cpp.o CMakeFiles/_ext.dir/src/python/forth.cpp.o CMakeFiles/_ext.dir/src/python/io.cpp.o  libawkward-static.a  libawkward-cpu-kernels-static.a && cd /awkward/awkward-cpp/build/cpython-311 && emstrip /awkward/awkward-cpp/build/cpython-311/_ext.cpython-311-x86_64-linux-gnu.so
Traceback (most recent call last):
  File "/tmp/tmp5ii37twm/c++", line 668, in <module>
    compiler_main()
  File "/tmp/tmp5ii37twm/c++", line 664, in compiler_main
    sys.exit(handle_command(args, build_args))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmp5ii37twm/c++", line 654, in handle_command
    returncode = subprocess.run(new_args).returncode
                 ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/subprocess.py", line 548, in run
    with Popen(*popenargs, **kwargs) as process:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/subprocess.py", line 1024, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/local/lib/python3.11/subprocess.py", line 1917, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
OSError: [Errno 7] Argument list too long: 'em++'
[174/174] Linking CXX shared library libawkward.so
em++: warning: EXPORTED_FUNCTIONS is not valid with LINKABLE set (normally due to SIDE_MODULE=1/MAIN_MODULE=1) since all functions are exported this mode.  To export only a subset use SIDE_MODULE=2/MAIN_MODULE=2 [-Wunused-command-line-argument]
em++: warning: ignoring unsupported linker flag: `-soname` [-Wlinkflags]
ninja: build stopped: subcommand failed.

(This works outside of cibuildwheel on awkward-array, by the way, now that 0.23.3 is out)

[rth]

OSError: [Errno 7] Argument list too long: 'em++'

Seems to be the same issue as in pyodide/pyodide#3427 (comment)

[henryiii]

Ahh, yes, that's there in:
https://github.com/scikit-hep/awkward/blob/2d259cc54e3787f1d285a57274e5ce5748ff110f/.github/workflows/docs.yml#L112

Not sure how I'm supposed to pass that with cibuildwheel, though.

[henryiii]

@hoodmane would adding an environment variable version of --exports whole_archive be a good idea for 0.23.4?

[hoodmane]

I think we could do that.

[rth]

Hmm, so the CI now fails building the pp310-manylinux_i686 wheel, unless I misread the logs. Not very clear to me what's the actual error..

[mayeut]

The error was "No space left", this should now be fixed in main.

[celestinoxp]

pyodide updated its version to 0.24 on 13-September-2023...

[joerick]

I merged with main to update this to the new build-frontend option.

[martinRenou]

Really nice work! Thank you for starting this!

Part from the emscripten-forge team, I could really see us providing the ability for our users to use pip-installed packages into emscripten environments.

For this reason, I have a small suggestion about the name of the platform, something more generic like emscripten-32 could suit our use-case as well!

cc. @DerThorsten

[martinRenou]

Suggested change

	choices=["auto", "linux", "macos", "windows", "pyodide"],
	choices=["auto", "linux", "macos", "windows", "emscripten-32"],

[hoodmane]

@martinRenou I see valid arguments in both directions for the platform name. I think calling it Pyodide is slightly better: the platform includes not just the ABI defined by the Emscripten version but also what libraries are statically linked into the main Python executable. These conventions are currently defined as "whatever Pyodide does." If other people decide to match these conventions for their own Emscripten Pythons it will work correctly. But I think calling it just "Emscripten" could mislead people? It's not completely clear.

[martinRenou]

If other people decide to match these conventions for their own Emscripten Pythons it will work correctly

@DerThorsten correct me if I'm wrong but I feel like this is totally fine?

This may be the right opportunity for pyodide and emscripten-forge to start to converge on a general solution for Python packaging in wasm?

[ryanking13]

the platform includes not just the ABI defined by the Emscripten version but also what libraries are statically linked into the main Python executable.

We (Pyodide) link only minimum static libraries to run Python to the main Python module, so if others do this way, too, then it is probably okay. I am not sure how other flags (e.g. WASM_BIGINT) should be matched too.

[hoodmane]

This may be the right opportunity for pyodide and emscripten-forge to start to converge on a general solution for Python packaging in wasm?

Well we keep discussing this but it's never all that clear what this entails specifically. Perhaps it would be a good idea to meet semiregularly, though so far we have never seemed to manage that.

converge on a general solution for Python packaging

If you mean "write down a definition for the platform", I agree that would be a nice thing to do. This is a prerequisite to getting to PyPI uploads.

Another point is that if we don't resolve https://discuss.python.org/t/im-stepping-down-as-sponsor-of-emscripten/41064 the ecosystem support for Emscripten will end up in a pretty confusing state, with support in various packaging tools but being an officially unsupported platform for the interpreter itself.