Better support for cross compiling to Windows

[messense]

1. Cross compile to MSVC targets can be done with integrating cargo-xwin.
2. Cross compile to GNU targets can already be done with --zig I think.
3. Even with abi3 an interpreter is required on Windows, maybe we can use something like the python3-dll-a project to generate an Windows import library for python3.dll automatically. Resolved in pyo3-build-config: Add python3-dll-a crate support pyo3#2282

[ravenexp]

Hi!

I've been using maturin to cross-compile PyO3 extension modules to Windows (MinGW) for years now.
I don't use MSVC, so I can't tell whether option 1 currently works. But as for cross-compiling to GNU targets, everything seems to be in place already. PyO3 already has decent Windows cross-compile support.
I've been using custom shell scripts to generate python3.dll import libraries, and then finally packaged this functionality into an easy-to-use python3-dll-a crate.

I think option 2 is not really needed because there are no libc version issues with MinGW and the default x86_64-pc-windows-gnu linker works just fine for Python extension modules (in my experience).

As for option 3, I think I should push for python3-dll-a integration into PyO3 itself (into the pyo3-build-config sub-crate),
which will solve the import library problem for good. I'll open an issue in the PyO3 project and see how it goes.

As for what maturin could do to improve the Windows cross-compile experience, I think solving #824 would be a good start. Ideally, maturin should not touch the PYO3_NO_PYTHON and PYO3_CROSS_LIB_DIR env variables at all.

cargo build --target x86_64-pc-windows-gnu already works when PYO3_CROSS_LIB_DIR is provided, and I think maturin
should only use this variable is it wants to override the library location. As for the PYO3_NO_PYTHON hack, I strongly feel that it should be removed, because currently there are two competing cross-compile code paths in PyO3 build.rs and it's a total mess. I've already submitted numerous bug fixes for PyO3 because cargo build and maturin build behaviors keep diverging because of the PYO3_NO_PYTHON use by maturin.

[messense]

I think option 2 is not really needed because there are no libc version issues with MinGW and the default x86_64-pc-windows-gnu linker works just fine for Python extension modules (in my experience).

Ideally it should work for cross compiling from macOS/Linux to Windows. I assume cross compile from Linux to Windows works for you already?

[ravenexp]

Ideally it should work for cross compiling from macOS/Linux to Windows. I assume cross compile from Linux to Windows works for you already?

Yes, cross-compiling from Linux works fine. I don't know anything about MacOS though, so I just assumed that the x86_64-pc-windows-gnu toolchain works the same on Linux and MacOS.

[messense]

Ideally, maturin should not touch the PYO3_NO_PYTHON and PYO3_CROSS_LIB_DIR env variables at all.

For PYO3_CROSS_LIB_DIR it's only set for PyPy cross compiling support which I think isn't a big issue and can be removed later since pyo3 0.16 added the support.

I'm not sure about PYO3_NO_PYTHON, it's only set for CPython abi3 mode right now. I think we can exclude Windows from it since on Windows it requires a python3.dll anyway?

[ravenexp]

I'm not sure about PYO3_NO_PYTHON, it's only set for CPython abi3 mode right now. I think we can exclude Windows from it since on Windows it requires a python3.dll anyway?

I believe this is how PYO3_NO_PYTHON came to be in maturin:

f1a8230

PyO3 itself can handle cross-compilation without PYO3_NO_PYTHON, and I think it was added to PyO3 precisely because maturin needed it at that point of time.

[messense]

PyO3 itself can handle cross-compilation without PYO3_NO_PYTHON, and I think it was added to PyO3 precisely because maturin needed it at that point of time.

OK, let's see what the CI says for #847

[messense]

Unfortunately it seems that pyo3 handles abi3 wheel cross compiling badly when PYO3_NO_PYTHON is absent.
https://github.com/PyO3/maturin/runs/5527269955?check_suite_focus=true

cc @davidhewitt

[ravenexp]

Hmm, it is interesting that its the Linux-to-Linux cross-compilation that fails, not Linux-to-Windows.

Also I think #824 should be solved before PYO3_NO_PYTHON can be get rid of.

[messense]

Well, I'm not sure how to solve #824, you're welcome to try :-).

[ravenexp]

Well, I'm not sure how to solve #824, you're welcome to try :-).

You've just fixed it with #847 :)

It turns out I was completely wrong about the true cause of #824, and it happened to be just another a consequence of PYO3_NO_PYTHON use by maturin. 🤦‍♂️

[ravenexp]

PyO3 itself can handle cross-compilation without PYO3_NO_PYTHON, and I think it was added to PyO3 precisely because maturin needed it at that point of time.

OK, let's see what the CI says for #847

Unfortunately it seems that pyo3 handles abi3 wheel cross compiling badly when PYO3_NO_PYTHON is absent. https://github.com/PyO3/maturin/runs/5527269955?check_suite_focus=true

Think this should work now with pyo3 v0.16.3+

[ravenexp]

Is there anything left on the TODO list? If there are some issues I haven't encountered yet, I could try to work on them in my free time.

I'm particularly interested in cross-compiling to Windows to avoid dealing with Windows docker images and non-free MSVC tools.

[messense]

@ravenexp I want to integrate cargo-xwin for cross compiling to MSVC target. It can also require --zig because then we can use zig dlltool (simply by creating a symlink as llvm-dlltool and put it in PATH) which is same as llvm-dlltool for generating python3.dll import library without a full LLVM/Clang installation.

[ravenexp]

python3-dll-a can be taught about zig dlltool too. It already tries LIB.EXE and then llvm-dlltool in sequence.

[ravenexp]

python3-dll-a can be taught about zig dlltool too. It already tries LIB.EXE and then llvm-dlltool in sequence.

I've started working on it in PyO3/python3-dll-a#18.
It already can call zig dlltool in place of llvm-dlltool, but does not attempt to use it yet.

Is there an easy way to detect that python3-dll-a is being built with maturin --zig or cargo zigbuild?
Perhaps there are some env vars it can check? Or should it try llvm-dlltool first and then fall back to zig dlltool?

[messense]

Perhaps there are some env vars it can check?

No env vars right now but I think we should add one. And we should support both zig dlltool and python3 -m ziglang dlltool. zig also distributes itself on PyPI but it needs to be invoked via python3 -m ziglang, see ziglang/zig-pypi#4 (comment)

https://github.com/messense/cargo-zigbuild/blob/9f31dcb97d5cff618d60b3c1d18dfb9f51045b7c/src/zig.rs#L218-L263