Cross compiling aarch64 wheel

[odidev]

Building wheels using emulation through qemu is much slower and the cross compilation wheel is almost similar as on x86_64.

Here is the comparison for the time taken in different ways of building wheels

X86_64 (Native build)
Start: Tue Feb 16 15:46:46 UTC 2021
End: Tue Feb 16 15:47:40 UTC 2021
Build Time: 0:54

Qemu (Emulation)
Start: Tue Feb 16 15:50:11 UTC 2021
End: Tue Feb 16 15:54:53 UTC 2021
Build Time: 4:42

Cross compilation
Start: Tue Feb 16 16:03:53 UTC 2021
End: Tue Feb 16 16:04:32 UTC 2021
Build Time: 0:39

Here are the modified/added files:
Modified files in cibuildwheel-
cibuildwheel.patch.txt

Used dockcross docker images with slight modification-
dockcross.patch.txt

Added files in dockcross (manylinux2014-aarch64 directory) -
install_deps.sh.txt
repair_wheel.sh.txt

Users need to modify their setup files to cross compile the wheel(cffi as an example)-
cffi.patch.txt

Please let me know your interest in adding cross compilation support. Thanks in advance.

[YannickJadoul]

Hi @odidev. I think we definitely would be interested! (it's been mentioned multiple times as a possibility, before, when adding QEMU support)

The main concern to be resolved is how to ensure that the produced wheels are fully conforming to the manylinux standard.
Especially if you say

the cross compilation wheel is almost similar as on x86_64

What exactly do you mean? How are the produced wheels different?

Apart from that, I assume we of course need to consider how to integrate this into cibuildwheel's configuration and whether to e.g. still run auditwheel and/or the tests in QEMU, but those are all details to work around. Having cross-compilation would be quite amazing, then :-)

[henryiii]

I've been hoping this would be worked on; compiles are painfully slow in emulation. Scikit-learn barely squeaks by in the 6 hour time window, I think. It needs to be optional, though, as it can take a little work in the package to set up.

Mildly related point: This feature would be just fine in a 2.x series release, because there are no Python 2.7 alternate arch manylinuxes.

tests in QEMU

Yes, at least tests should run in emulation, just like Apple on a AS runner. :)

[YannickJadoul]

It needs to be optional, though, as it can take a little work in the package to set up.

It does? So we can't just shove in a cross-compiler and have things work seemlessly? :-/

[odidev]

Please find my response below-

The main concern to be resolved is how to ensure that the produced wheels are fully conforming to the manylinux standard.

PEP-599 says- If the wheel contains binary executables or shared objects linked against any allowed libraries that also export versioned symbols, they may only depend on the following maximum versions:
GLIBC_2.17
CXXABI_1.3.7, CXXABI_TM_1 is also allowed
GLIBCXX_3.4.19
GCC_4.8.0

We can prepare the cross compilation toolchain to honor the above requirements. However manylinux2014 containers provides gcc9 and we can go with the same.

the cross compilation wheel is almost similar as on x86_64

What exactly do you mean? How are the produced wheels different?

Here I am talking about the time taken to cross compile the wheels. The time taken to cross compile the aarch64 wheel is almost similar as native compilation on x86_64 machine.

still run auditwheel and/or the tests in QEMU

Yes we can use qemu to repair and test the wheel.

So we can't just shove in a cross-compiler and have things work seamlessly? :-/

I will work on it and will share my findings on this.

[odidev]

It needs to be optional, though, as it can take a little work in the package to set up.

It does? So we can't just shove in a cross-compiler and have things work seemlessly? :-/

@YannickJadoul, the code change in the package requires to change the name of the cross compiled artifacts. The code change in cibuildwheel is not going to help us because cibuildwheel calls pip wheel in another container where the environment is different. The easiest way to change the build environment is to modify the setup.py file.

What we can do here is, instead of asking user to modify their setup.py file, we can internally patch the setup.py file within cibuildwheel during cross compilation of the wheels.

Please let me know your thought here?

[YannickJadoul]

The code change in cibuildwheel is not going to help us because cibuildwheel calls pip wheel in another container where the environment is different.

Will it? cibuildwheel has control over the environment and over the images in which a wheel is built, no? But maybe I'm missing something.

What we can do here is, instead of asking user to modify their setup.py file, we can internally patch the setup.py file within cibuildwheel during cross compilation of the wheels.

If that's what needs to happen, then yes, it would be the goal for cibuildwheel to do it :-) (However, "internally patch the setup.py file" sounds very scary to me ;-) )

[odidev]

The code change in cibuildwheel is not going to help us because cibuildwheel calls pip wheel in another container where the environment is different.

Will it? cibuildwheel has control over the environment and over the images in which a wheel is built, no? But maybe I'm missing something.

Basically we need to, override below method to change the naming convention during the cross compilation.

• sysconfig.get_platform
• sysconfig.get_config_vars
• distutils.sysconfig.get_config_vars

I don't see any other way rather than patching setup.py file. Do you have any suggestions on this?

What we can do here is, instead of asking user to modify their setup.py file, we can internally patch the setup.py file within cibuildwheel during cross compilation of the wheels.

If that's what needs to happen, then yes, it would be the goal for cibuildwheel to do it :-) (However, "internally patch the setup.py file" sounds very scary to me ;-) )

The methods which we need to override is related to below module/packages

• import sysconfig
• import distutils
• import copy
• import distutils.sysconfig
  We can parse the setup file and remove the imports if user has already done for any of the above and put our own implementation at start of the setup file.

[Czaki]

Basically we need to, override below method to change the naming convention during the cross compilation.

* sysconfig.get_platform

* sysconfig.get_config_vars

* distutils.sysconfig.get_config_vars

I don't see any other way rather than patching setup.py file. Do you have any suggestions on this?

wrap pip wheel with could be done with the call of pip._internal.cli.main:main? (from setup.py of pip)

[joerick]

Basically we need to, override below method to change the naming convention during the cross compilation.

• sysconfig.get_platform
• sysconfig.get_config_vars
• distutils.sysconfig.get_config_vars

sysconfig.get_platform is the easy one, we already do this on mac to cross-compile by setting the env var _PYTHON_HOST_PLATFORM. See here for the implementation in CPython:
https://github.com/python/cpython/blob/master/Lib/sysconfig.py#L670

Unfortunately the other two don't seem to have easy ways to override. However I do note that sysconfig.get_config_vars could perhaps be customised using a custom _PYTHON_SYSCONFIGDATA_NAME environment variable - see https://github.com/python/cpython/blob/4958f5d69dd2bf86866c43491caf72f774ddec97/Lib/sysconfig.py#L366 . This does seem pretty hacky, but perhaps less hacky than injecting code in setup.py?

[messense]

FYI, I have built manylinux compliant(I think?) docker images for cross compiling to aarch64 and armv7l recently: https://github.com/messense/manylinux-cross

It should be easy to use with crossenv since it has both host and target Python builtin, for exampe: https://github.com/messense/fasttext-wheel/blob/master/manylinux-cross.sh

[joerick]

That's really cool @messense ! Props for figuring out all the fiddly details. I'm curious how generalisable this approach is.

[odidev]

Sorry for the late reply, I was caught up in few other activities. @joerick, thanks for the suggesting to override sysconfig.get_platform, sysconfig.get_config_vars and distutils.sysconfig.get_config_vars methods. I applied the similar logic in the cross compilation image itself and it is working as expected.

Now we can apply this approach in two ways:
1) Manually changing the methods in question for each python versions and replacing the original file with the changed one.
2) Write a script which will apply a patch to distutils/sysconfig.py and sysconfig.py. The patch will be straight forward which is just wrapping up the methods.

Below are the demerits of both the approaches:
Manual approach:
We need to manually do the changes every time a new python version is released.

Script approach:
There is a rare chance that the patch application would fail for newer python release.

Please provide your feedback regarding both the approaches.

[odidev]

@joerick, could you please let me know your thoughts on the above approach? Thanks.

[joerick]

Hi odidev!

This is quite a complex question, as I'm not yet sure how this would factor in to our existing feature set yet. Of the above 2 options, I'm not sure that either really fits into the model of how cibuildwheel works currently. So I'm actually more curious if we could leverage crossenv rather than doing this ourselves, as @messence mentioned? From crossenv's readme:

Cross-python is set up carefully so that it reports all system information exactly as Host-python would. When done correctly, a side effect of this is that distutils and setuptools will cross-compile when building packages. All of the normal packaging machinery still works correctly, so dependencies, ABI tags, and so forth all work as expected.

On a side note, I suppose UI-wise, it would be nice to just set CIBW_CROSS_COMPILE_ARCHS_LINUX=aarch64 ppc64le and have cibuildwheel set all this up. But, I'm not sure that it can be that simple, due to the need for a cross-compilation toolchain - as messence has created using their docker image. But that side of things seems like it could be a lot more complicated, so I'm a little trepidatious!

[odidev]

Hi @joerick, to cross compile the Linux AArch64 wheels, I have done the below changes(using crossenv)-

Enable cross compilation flag:
CIBW_CROSS_COMPILE_ARCHS_LINUX=aarch64

Docker image:
Github: odidev/dockcross@39729cd
Docker-hub: odidev/manylinux2014_aarch64:2021-07-29-39729cd

Cibuildwheel:
Patch in cibuildwheel- odidev@2c8c109

Testing the patches:
Build Logs: https://github.com/odidev/cffi-mirror/runs/3189850544?check_suite_focus=true
GHA workflows: https://github.com/odidev/cffi-mirror/blob/master/.github/workflows/python-publish.yml

Limitations:
During the cross compilation, CIBW_BEFORE_BUILD and CIBW_BEFORE_ALL will only accepts yum/python/pip related commands. Invocation of Makefile or bash script is not allowed because preparing the toolchain based on scripts/Makefile will not be feasible.

@joerick,
If all looks ok, can you please add https://github.com/dockcross/dockcross into https://github.com/pypa so that I can raise a PR in dockcross(your forked) to build the image. Once the image is upload on docker hub, I will raise a PR in cibuildwheel to cross compile the Linux AArch64 wheels.

Please let me know if you have any questions?

Thank You.

[odidev]

Hi @joerick, can you please let me know if above approach/patches looks ok to you?

[henryiii]

I do know we can't add packages to pypa without a vote, and I don't think a fork is a valid thing to put there. But you could try to see if something could be upstreamed, and until then, just require that users manually specify an image if they want to cross-compile. Setting "odidev/manylinux2014_aarch64:2021-07-29-39729cd" isn't that bad. Also, since most of the forking is related to setting up simlinks (IIRC when I checked before), maybe cibuildwheel could do this before building. I'm busy with a few other things (and today's release of Python 3.10rc1 will likely take quite a bit of time in several of my projects), so can't fully review for now.

[odidev]

@henryiii, thanks for your response. User don't need to to set an image for cross compilation and just setting CIBW_CROSS_COMPILE_ARCHS_LINUX=aarch64 will take care of the image selection. So, I am going ahead to raise a PR in cibuildwheel which will use odidev/manylinux2014_aarch64:2021-07-29-39729cd image during cross compilation.

[joerick]

Hi odidev. Apologies for the slow response. There's a lot to go through here and I didn't want to rush a response without taking the time to review it properly.

I'd like to discuss a few things before we get to PRs - this is an enormous jump in complexity of cibuildwheel so it really needs to justify that complexity. cibuildwheel isn't really a "build system" exactly, it's more of a framework that stitches together preexisting tools in the Python packaging ecosystem. cibuildwheel has enough complexity to worry about in presenting a unified API to build wheels! So if it's possible, I'd like this to fit the same approach, to keep the mechanisms of a cross-compile separate to the cibuildwheel project, so that cibuildwheel can interact with that system with a documented and stable API.

Perhaps it would help if you could explain how the pieces of your current solution fit together currently? I see large modifications to linux.py, but also a cross_compile.py, different docker images used in building and testing, a direct use of qemu, use of the /host mount, setup.py bdist_wheel, and in the Docker image, a custom repair script, a Python recompile (?).

(Just to be clear, I think the current state of the cibuildwheel patch is too large to be merged. I wouldn't want to waste any of your time working on this if the problem is fundamentally this complex and we can't reduce it further.)

[odidev]

Hi @joerick,

On a high level, in this scenario, cibuildwheel determines the architectures to cross compile and then setup the environment for them. For this, cibuildwheel creates a docker instance with the image used for cross compilation. Before building the wheel, we would need to install few dependencies (executing before_all) into toolchian for example libffi-devel while building cffi wheels. We can't directly install such dependencies in docker instance because it is x86_64 based and will pull the binaries for same architecture only. To overcome this problem, cibuildwheel runs such commands in a native container and copy back the installed artifacts into the toolchain.

coming to python related commands, crossenv uses two versions of python-
build-python [compatible with build machine]: We need to install all the prerequisites using build-python
python [cross compiled for host machine]: We need to build the wheel from here.

When all the prerequisites are done, cibuildwheel builds the wheel with python setup bdist_wheel as pip wheel is throwing error. Once the wheel is built, cibuldwheel creates another docker instance with targets native container to repair and test the wheel because the same can't be done in x86_64 based contiainer (used for cross compilation)

Modifications are explained here-
linux.py: Calling apis from cross_compile.py to install the dependencies in toolchain and build-python. Also, it triggers a docker instance for targets native container to repair and test the wheel.
cross_compile.py: It has the apis defined to setup the cross-compilation environment.
different docker images used in building and testing: Because we can't test AArch64 wheels in x86_64 based container
a direct use of qemu: Qemu is required to spawn targets native container on x86_64 machine
use of the /host mount: We are building the wheel in a docker container and repairing/testing in another so aarch64 wheels needs to be copied into test container. to achieve this, we need to copy the artifacts on host machine first and then into the test container.
setup.py bdist_wheel: pip wheel is throwing error so let’s use this one for now.
custom repair script: While repairing we need to have all the build dependencies available, and we already have collected these while preparing the toolchian. So, this script will copy already stored artifacts into the test/repair container.
Python recompile: This is crossenv requirement. It requires python for host env and targets env. please check this: https://crossenv.readthedocs.io/en/latest/quickstart.html#

I will explore to minimize the changes, but steps can't be eliminated.

What if I add a file like linux_cross_compile.py and define a build api in that and put all the things there only? Will that be ok for you?

Please let me know if you have any questions?

[joerick]

Thanks @odidev for this response. It's very helpful. And also massive props getting this system working!

I'd like to hear what other maintainers are thinking in regards to this before moving forward. My current feeling is that this might be outside the scope of cibuildwheel, because as the jump in complexity is going to contribute a lot of extra work when maintaining the existing feature set and adding new features.

I wonder therefore, if there would be merit in publishing this build system as a complementary tool e.g. 'crossbuildwheel' tool that's compatible with some of cibuildwheel options¹ and called by users separately, or perhaps, eventually, invoked by cibuildwheel itself to perform a cross-compile.

---

¹ I note that the semantics of CIBW_BEFORE_* options are changed in this system, CIBW_BUILD_FRONTEND isn't used, and the semantics of others are subtly changed by the use of different execution environments.

[odidev]

@joerick, I don't see any response from other maintainers on this thread. Can you please suggest how to proceed on this?

[joerick]

Hmm. This is a tricky one. I can see that cross-compilation of aarch64 would be a valuable feature for users, but I also see lots of complexity here that I don't currently have time to integrate and I'm not sure I want to commit to maintain long-term.

I think perhaps the best route forward would be that your build system could be stood up as a standalone tool (cicrossbuildwheel or something like that), that would be an alternative to cibuildwheel's emulation approach. If options are sufficiently similar, it might not be a huge amount of work for users to run your tool alongside ours. And we could cross-promote your tool in our documentation. Is that something you might be interested in?

[odidev]

@joerick, Thanks for your response and suggestions. If I create tool like cicrossbuildwheel, then will it be possible to add in pypa?

[henryiii]

Sorry for the delay from my side (and it's not over), but I'd like to jot down a few thoughts.

First, the best thing would have been a simple, first-party addition to just make it possible to cross compile with some user configuration. I still have to look over it more carefully, but that's likely not feasible if too much has to be changed and the cost of maintenance grows too much. If done very carefully, there still might be a chance - I'll look over what you've done in the near future. It would have to be general to not be tied to a specific fork, etc.

Assuming that's not possible, then I think the second best option would be to develop an extension API. We could provide specific, documented hooks so that users could write a "plugin" for cibuildwheel that would extend it, in this case, to add cross compilation. Then you could avoid forking (with would be a disaster, you'd have to sync the constant maintenance changes, etc).

If we had a "minimal" set of changes that would make it "possible" (not "pretty") to cross-compile, that might be a good starting point.

Also, to get a repository into the PyPA, a vote has to be taken of all active members, first on the forum, then in a mailing list if it has a current member willing to start the vote. It's not something we can just add. The process is outlined in PEP 609.

[joerick]

Thanks for weighing in, @henryiii !

Assuming that's not possible, then I think the second best option would be to develop an extension API.

The thought also occurred to me. But TBH I wasn't sure what advantage an plugin API would offer over a separate, decoupled standalone tool. I suppose there's options parsing, and perhaps our DockerContainer helper..? The downside of an extension API are of course the burden of maintaining it - ensuing that internal (now external) APIs don't break backwards compatibility is a big one. Also I'd want to be convinced that we might see another few consumers of this API to make that maintenance burden worth it.

Then you could avoid forking (with would be a disaster, you'd have to sync the constant maintenance changes, etc).

Apologies, I didn't intend to propose @odidev's new tool would be a fork of cibuildwheel, although I can see why you assumed that. I would recommend that odidev's crossbuildwheel tool doesn't do any of the mac/windows/conventional manylinux building, leave that to cibuildwheel, but just tackle this one issue of cross compiling manylinux. So users might do their build something like:

- name: Build wheels
  run: |
    pipx run cibuildwheel==2.1.1 --output-dir wheelhouse .
    pipx run crossbuildwheel==0.1.0 --output-dir wheelhouse .

...which would invoke cibuildwheel to do the conventional builds, and then crossbuildwheel to do the aarch64 cross-compilation.

[henryiii]

The downside of an extension API are of course the burden of maintaining it

The benefit is we'd only have to maintain that, not the cross-compiling extension too, though. Specifically, I don't want to be tied to a fork of dockcross, that would be more maintenance than I'd like to commit to.

I suppose there's options parsing, and perhaps our DockerContainer helper..?

Pretty much everything? All the options / configurations / pinning / etc? Basically you'd have to rewrite cibuildwheel except you could skip macOS and Windows. You'd still need the "traditional" emulated linux support as well to run the tests, which would be extra important when cross compiling. Also, would all configuration need to be duplicated under a new prefix for users?

Honestly, the overall size of the patch (odidev@2c8c109) isn't that bad - most of the worry really comes because of the extra requirements of the forked docker image (odidev/dockcross@39729cd). I really want to look through this in the near future and see if we could support just the cibuildwheel patch without taking on the dockcross image.

[henryiii]

Could this be cleanly implemented as a new platform? "crosslinux" as the platform, perhaps? then crosslinux.py would sit beside linux.py, windows.py, and macos.py. This would get rid of a lot of the if statements, etc. above. You would have to specify --platform=crosslinux, since the OS is never detected as "crosslinux", but no new option has to be added.

[henryiii]

(this is also what I was thinking about as a "plugin" system - if an unrecognized platform was looked up through an entrypoint, then most of the logic would sit there, and we'd just have to have a few functions we expect to call and a way to load identifiers, I think).

[joerick]

Could this be cleanly implemented as a new platform? "crosslinux" as the platform, perhaps? then crosslinux.py would sit beside linux.py, windows.py, and macos.py

Yes, I do think this would be the neatest way to do it, if we wanted to pull that functionality into cibuildwheel.

Pretty much everything? All the options / configurations / pinning / etc?

Maybe I'm missing something, but I don't see huge amounts of code... the cross-build wouldn't need pip constraint pins, as it uses the docker container. It uses a different docker container from the manylinux ones, so those pins aren't needed. And any bits of code that are useful, crossbuildwheel could copy from cibuildwheel to get started.

(this is also what I was thinking about as a "plugin" system - if an unrecognized platform was looked up through an entrypoint, then most of the logic would sit there, and we'd just have to have a few functions we expect to call and a way to load identifiers, I think).

Yes, this could work. I'd still be concerned about the resulting API stability requirements, though.

I really want to look through this in the near future and see if we could support just the cibuildwheel patch without taking on the dockcross image.

I'd be interested in what you find out! I know that cross compiling aarch64 is becoming more of a priority for the community, so I would like to support it. So if there's a lower-maintenance way forward, I'm all ears.

Btw, another thing in the back of my mind is - how long until aarch64 CI runners are available? We are starting to see more and ARM servers be deployed in cloud environments, I kinda expect that within 2-3 years ARM will be a more mainstream cloud platform. If so, cross-building is more of a stop-gap than a long-term solution. At the very least we should see Mac ARM64 runners within the next 1-2 years, these machines can run aarch64 Docker images natively. So that is slightly reducing my appetite for big cibuildwheel changes to support this feature.

[odidev]

@henryiii and @joerick, I have implemented a new platform crosslinux and here are the changes: odidev@fcbaef8

Please let me know if this looks OK.

Also, I have pushed changes in dockcross to support cross compiling the wheels. The images has been released on PyPI: dockcross/manylinux2014-aarch64:20210823-db51f29

[henryiii]

This is starting to look very good, especially since official dockcross gained support and we can just build on that. A few questions (you do not need to implement them right now):

• Is this extendable to the other platforms? ppc64le is becoming much more interesting in HPC due to some large installations recently, IIRC. s390x too, though less interesting as far as I know.
• We still have a custom identifier for Python. Is there a way that could be auto-detected? xc36-manylinux_aarch64 is a little irritating, I think I'd rather have cp36-manylinux_aarch64 just work if the platform is crosslinux. Possibly by having a new optional xpath_str argument? Maybe it'd not easier, but I'd think you would not set --arch to aarch64, so cp36-manylinux_aarch64 would not try to emulate-compile, but would only match on the cross compile (--platform=crosslinux).

[odidev]

@henryiii, I will add cross compilation support for ppc64le once cross compilation support for AArch64 is available. I am working on replacing xc36-manylinux_aarch64 with cp36-manylinux_aarch64.

[odidev]

@joerick and @henryiii, I have raise PR:#804
Could you please approve the workflow and review the changes?

[ax3l]

That's awesome! Just curious, could we have a cross_ppc64le, too? Relying on Travis for this #904 / #906 is currently broken.

[odidev]

@ax3l, cross compiler is not following manylinux compliance which is a blocker. So, this activity is on hold as of now.

[ax3l]

Oh I see, that's too bad for now. Thanks for the update.

[nikolaik]

cross compiler is not following manylinux compliance which is a blocker. So, this activity is on hold as of now.

Just curious, how would one test and/or achieve manylinux compliance?

[odidev]

@nikolaik, please check this: #804 (comment)

[jcfr]

I really want to look through this in the near future and see if we could support just the cibuildwheel patch without taking on the dockcross image
[...] especially since official dockcross gained support and we can just build on that

@henryiii It turns out that @thewtex and I are maintaining the dockcross organization, we can definitely discuss approaches to consolidate the manylinux images are reduce overall maintenance effort.