New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix cross-compiling by searching the right lib and include directories #7634
base: main
Are you sure you want to change the base?
Conversation
8b7490d
to
c8d257e
Compare
I slept on it and changed my mind about virtualenvs. I think dependencies may be installed there, so I have continued to support that case, even though it is incompatible with cross-compiling. |
We were previously searching the `{sys.prefix}/lib` and `{sys.prefix}/include` directories unconditionally. This is problematic when cross-compiling, as it does not take account of any sysroot where alternative libraries and headers are located. Adding `-I/usr/include` causes the build to explode, at least when cross-compiling from 64-bit to 32-bit. Python does not officially support cross-compiling, but Gentoo achieves this by modifying the sysconfig variables like `LIBDIR` and `INCLUDEDIR` with great results. Assuming "lib" is bad. 64-bit Linux systems often use lib64, putting 32-bit libraries under lib. You cannot assume that either though, as pure 64-bit Linux systems may just use lib instead. Things get even stranger on RISC-V. The value of `sys.prefix` changes when using a virtualenv. Dependencies may be installed here, so it does make sense to continue supporting this case, even if it is incompatible with cross-compiling. Unlike regular environments, "lib" is generally used for libraries, although a lib64 symlink may also be present.
I don't suppose it would be possible for you to create a Dockerfile that demonstrates the need for this? |
I did try very hard, but it's difficult to demonstrate on other distros I'm familiar with. I could use Gentoo, but then it might just look like Gentoo is doing something weird. Debian and friends avoid the issue with their multilib approach to cross-compiling. Fedora doesn't have a formal cross-compile mechanism, but you can usually make it work by setting a few flags. Unfortunately, it has some quirks that make it fall over in this case, and while I could work around them, I think that would just distract from the issue at hand. It's better to focus on the command that leads to the errors.
The above is truncated. It actually spews out pages and pages of this stuff. 64-bit x86_64 headers from /usr/include, particularly those under /usr/include/bits, massively break 32-bit targets like armv7a. Simply taking out |
Fix cross-compiling by searching the right lib and include directories
We were previously searching the
{sys.prefix}/lib
and{sys.prefix}/include
directories unconditionally. This is problematic when cross-compiling, as it does not take account of any sysroot where alternative libraries and headers are located. Adding-I/usr/include
causes the build to explode, at least when cross-compiling from 64-bit to 32-bit.Python does not officially support cross-compiling, but Gentoo achieves this by modifying the sysconfig variables like
LIBDIR
andINCLUDEDIR
with great results.Assuming "lib" is bad. 64-bit Linux systems often use lib64, putting 32-bit libraries under lib. You cannot assume that either though, as pure 64-bit Linux systems may just use lib instead. Things get even stranger on RISC-V.
The value of
sys.prefix
changes when using a virtualenv. Dependencies may be installed here, so it does make sense to continue supporting this case, even if it is incompatible with cross-compiling. Unlike regular environments, "lib" is generally used for libraries, although a lib64 symlink may also be present.