BLD: Unable to build from source on Rocky Linux 8.4 with python 3.10.2

[nadyawilliams]

Describe the issue:

1 OS: Rocky Linux 8.4 (Green Obsidian)
2 Python: 3.10.2 built from source, configured with ./configure --prefix=/opt/apps/python/3.10.2
--enable-shared --enable- optimizations --with-lto --enable-ipv6
--enable-loadable-sqlite-extensions --without-ensurepip
3 setuptools (60.9.3) installed using python 3.10.2
4 cython (0.29.28) installed using python 3.10.2
5 BLAS, LAPACK (3.8.0) , Atlas (3.10.3) - OS provided RPMs with libs and includes in standard /usr/lib64 and /usr/include
6 gcc: OS provided RPMS version 8.5.0 20210514 (Red Hat 8.5.0-4) (GCC)
alternate gcc: version 11.2.0 built from source.

Compiling NumPy-1.22.2 from source fails.

With all above prerequisites being the same
any build with versions ( 1.21.0, 1.21.5, 1.22.0) fails as well, not exactly the same
error as with 1.22.2 but all errors in all versions builds were of the type
AttributeError: 'UnixCCompiler' object has no attribute , for examples for NumPy 1-21.0 build:

File "/export/repositories/python-admix/yamlspecs/numpy-1.21.0/numpy/distutils/command/build_clib.py", line 106, in run
self.compiler.customize(self.distribution,
AttributeError: 'UnixCCompiler' object has no attribute 'customize'

Reproduce the code example:

wget https://github.com/numpy/numpy/archive/v1.22.2.tar.gz
tar xzf v1.22.2.tar.gz
cd numpy-1.22.2/
python setup.py build

Error message:

--- cut ---
creating build/src.linux-x86_64-3.10/numpy/distutils
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Inumpy/core/src/common -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/src/_simd -I/opt/apps/python/3.10.2/include/python3.10 -c _configtest.c -o _configtest.o
gcc -pthread _configtest.o -o _configtest
success!
removing: _configtest.c _configtest.o _configtest.o.d _configtest
Traceback (most recent call last):
  File "/export/repositories/python-admix/yamlspecs/numpy-1.22.2/setup.py", line 461, in <module>
    setup_package()
  File "/export/repositories/python-admix/yamlspecs/numpy-1.22.2/setup.py", line 453, in setup_package
    setup(**metadata)
  File "/export/repositories/python-admix/yamlspecs/numpy-1.22.2/numpy/distutils/core.py", line 169, in setup
    return old_setup(**new_attr)
  File "/opt/apps/python/3.10.2/lib/python3.10/site-packages/setuptools/__init__.py", line 155, in setup
    return distutils.core.setup(**attrs)
  File "/opt/apps/python/3.10.2/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 148, in setup
    return run_commands(dist)
  File "/opt/apps/python/3.10.2/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 163, in run_commands
    dist.run_commands()
  File "/opt/apps/python/3.10.2/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 967, in run_commands
    self.run_command(cmd)
  File "/opt/apps/python/3.10.2/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
    cmd_obj.run()
  File "/export/repositories/python-admix/yamlspecs/numpy-1.22.2/numpy/distutils/command/build.py", line 61, in run
    old_build.run(self)
  File "/opt/apps/python/3.10.2/lib/python3.10/site-packages/setuptools/_distutils/command/build.py", line 135, in run
    self.run_command(cmd_name)
  File "/opt/apps/python/3.10.2/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/opt/apps/python/3.10.2/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
    cmd_obj.run()
  File "/export/repositories/python-admix/yamlspecs/numpy-1.22.2/numpy/distutils/command/build_src.py", line 144, in run
    self.build_sources()
  File "/export/repositories/python-admix/yamlspecs/numpy-1.22.2/numpy/distutils/command/build_src.py", line 155, in build_sources
    self.build_library_sources(*libname_info)
  File "/export/repositories/python-admix/yamlspecs/numpy-1.22.2/numpy/distutils/command/build_src.py", line 288, in build_library_sources
    sources = self.generate_sources(sources, (lib_name, build_info))
  File "/export/repositories/python-admix/yamlspecs/numpy-1.22.2/numpy/distutils/command/build_src.py", line 378, in generate_sources
    source = func(extension, build_dir)
  File "/export/repositories/python-admix/yamlspecs/numpy-1.22.2/numpy/core/setup.py", line 702, in get_mathlib_info
    config_cmd.compiler = bk_c.cxx_compiler()
AttributeError: 'UnixCCompiler' object has no attribute 'cxx_compiler'

NumPy/Python version information:

import sys
print (sys.version)
3.10.2 (main, Feb 22 2022, 17:11:19) [GCC 8.5.0 20210514 (Red Hat 8.5.0-4)]

[charris]

Do you have C++ installed?

[charris]

Looks like it is getting distutils and setuptools from the Python 3.10 install. Did you install them with python3.10 -m pip or just pip or did you install them at all? Only recent NumPy 1.21 or 1.22 supports Python 3.10.

[charris]

Is there a reason you aren't using the manylinux NumPy wheels on pipy?

[nadyawilliams]

Yes, c++ is installed via the same RPM as gcc. Distutils comes native with python 3.10.2 build, Setuptools (60.9.3) and cython (0.29.28) are installed using python 3.10.2, that we build from source and then build all needed python modules also from source. The reason for doing this is we need reproducible software that we install on the clusters to support many scientific applications. Building from source and making an RPM of the build gives us the ability to easily manage software and its dependencies on hundreds of nodes. With all the same conditions that i already mentioned (gcc, setup tools, python, building machine, etc) i can build numpy 1.22.2 if i were to use an old python 3.8.0. Both python 3.8.0 and 3.10.2 are build on the same host with the same compiler and same configuration i already mentioned in original post.

[charris]

My guess is that the problem is setuptools, as that is a moving target. Could you try installing setuptools==59.2.0 for Python3.10? Note that Python distutils is going away in Python 3.12, so we will be moving the build system over the next year.

[nadyawilliams]

I think that using distutils is a problem in conjunction with setuptools. Here are results.

1. compiled and installed setuptools 59.2.0. Can build NumPy, however all 3 scripts in bin/ are missing: f2py, f2py3, f2py3.10
2. went back to using setuptools 60.9.3 (per NumPy source code is it acceptable version). Had to patch numpy-1.22.2/setup.py file and moved "import setuptools" line to be before "import numpy.distutils.command.sdist". Able to compile but also missing all 3 scripts in bin/.

I suspect something similar to what i did with setup.py needs to be done for building f2py scripts but i cant find where exactly.

Here is the info that made me think about setuptools+distutils and making a patch:

[root@admixdev-6 yamlspecs]# python
Python 3.10.2 (main, Feb 22 2022, 17:11:19) [GCC 8.5.0 20210514 (Red Hat 8.5.0-4)] on linux
Type "help", "copyright", "credits" or "license" for more information.

import distutils
:1: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
import setuptools
/opt/apps/python/3.10.2/lib/python3.10/site-packages/_distutils_hack/init.py:17: UserWarning: Distutils was imported before Setuptools, but importing Setuptools also replaces the distutils module in sys.modules. This may lead to undesirable behaviors or errors. To avoid these issues, avoid using distutils directly, ensure that setuptools is installed in the traditional way (e.g. not an editable install), and/or make sure that setuptools is always imported before distutils.
warnings.warn(
/opt/apps/python/3.10.2/lib/python3.10/site-packages/_distutils_hack/init.py:30: UserWarning: Setuptools is replacing distutils.
warnings.warn("Setuptools is replacing distutils.")

If i am to import setuptools and after that import distutils, then there is no warning:
[root@admixdev-6 yamlspecs]# python
Python 3.10.2 (main, Feb 22 2022, 17:11:19) [GCC 8.5.0 20210514 (Red Hat 8.5.0-4)] on linux
Type "help", "copyright", "credits" or "license" for more information.

import setuptools
import distutils

I believe setuptools+distutils (or absence of setuptools?) comes into play for f2py.
Also, even if i build numpy, then when i try to run tests via "python runtests.py -v -m full" the tests fail with:
Writing /export/repositories/python-admix/yamlspecs/numpy-1.22.2/build/testenv/lib/python3.10/site-packages/numpy-1.22.2-py3.10.egg-info
running install_scripts
running build_scripts
error: cannot copy tree 'build/scripts.linux-x86_64-3.10': not a directory

This is again because the 3 scripts are not generated.

[charris]

If moving the order of the two imports helps, we would welcome a PR. I build with 3.10 all the time and haven't had a problem, but I only build using python runtests.py or python setup.py install --local, and the latter works for the f2py scripts. I have no idea what could be the problem with that.

[charris]

Are you building rpms or wheels?

[nadyawilliams]

I am building RPMS

[nadyawilliams]

running python setup.py install --local gives an error:
error: option --local not recognized

What version of pytest are you using? The INSTALL file mentions version 1.15 but this must be a typo. Current version is 7* and even in 2010 the version was 2.0.0.

[charris]

What version of pytest are you using?

Pytest 6.2.5. There are *_requirements.txt files with many of the requirements.

You might try an even earlier version of setuptools. We changed the pin from <49.2.0 to ==59.2.0 for NumPy 1.22, but didn't test building rpms.

[charris]

python setup.py install --local

Darn, I've made that error more than once. It should be --user

[nadyawilliams]

Using --user mainly installs under another dir. Still bin scripts are missing. I am certain it is an interplay of setuptools and distutils. For python 3.8.0 and setuptools v.60.9.3 there is no problem.

[charris]

For python 3.8.0 and setuptools v.60.9.3 there is no problem.

Python 3.10 changed things, that is why I needed to update setuptools to 59.2.0 for my builds. I expect even more changes in 3.11.

Still bin scripts are missing

Mine go into ~/.local/bin/f2py. That is on Fedora 35, I don't know how much that differs from Rocky linux.

[nadyawilliams]

Downgrading setuptools to 49.2.0 worked, thanks for suggestion. So this was definitely distutils+setuptools. I would say a more definitive info is needed when people build from source as to what versions work and what don't. Looks like setuptools has changed a lot. Python 3.10.2 still claims distutils as deprecated but not removed. I suspect the higher version of setuptools will bring even more issues. If i knew where the creation of f2py was handled in bumpy i would attempt to play with setuptools and see if the patch can be created. There are just too many places where numpy.distutils comes into play to blindly go and change as i am not familiar with numpy code inner dependencies.

[charris]

Looks like setuptools has changed a lot.

Yep. Good to know 49.2.0 worked. There were some changes in 3.10 that needed more than < 49.2.0, I should check if 49.2.0 works for me. @rgommers Do you know if meson will build rpms?

There are just too many places where numpy.distutils comes into play to blindly go and change

None of us claim to understand it either :) Changing things in numpy.distutils is like disarming a live bomb. One of the reasons we are looking forward to getting rid of it.

[eli-schwartz]

Do you know if meson will build rpms?

Meson doesn't have a native facility to build rpms (neither does distutils, numpy.distutils, or setuptools). However, as a build system its main job is to compile and install -- and Meson supports all the standard mechanisms that distros expect and RPM uses.

Meson even maintains the official Fedora macros file which RPM specs then use as %meson, %meson_build, and %meson_install.

Writing a spec file is pretty trivial. There is also https://github.com/jordansissel/fpm if you really want a built rpm and don't want to learn how to make one because your objective is to get ahold of numpy rather than submit a Fedora Maintainers application.

[rgommers]

I would say a more definitive info is needed when people build from source as to what versions work and what don't. Looks like setuptools has changed a lot.

I wish we could give that guidance, but we can't. Right now in main, there's a hard setuptool<60.0 limit. So the latest 59.x version seems to work in the vast majority of cases; if there's a problem then downgrading to 49.2.0 (our previous pinned version) is the next-best bet. That's about as much as we can say.

Meson doesn't have a native facility to build rpms (neither does distutils, numpy.distutils, or setuptools).

setuptools kinda does, it has a builtin bdist_rpm command. Completely agreed that is not a good idea though; Meson has the right focus/design here.

@charris each packaging format is basically some flavor of "wrap a built and installed package into some kind of container with metadata". Optionally with some things like post-processing binaries (making it relocatable, vendoring some linked dependencies for wheels, etc.). That should be separate from the build system which does all the heavy lifting. Meson will produce an installed package and the metadata about what was installed, from there the distro-specific tooling will take over. That's also how conda packages get built for example.

[nadyawilliams]

Thank you all for the comments. Building RPMs is not an issue. In any method of RPM building the first thing that happens is building a desired software and then packaging the result into the RPM. I am able to build numpy with setuptools 49.2.0. The question is if i update setuptools in our python installation later on to a higher version will it impact numpy? Any test you can suggest to check this?
A wish: would be nice to synchronize versions requirements in INSTALL.rst.txt and test_requirements.txt.

[adelplanque]

Same bug with python 3.10.2 and setuptools 57.5.0, rocky-linux 8.5

[sandrotosi]

i seeing the same on debian unstable with setuptools 65.3.0 and numpy 1.23.3 (.2 built fine) and Python 3.10.7; didnt have time to read the whole thread in details yet, but just wanted to add this datapoint

[rgommers]

i seeing the same on debian unstable with setuptools 65.3.0

I'll just note that this is explicitly unsupported, 59.x will work better in general.

and numpy 1.23.3 (.2 built fine) and Python 3.10.7

If that's true, that narrows things down to a few commits (see the changelog). The only one that even touches a setup.py file is gh-22215. Are you sure you are using identical environment for 1.23.2 and 1.23.3?

[sandrotosi]

i seeing the same on debian unstable with setuptools 65.3.0

I'll just note that this is explicitly unsupported, 59.x will work better in general.

that's a bit unfortunate, as in Debian build system we dont have a choice to pick any version, only the one that's available globally in the whole distribution, and for setuptools that's 65.3.0

and numpy 1.23.3 (.2 built fine) and Python 3.10.7

If that's true, that narrows things down to a few commits (see the changelog). The only one that even touches a setup.py file is gh-22215. Are you sure you are using identical environment for 1.23.2 and 1.23.3?

yeah i lied :) what i meant is that 1.23.2 built fine the last time i built, but it was not in the same environment as yday build attempt of 1.23.3. When i rebuild .2 in the same env, it fails the same.

Is there any plan to support the newer setuptools and build ecosystem? at the moment, we are not able to build numpy in debian :(

[eli-schwartz]

Is there any plan to support the newer setuptools and build ecosystem? at the moment, we are not able to build numpy in debian :(

My understanding is it's become too hard to adapt to new setuptools changes, so the current build system has reached the end of the line and will effectively see no changes other than for example adding new source files.

Actual improvements to the build system will be exclusively targeted at adopting Meson as a pep517 build backend, and paying any attention to setuptools at all would distract from that.

[sandrotosi]

Is there any plan to support the newer setuptools and build ecosystem? at the moment, we are not able to build numpy in debian :(

My understanding is it's become too hard to adapt to new setuptools changes, so the current build system has reached the end of the line and will effectively see no changes other than for example adding new source files.

Actual improvements to the build system will be exclusively targeted at adopting Meson as a pep517 build backend, and paying any attention to setuptools at all would distract from that.

while i sympatize with the issue of chasing a moving target (python build ecosystem), this resolution essentially prevents Debian from upgrading to 1.23.*, which may not be what the numpy project desires (given the downstream effect it may have).

It looks like 1.21.5 is able to build fine with setuptools 65.3.0, if that helps identify the problem

[sandrotosi]

It looks like 1.21.5 is able to build fine with setuptools 65.3.0, if that helps identify the problem

scratch that, 1.21.5 fails to build too

[eli-schwartz]

I think it's less about the moving target and more about the fact that development-wise, it's untenable and no one knows how to fix it.

A major factor here is that setuptools began forcing its distutils fork instead of the stdlib, and intentionally breaking something that is stable. Literally hoping to find out via bug reports whether anyone actually uses distutils APIs described at docs.python.org.

this resolution essentially prevents Debian from upgrading to 1.23.*, which may not be what the numpy project desires (given the downstream effect it may have).

Debian has one option: it can package "python3-setuptools-59" and build-depend on that instead.

If numpy had the resources or interest in maintaining the current build system so that Debian could avoid packaging old setuptools versions, they wouldn't need to switch build systems anyway. In fact, I seem to recall they tried, but couldn't, so the resolution is less "numpy doesn't desire to build on Debian" and more "numpy tried, failed, and gave up".

On the other hand, if someone has a patch to make it work I bet they wouldn't refuse to merge it... :)

[rgommers]

I think it's less about the moving target and more about the fact that development-wise, it's untenable and no one knows how to fix it.

That's pretty much it, it's untenable. This is fixable of course, but it's hard to estimate how long it will take - last time I had to tackle one of these regressions (admittedly one that was probably more hairy than this one) it took me the better part of a weekend. And that time is better spent on contributing the missing features we need to Meson and moving the whole build over on Meson. We're already going to be cutting it pretty close with being ready for Python 3.12 ...

On the other hand, if someone has a patch to make it work I bet they wouldn't refuse to merge it... :)

Yes, I'll be happy to review and merge a PR.

Debian has one option: it can package "python3-setuptools-59" and build-depend on that instead.

This is actually a very reasonable solution, and less work long-term because it prevents having to deal with future regressions (which are pretty much guaranteed even if we fix this one). NumPy is also not the only package that needs a stable version of setuptools. PyWavelets just added a pin for example: https://github.com/PyWavelets/pywt/blob/master/pyproject.toml#L10. SciPy also pinned to <60.0 before moving away from setuptools: https://github.com/scipy/scipy/blob/maintenance/1.8.x/pyproject.toml#L13

[sandrotosi]

that's an understandable position, since python3-setuptools-59 is not really a viable solution (in Debian's context), i started having a look at what needed to be patched to make this work. I didnt go that far, so i'm wondering if you have any pointers, suggestions, maybe past PRs i can look at to try and make numpy build system compatible with setuptools 65.

thanks!

[rgommers]

Given that the 'UnixCCompiler' object has no attribute 'cxx_compiler' occurs very quickly, the easiest thing to do is probably to bisect to the offending setuptools commit (hopefully it's just one commit). The monkey patch for cxx_compiler is here: https://github.com/numpy/numpy/blob/main/numpy/distutils/ccompiler.py#L704

[merwok]

There is a compiler_cxx attribute on CCompiler classes, but I can’t say if it’s exactly the same thing that numpy expects/defines.

[sandrotosi]

we were able to build numpy/1.23.3 on Debian by exporting SETUPTOOLS_USE_DISTUTILS=stdlib during build

[andyfaff]

I believe this issue can be closed now. Meson is the new build system for numpy, not distutils.