BLD: Build wheels with cibuildwheel 2.12.0

[charris]

The 2.11.4 version of cibuildwheel was failing on 32 bit windows. This is to check that and provide an error log that can be linked to in a cibuildwheel issue.

[henryiii]

What about adding a os.makedirs(os.path.join(sys.prefix, 'libs'), exist_ok=True) right above

numpy/numpy/distutils/mingw32ccompiler.py

Line 441 in 4614245

def_file = os.path.join(sys.prefix, 'libs', def_name)

?

[charris]

What about adding a os.makedirs(os.path.join(sys.prefix, 'libs'), exist_ok=True)

I'm a bit hesitant without knowing why the current code breaks in 2.11.3. I wonder if the meson build also needs to deal with this. @rgommers Thoughts?

[henryiii]

Running in bash was likely a more GNU-like environment than cmd. Meson behaves very differently if you switch between bash & cmd - this is primarily why we changed this default to cmd in cibuildwheel. In bash it was not able to find MSBC, and was stuck with mingw.

You can manually test the last two patch versions with bash by running it via pipx instead of using the action (in a bash run action).

[rgommers]

So the issue is:

FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\runneradmin\\AppData\\Local\\Temp\\cibw-run-oedjqo95\\cp39-win32\\build\\venv\\libs\\python39.def'

I'm a bit hesitant without knowing why the current code breaks in 2.11.3. I wonder if the meson build also needs to deal with this. @rgommers Thoughts?

@henryiii's suggestion seems fine to me - whatever works here I'd say. Meson build: probably not affected by this, since it's choking on something in numpy.distutils and we won't be using that anymore in the near future.

[charris]

@henryiii That didn't fix things, but pushed the failure further on to link time. Looks like the Python library wasn't available.

[henryiii]

Somehow it's not resolving in the non-bash environment correctly. As @joerick points out in the cibuildwheel issue, in #20391, I think this is known to be fragile.

I expect it finds the library outside the venv:

numpy/numpy/distutils/mingw32ccompiler.py

Lines 420 to 436 in 4614245

	if not os.path.isfile(lib_file):
	# didn't find library file in virtualenv, try base distribution, too,
	# and use that instead if found there. for Python 2.7 venvs, the base
	# directory is in attribute real_prefix instead of base_prefix.
	if hasattr(sys, 'base_prefix'):
	base_lib = os.path.join(sys.base_prefix, 'libs', lib_name)
	elif hasattr(sys, 'real_prefix'):
	base_lib = os.path.join(sys.real_prefix, 'libs', lib_name)
	else:
	base_lib = '' # os.path.isfile('') == False
	if os.path.isfile(base_lib):
	lib_file = base_lib
	else:
	log.warn('Cannot build import library: "%s" not found', lib_file)
	return
	log.info('Building import library (ARCH=x86): "%s"', out_file)

Then builds the .def inside the venv. I would have thought it would be valid to have the .def and the file in different directories?

g++ -shared ...
  -LC:\Users\runneradmin\AppData\Local\Temp\cibw-run-koteh10f\cp310-win32\build\venv\libs
  -LC:\Users\runneradmin\AppData\Local\pypa\cibuildwheel\Cache\nuget-cpython\pythonx86.3.10.8\tools\libs
  -LC:\Users\runneradmin\AppData\Local\pypa\cibuildwheel\Cache\nuget-cpython\pythonx86.3.10.8\tools
  -LC:\Users\runneradmin\AppData\Local\Temp\cibw-run-koteh10f\cp310-win32\build\venv\PCbuild\win32
  -Lbuild\temp.win32-3.10
  -lnpymath -lpython310 ...

[charris]

close/reopen to rerun.

[charris]

The macosx failures are new and probably due to a rebase that exposed recent numpy SIMD merges. I suspect they show up here due to a change in compiler versions. @henryiii Has the macosx compiler been updated in cibuildwheel? @seiko2plus Thoughts?

[mattip]

The clang compiler does not recognize the asm() calls from #22954. Note that the CI for that PR ran on azure, this is running on github actions. The image software in this PR is different from the other PR: this one is using clang 14.0.0 where the other is using clang 13.0.0.

Also for some reason the 32-bit windows builds are linking with g++ instead of MSVC.

[mattip]

Thanks @seiko2plus, now all that is left is the win32 failures. Here was the last successful run. I wonder what changed.

[mattip]

The real differences start just after running build_clib. The bad run prefers the mingw 32-bit compiler, while the good run prefers the MSVC compiler.

[mattip]

The 64-bit build uses MSVC in both cases, here is the running build_clib part of the log for the successful 64-bit build

[seiko2plus]

I tried to install python-32bit but still got 64-bit version :

Run pipx run --python "C:\hostedtoolcache\windows\Python\3.11.1\x64\python.exe" --spec "D:\a\_actions\pypa\cibuildwheel\v2.11.3" cibuildwheel "." --output-dir '"wheelhouse"' --config-file '""' --only '""'

[seiko2plus]

the linker wasn't able to identify 32bit of libpython

2023-01-09T11:30:39.9452849Z   C:/rtools40/mingw32/bin/../lib/gcc/i686-w64-mingw32/8.3.0/../../../../i686-w64-mingw32/bin/ld.exe: build\temp.win32-3.9\Release\build\src.win32-3.9\numpy\core\src\multiarray\_multiarray_tests.o:_multiarray_tests.c:(.text+0x6f): undefined reference to `_imp___Py_NoneStruct'
2023-01-09T11:30:39.9463148Z   C:/rtools40/mingw32/bin/../lib/gcc/i686-w64-mingw32/8.3.0/../../../../i686-w64-mingw32/bin/ld.exe: build\temp.win32-3.9\Release\build\src.win32-3.9\numpy\core\src\multiarray\_multiarray_tests.o:_multiarray_tests.c:(.text+0x252): undefined reference to `_imp___Py_NoneStruct'
2023-01-09T11:30:39.9464256Z   C:/rtools40/mingw32/bin/../lib/gcc/i686-w64-mingw32/8.3.0/../../../../i686-w64-mingw32/bin/ld.exe: build\temp.win32-3.9\Release\build\src.win32-3.9\numpy\core\src\multiarray\_multiarray_tests.o:_multiarray_tests.c:(.text+0x2de): undefined reference to `_imp__PyType_IsSubtype'
2023-01-09T11:30:39.9465285Z   C:/rtools40/mingw32/bin/../lib/gcc/i686-w64-mingw32/8.3.0/../../../../i686-w64-mingw32/bin/ld.exe: build\temp.win32-3.9\Release\build\src.win32-3.9\numpy\core\src\multiarray\_multiarray_tests.o:_multiarray_tests.c:(.text+0x2ff): undefined reference to `_imp___Py_NoneStruct'
2023-01-09T11:30:39.9466173Z   C:/rtools40/mingw32/bin/../lib/gcc/i686-w64-mingw32/8.3.0/../../../../i686-w64-mingw32/bin/ld.exe: build\temp.win32-3.9\Release\build\src.win32-3.9\numpy\core\src\multiarray\_multiarray_tests.o:_multiarray_tests.c:(.text+0x369): undefined reference to `_imp__PyExc_TypeError'
2023-01-09T11:30:39.9467149Z   C:/rtools40/mingw32/bin/../lib/gcc/i686-w64-mingw32/8.3.0/../../../../i686-w64-mingw32/bin/ld.exe: build\temp.win32-3.9\Release\build\src.win32-3.9\numpy\core\src\multiarray\_multiarray_tests.o:_multiarray_tests.c:(.text+0x374): undefined reference to `_imp__PyErr_SetString'
2023-01-09T11:30:39.9468031Z   C:/rtools40/mingw32/bin/../lib/gcc/i686-w64-mingw32/8.3.0/../../../../i686-w64-mingw32/bin/ld.exe: build\temp.win32-3.9\Release\build\src.win32-3.9\numpy\core\src\multiarray\_multiarray_tests.o:_multiarray_tests.c:(.text+0x3b5): undefined reference to `_imp___PyArg_ParseTuple_SizeT'
2023-01-09T11:30:39.9468927Z   C:/rtools40/mingw32/bin/../lib/gcc/i686-w64-mingw32/8.3.0/../../../../i686-w64-mingw32/bin/ld.exe: build\temp.win32-3.9\Release\build\src.win32-3.9\numpy\core\src\multiarray\_multiarray_tests.o:_multiarray_tests.c:(.text+0x3c3): undefined reference to `_imp__PyFloat_Type'
2023-01-09T11:30:39.9471250Z   C:/rtools40/mingw32/bin/../lib/gcc/i686-w64-mingw32/8.3.0/../../../../i686-w64-mingw32/bin/ld.exe: build\temp.win32-3.9\Release\build\src.win32-3.9\numpy\core\src\multiarray\_multiarray_tests.o:_multiarray_tests.c:(.text+0x3d4): undefined reference to `_imp__PyType_IsSubtype'
2023-01-09T11:30:39.9481568Z   C:/rtools40/mingw32/bin/../lib/gcc/i686-w64-mingw32/8.3.0/../../../../i686-w64-mingw32/bin/ld.exe: build\temp.win32-3.9\Release\build\src.win32-3.9\numpy\core\src\multiarray\_multiarray_tests.o:_multiarray_tests.c:(.text+0x3e7): undefined reference to `_imp__PyComplex_Type'
2023-01-09T11:30:39.9482597Z   C:/rtools40/mingw32/bin/../lib/gcc/i686-w64-mingw32/8.3.0/../../../../i686-w64-mingw32/bin/ld.exe: build\temp.win32-3.9\Release\build\src.win32-3.9\numpy\core\src\multiarray\_multiarray_tests.o:_multiarray_tests.c:(.text+0x408): undefined reference to `_imp__PyBool_Type'
2023-01-09T11:30:39.9483479Z   C:/rtools40/mingw32/bin/../lib/gcc/i686-w64-mingw32/8.3.0/../../../../i686-w64-mingw32/bin/ld.exe: build\temp.win32-3.9\Release\build\src.win32-3.9\numpy\core\src\multiarray\_multiarray_tests.o:_multiarray_tests.c:(.text+0x421):

[mattip]

Did you convince the build process to use MSVC and not g++?

[seiko2plus]

Did you convince the build process to use MSVC and not g++?

passed after uses MSVC

[charris]

@henryiii Updating to 2.11.4 brings back the pp38 segfault on manylinux2014.

[mattip]

That was fixed in PyPy issue 3870 which was part of the PyPy v7.3.11 release (cibuildwheel 2.11.4 uses v7.3.10). There is a PR to upgrade PyPy in pypa/cibuildwheel#1372

[charris]

@mattip Thanks for the info. Looks like the next step is to wait for cibuildwheel 2.11.5 and upgrade the version again. Or we could put this in with 2.11.3, but there is no hurry.

[charris]

Updated to use cibuildwheel 2.12.0 released four hours ago.

[charris]

Self merging so work on top of this can proceed. @mattip You should be able to use pp39 now.