CI/BLD: use scipy-openblas wheel when building

[mattip]

Replaces #20074. Uses the scipy-openblas32 wheel for building Scipy

The first commit changes linux wheel builds to use the scipy-openblas32 wheel

Reference issue

What does this implement/fix?

• Add a new cibw_before_build.sh script
• Remove the OpenBLAS version check from tests

Additional information

cc @rgommers

[andyfaff]

This line will override any config specified in [tool.cibuildwheel] (i.e. the new file you add won't get called). Ditto the specific windows file.

For me it makes sense to separate configuration into separate cibw_before_build_platform files, a single file can get convoluted.

[andyfaff]

Perhaps there could be a common file, and individual files in this line, i.e.

before-build = "bash {project}/tools/wheels/cibw_before_build_common.sh; bash {project}/tools/wheels/cibw_before_build_macos.sh {project}"

[mattip]

This line will override any config specified in [tool.cibuildwheel] (i.e. the new file you add won't get called).

At this time, I only want to override the linux wheel building to prove the wheels work. If it works, we can think about how to generalize it in another PR/other commits.

[andyfaff]

Ok. The changes you make here are also overridden in wheels.yml

[mattip]

Oy. I will try removing the redundant line from wheels.yml.

[andyfaff]

Having a duplicate is sometimes useful. e.g. if you want to develop locally you want things specified in pyproject.toml. If you want a fixed config for scipy/scipy to build wheels for public consumption you may want to do that in wheels.yml.

[mattip]

For me it makes sense to separate configuration into separate ...

OK, adopted in the latest commit

[mattip]

Having a duplicate is sometimes useful.

Hmm. On the other hand, it is a bit obscure and makes replicating the wheel build locally more difficult...

[mattip]

Maybe we could document the ability to override the pyproject.toml values with local environment variables, and local specialized builds would do that

[andyfaff]

Sure. The main reason to do so is e.g. employ specific package versions, etc.

[mattip]

The scipy_ BLAS_SYMBOL_PREFIX is not getting prepended to srotg. I see that scpiy-openblas 0.3.27 is found, which should result in using the cflags from the pkgconfig file. I wonder where that is getting lost.

[rgommers]

This need for LD_LIBRARY_PATH is quite annoying. @ev-br pinged me about an issue with this as well yesterday. I haven't been able to reproduce it myself, but my conclusion was that Libs: -L{libdir} -lscipy_openblas in the generated .pc file is not reliable since it depends on the loader's library search path configuration as well as whether site-packages is under sysroot or not, and it should be replaced with /abspath/to/scipy_openblas.so to ensure the built extension modules will always find scipy_openblas.so at runtime.

All of the boilerplate of copying shared libraries around should be reduceable to:

$ python -c "import scipy_openblas32; print(scipy_openblas32.get_pkg_config())" > scipy-openblas.pc
$ export PKG_CONFIG_PATH=$PWD

Does that sound right to you @mattip?

[ev-br]

For LD_LIBRARY_PATH, my band-aid fix was to copy the .so file from build/ to build-install:

https://github.com/ev-br/openblas-bench/blob/main/.github/workflows/codspeed.yml#L42

Still puzzling why the two libs have different linkage.

[mattip]

LD_LIBRARY_PATH is needed at runtime, not build time. The linker arguments in the pkg_config file are for build time, not runtime. In a proper wheel, delocate/auditwheel are used so LD_LIBRARY_PATH is no longer needed. But in the CI workflows, we need to either copy the dependencies into somwhere like /usr/local/lib, which is what the current scripts do, or use LD_LIBRARY_PATH

[rgommers]

I know it's a runtime thing, I am saying that it should not be needed. It is bad for a build to succeed and then fail at runtime.

In a proper wheel, delocate/auditwheel are used

We have a goal of wanting it to work without vendoring. Also, folks are trying to use the scipy-openblas32/64 locally. And we want the cibuildwheel config to be as simple as possible. That's three reasons to make this work without LD_LIBRARY_PATH.

I don't think it's hard. It should only require passing an absolute path rather than -L/-l, as I said above. We should be able to do this in scipy_openblas32.get_pkg_config(), right? It knows exactly where the shared library is located, so writing out the .pc file containing the correct absolute path seems like only a small tweak.

[mattip]

writing out the .pc file containing the correct absolute path…

Makes sense. I can try this next week

[mattip]

it should be replaced with /abspath/to/scipy_openblas.so to ensure the built extension modules will always find scipy_openblas.so at runtime

That seems to work on linux. How should this work on macos? Linking with the absolute path does not preserve the path for runtime, nor does using -Wl,-rpath,/abspath/to properly add an RPATH to the shared object (at least as far as I can tell).

[rgommers]

nor does using -Wl,-rpath,/abspath/

This is what conda-forge compilers use on macOS, so I expect it to work:

% echo $LDFLAGS
-Wl,-headerpad_max_install_names -Wl,-dead_strip_dylibs -Wl,-rpath,/Users/rgommers/mambaforge/envs/scipy-dev/lib -L/Users/rgommers/mambaforge/envs/scipy-dev/lib

It would be surprising if it works when given via LDFLAGS (so for all targets) but not if it's only in the list of link args for the targets where it's needed. Are you sure it doesn't work? If so, does using LDFLAGS instead work?

[mattip]

The -Wl,-rpath,<path> part is needed in the Libs: line of scipy-openblas.pc to set the rpath search path.
It doesn't matter whether I use the full/path/to/libscipy_openblas.dylib or -L/path -lscipy_openblas, the compiled linalg shared objects still only have a dependency on libscipy_openblas.dylib

The missing part is that I need to do this in order to "burn" the @rpath prefix onto the libscipy_openblas in the LC_LOAD_DYLIB stanza of the dylib header in all the shared objects that use scipy_openblas:

install_name_tool -change libscipy_openblas.dylib @rpath/libscipy_openblas.dylib <path-to>/linalg/shared-object.cpython-311-darwin.so

I am still trying to figure out how to set the @rpath prefix from the linker without install_name_tool.

[rgommers]

The -Wl,-rpath addition is correct. The problem is with libscipy_openblas.dylib. Its install name (LC_ID_DYLIB) should include @rpath. You can check with:

% otool -D libscipy_openblas.dylib
libscipy_openblas.dylib:
libscipy_openblas.dylib

or with:

% otool -l libscipy_openblas64_.dylib | grep -A 2 LC_LOAD_DYLIB

To test that, you can fix it locally like this:

% install_name_tool -id @rpath/libscipy_openblas.dylib libscipy_openblas.dylib
% codesign --force -s - libscipy_openblas.dylib

After that, the install name is correct:

% otool -l libscipy_openblas.dylib | grep -A 2 LC_ID_DYLIB
          cmd LC_ID_DYLIB
      cmdsize 56
         name @rpath/libscipy_openblas.dylib (offset 24)

and adding -Wl,-rpath,${libdir} to the .pc file will work as expected.

You can also try checking install names of for example the <prefix>/lib directories of conda-forge or Homebrew. For conda-forge, all dylibs start with @rpath. For Homebrew, they're all absolute paths. Both work - but a plain libscipy_openblas.dylib will not. The conda-forge approach is preferred (see, e.g., https://forums.developer.apple.com/forums/thread/736719 - generally a very useful article).

tl;dr rebuilding scipy-openblas with a change to the install name of the shared library on macOS, as well as including -Wl,-rpath,${libdir} for all platforms except Windows, should fix this.