BLD: default to Meson in pyproject.toml

[rgommers]

This switches to Meson in pyproject.toml (see gh-13615), by using the meson-python build backend. There's an important conceptual change: pyproject.toml file is now by default usable for development and does not deal with pinning numpy specifically for PyPI - that is now done in a separate script.

This is an update/alternative to gh-15476, which wasn't going to quite work as planned (as in this comment). Given how the build hooks in pyproject.toml work, it is fundamentally not possible to have both the development/CI/etc. config in there and the PyPI-specific pins. Having a separate script for generating our PyPI artifacts isn't a problem, because this should only be used for releases in well-defined CI configs. One goal I have with this new PyPI-specific script though is to make it much easier to build such wheels locally for testing; right now that's basically impossible, given the many hoops that https://github.com/MacPython/scipy-wheels jumps through.

Still TODO:

• update release procedure docs
• implement PyPI-specific wheel build changes EDIT: no longer needed because meson-python now picks up uncommitted changes. So we should be good with what multibuild does for the distutils-based builds.
• add 1.9.0 release notes
• add user/packager-facing docs on using pip, build, etc. no change here in behaviour, better guidance to users about build isolation would be nice, but is decoupled from this PR.
• implement the change to ensure wheels have the correct runtime dependency on numpy (>= build-time version): gh-14541#issuecomment-968317738. EDIT: doing this in a follow-up, since the meson-python feature won't be available in time. That's not an issue, because this behavior was never available with our numpy.distutils based builds.

There is still work to do, but the sdist generation works and I'd like some feedback now on the approach.

Cc @FFY00, @matthew-brett, @tylerjereddy, @eli-schwartz, @mattip

[rgommers]

As the current version of the build-pypi-artifacts.py script shows, it is cumbersome to apply patches - because you have to commit the result for it to be picked up. That has other side-effects, like ending up with a git commit hash in scipy/version.py that doesn't exist in the main repo.

I thought about it some more, and this isn't just going to be painful for building PyPI artifacts, it's a problem in general:

• The typical workflow for any packager is: (1) apply patches, (2) build with pip install . or python -m build (with appropriate flags like --no-build-isolation etc. if needed), (3) repackage into native package format.
• For users/contributors and local development, pip install . should pick up local changes they made; that's how it works today and anything else is going to be very confusing.

The above is true not just for SciPy, but for any complex package. So why do meson dist, meson-python and flit rely on extracting dist/sdist from committed changes only? The main reason is to avoid producing unreproducible/incorrect tarballs at release time. A good goal, but also very easy to avoid when you build your release artifacts in CI (which serious projects should do anyway).

So, tentative conclusion 1: meson-python should include changes to files tracked in git but not committed, that's better all around.

The second part of this is: what should the SciPy sdist on PyPI look like? Should it mirror as closely as possible what's in the corresponding release tag in the repo (e..g, v1.9.0), or should it include the changes (as far as possible, 100% is not possible) we need to build the official SciPy wheels that we upload on PyPI? Right now what we do is a mix:

• for numpy== pins we use the ones we use for wheel builds
• we leave scipy/_distributor_init.py unmodified (while wheels have this patched version - and that contains more than just DLL vendoring tweaks)

It's not at all obvious that that's the right thing to do. Sdist's have multiple purposes, and the numpy== pins are wrong for most of them. I'm leaning towards it being better to not put any PyPI-specific patches in the sdist. That would keep pip install <sdist> usable for everyone, and not make it so hard to even produce sdists (see again the temporary commit in this PR). The current downside of that is that it's possible for users to build multiple times from the PyPI-uploaded sdist in different envs with different numpy versions, and get different results - which will result in an issue in combination with pip's caching. This can be fixed with the numpy='pin-compatible' idea in https://github.com/FFY00/meson-python/issues/29.

So, tentative conclusion 2: SciPy sdists on PyPI should no longer contain numpy== pins, to keep sdists usable with pip install for all packaging systems.

Thoughts?

[tylerjereddy]

I didn't get to the second set of detailed questions yet, but my comments/concerns are perhaps the most obvious ones:

1. These adjustments to pyproject.toml hardly seem to fit with its original purpose, it would be great if the need for developer flexibility could simply be enforced with a build flag similar to --no-build-isolation--this is definitely not possible? And we can't accomplish a meson/mesonpy override with a [tool] type field in pyproject.toml instead of purging out our pins?
2. The relationship to the ongoing work in MAINT, BLD: cibuildwheel Linux start #15925 with cibuildwheel isn't clear to me--there seem to be some areas that overlap, and the desired incantation for producing release artifacts is getting confusing for me now (which one is the "driver?" I would have thought we'd aim for cibuildwheel as the driver, but not sure on the sdist part...)

[tylerjereddy]

I guess it may be good to clarify the relationship with cibuildwheel here. I was kind of hoping I'd be able to swap the build backend and method of providing OpenBLAS/gfortran stuff and then be ready to swap, but perhaps that's a bit optimistic.

Having two potential/custom routes to PyPI wheel generation is confusing to me though, so we should sort it out.

[rgommers]

There will be only one way. Look at this like this:

1. cibuildwheel (or the current scipy-wheels repo based on multibuild) are meant to (a) set up the environments needed build wheels for PyPI, and (b) provide a means to trigger those builds.
2. Both cibuildwheel and multibuild need to run one single command, e.g. pip wheel or python -m build, to actually build the wheel after all of the environment setup (and maybe applying some patches). Currently that is python setup.py bdist_wheel. We need a replacement for that - which has to come from this PR.

[rgommers]

That said, I would like to have more of the logic of what is now in scipy-wheels/cibuildwheel CI config files to live in scripts, so that it can be tested and debugged locally rather than only in CI itself. That's always a good thing.

[tylerjereddy]

pin-compatible is going to be a PyPI package or something?

[rgommers]

No, it should be a string that meson-python dynamically replaces to >=<version-at-build_time>. So if you build against numpy 1.18.5, this will say >=1.18.5 (now our lowest-supported version), if you build against 1.22.1, then >=1.22.1, etc. This will solve a long-standing issue where our install_requires for numpy is simply wrong when users build wheels, or even when we build wheels for PyPI where due to Python or platform version support the numpy version we build against is not the lowest-supported one.

The actual syntax of this still needs to be finalized, see mesonbuild/meson-python#29.

[eli-schwartz]

That sounds a bit like what you'd get from using setup_requires with unversioned numpy, and then an install_requires that is dynamically calculated using homebrew functions in setup.py based on whichever version got installed... which setuptools may not have ever supported because you'd need to calculate that after the build requirements got installed and calculated. At least it would definitely work if you installed numpy before running ye olde python3 setup.py install.

I can see why you'd want proper support for that in the build backend, lol.

[rgommers]

1. These adjustments to pyproject.toml hardly seem to fit with its original purpose, it would be great if the need for developer flexibility could simply be enforced with a build flag similar to --no-build-isolation--this is definitely not possible? And we can't accomplish a meson/mesonpy override with a [tool] type field in pyproject.toml instead of purging out our pins?

Good question, let me write a (long) problem statement and distutils/meson comparison later today. That will make the problems we're running into clearer.

Re "its original purpose," that is part of the problem - there isn't just a single purpose, there are multiple - and it's not even written down clearly which ones those are. It's mostly a handwavy "allow tools like pip to decouple from distutils and call another build system".

2. The relationship to the ongoing work in gh-15925 with cibuildwheel isn't clear to me--there seem to be some areas that overlap, and the desired incantation for producing release artifacts is getting confusing for me now (which one is the "driver?" I would have thought we'd aim for cibuildwheel as the driver, but not sure on the sdist part...)

Let me repeat here what I just wrote in an inline comment:

1. cibuildwheel (or the current scipy-wheels repo based on multibuild) are meant to (a) set up the environments needed build wheels for PyPI, and (b) provide a means to trigger those builds.
2. Both cibuildwheel and multibuild need to run one single command, e.g. pip wheel or python -m build, to actually build the wheel after all of the environment setup (and maybe applying some patches). Currently that is python setup.py bdist_wheel. We need a replacement for that - which has to come from this PR.

[rgommers]

This is the state with SciPy's current main branch at commit 7076a90 for the setup.py behavior, main plus only adding a meson-python build hook without changing any of the pins for Meson, and the following library versions:

• python: 3.10.2
• numpy: 1.22.3
• pip: 22.0.4
• setuptools: 59.8.0
• meson: 0.62.1
• meson-python: 0.4.0
• build: 0.7.0

For completeness, this is the diff of the one commit on top of main for testing with meson-python:

diff --git a/pyproject.toml b/pyproject.toml
index 1b44bc54e..f5c49aab6 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -8,7 +8,9 @@
 #     "pybind11>=2.4.3,<2.5.0",
 
 [build-system]
+build-backend = 'mesonpy'
 requires = [
+    "meson-python>=0.4.0",
     "wheel",
     "setuptools<60.0",  # do not increase, 60.0 enables vendored distutils
     "Cython>=0.29.18",

Note that I do git clean -xdf after each individual test, and am omitting the (often very verbose) terminal output.

Behavior of sdist/wheel build and install commands

Generating an sdist

With setup.py:

$ python setup.py sdist
$ ls dist
scipy-1.9.0.dev0+2035.7076a90.tar.gz

$ python -m build --sdist
* Creating venv isolated environment...
* Installing packages in isolated environment... (Cython>=0.29.18, numpy; python_version>='3.11', numpy; python_version>='3.8' and platform_python_implementation=='PyPy', numpy==1.18.5; python_version=='3.8' and (platform_machine!='arm64' or platform_system!='Darwin') and platform_machine!='aarch64' and platform_python_implementation != 'PyPy', numpy==1.19.2; python_version=='3.8' and platform_machine=='aarch64' and platform_python_implementation != 'PyPy', numpy==1.19.3; python_version=='3.9' and (platform_machine!='arm64' or platform_system!='Darwin') and platform_python_implementation != 'PyPy', numpy==1.20.0; python_version=='3.8' and platform_machine=='arm64' and platform_system=='Darwin', numpy==1.20.0; python_version=='3.9' and platform_machine=='arm64' and platform_system=='Darwin', numpy==1.21.4; python_version=='3.10' and platform_python_implementation != 'PyPy', pybind11>=2.4.3, pythran>=0.9.12, setuptools<60.0, wheel)
* Getting dependencies for sdist...
running egg_info
...
$ ls dist
scipy-1.9.0.dev0+2035.7076a90.tar.gz

$ python -m build --sdist --no-isolation
ERROR Missing dependencies:
        numpy==1.21.4; python_version=='3.10' and platform_python_implementation != 'PyPy'

Note that the above is actually a difference between pip and build. The latter has a cleaner model imho: --no-isolation simply turns off build isolation, but still honors/checks the specified dependencies. With pip it's more like "don't use build isolation and ignore the specified dependencies in pyproject.toml completely.

$ python -m build --sdist --no-isolation --skip-dependency-check
* Building sdist..
$ ls dist
scipy-1.9.0.dev0+2035.7076a90.tar.gz

Conclusion for setup.py: we have 3 ways of generating an sdist, directly with setup.py sdist, and with python -m build with and without isolation. Those methods all result in the same sdist, provided that you have reasonable versions of numpy and setuptools installed. They all have:

• checks that git submodules are present,
• concatenated license files,
• a file name containing the dynamically generated version; the part after dev0+ contains (a) an incrementing number for "commits since last git tag, and (b) the first 7 characters of the current git commit hash
• files that do contain uncommitted changes, if such changes were made in the repo

The default python -m build --sdist version is the most reproducible (since it fixes versions of numpy/setuptools), but it's also the slowest and it does need network access to download tens of MBs worth of wheels (something that isn't always desirable or plain not allowed).

With meson-python:

$ python -m build --sdist
* Creating venv isolated environment...
* Installing packages in isolated environment... (Cython>=0.29.18, meson-python>=0.4.0, numpy; python_version>='3.11', numpy; python_version>='3.8' and platform_python_implementation=='PyPy', numpy==1.18.5; python_version=='3.8' and (platform_machine!='arm64' or platform_system!='Darwin') and platform_machine!='aarch64' and platform_python_implementation != 'PyPy', numpy==1.19.2; python_version=='3.8' and platform_machine=='aarch64' and platform_python_implementation != 'PyPy', numpy==1.19.3; python_version=='3.9' and (platform_machine!='arm64' or platform_system!='Darwin') and platform_python_implementation != 'PyPy', numpy==1.20.0; python_version=='3.8' and platform_machine=='arm64' and platform_system=='Darwin', numpy==1.20.0; python_version=='3.9' and platform_machine=='arm64' and platform_system=='Darwin', numpy==1.21.4; python_version=='3.10' and platform_python_implementation != 'PyPy', pybind11>=2.4.3, pythran>=0.9.12, setuptools<60.0, wheel)
* Getting dependencies for sdist...
+ meson setup --native-file=/home/rgommers/code/bldscipy/.mesonpy-native-file.ini -Ddebug=false -Dstrip=true -Doptimization=2 --prefix=/home/rgommers/anaconda3/envs/scipy-dev /home/rgommers/code/bldscipy /home/rgommers/code/bldscipy/.mesonpy-0riomdni/build
...
* Installing packages in isolated environment... (pep621 >= 0.3.0)
* Building sdist...
+ meson setup --native-file=/home/rgommers/code/bldscipy/.mesonpy-native-file.ini -Ddebug=false -Dstrip=true -Doptimization=2 --prefix=/home/rgommers/anaconda3/envs/scipy-dev /home/rgommers/code/bldscipy /home/rgommers/code/bldscipy/.mesonpy-48mc7pbv/build
...
+ meson dist --no-tests --formats gztar
$ ls dist
scipy-1.9.0.dev0.tar.gz

Full output of the above (only irrelevant Cython-related warnings omitted):

$ python -m build --sdist
* Creating venv isolated environment...
* Installing packages in isolated environment... (Cython>=0.29.18, meson-python>=0.4.0, numpy; python_version>='3.11', numpy; python_version>='3.8' and platform_python_implementation=='PyPy', numpy==1.18.5; python_version=='3.8' and (platform_machine!='arm64' or platform_system!='Darwin') and platform_machine!='aarch64' and platform_python_implementation != 'PyPy', numpy==1.19.2; python_version=='3.8' and platform_machine=='aarch64' and platform_python_implementation != 'PyPy', numpy==1.19.3; python_version=='3.9' and (platform_machine!='arm64' or platform_system!='Darwin') and platform_python_implementation != 'PyPy', numpy==1.20.0; python_version=='3.8' and platform_machine=='arm64' and platform_system=='Darwin', numpy==1.20.0; python_version=='3.9' and platform_machine=='arm64' and platform_system=='Darwin', numpy==1.21.4; python_version=='3.10' and platform_python_implementation != 'PyPy', pybind11>=2.4.3, pythran>=0.9.12, setuptools<60.0, wheel)
* Getting dependencies for sdist...
+ meson setup --native-file=/home/rgommers/code/bldscipy/.mesonpy-native-file.ini -Ddebug=false -Dstrip=true -Doptimization=2 --prefix=/home/rgommers/anaconda3/envs/scipy-dev /home/rgommers/code/bldscipy /home/rgommers/code/bldscipy/.mesonpy-0riomdni/build
The Meson build system
Version: 0.62.1
Source dir: /home/rgommers/code/bldscipy
Build dir: /home/rgommers/code/bldscipy/.mesonpy-0riomdni/build
Build type: native build
Project name: SciPy
Project version: 1.9.0.dev0
C compiler for the host machine: /home/rgommers/anaconda3/envs/scipy-dev/bin/x86_64-conda-linux-gnu-cc (gcc 9.4.0 "x86_64-conda-linux-gnu-cc (GCC) 9.4.0")
C linker for the host machine: /home/rgommers/anaconda3/envs/scipy-dev/bin/x86_64-conda-linux-gnu-cc ld.bfd 2.36.1
C++ compiler for the host machine: /home/rgommers/anaconda3/envs/scipy-dev/bin/x86_64-conda-linux-gnu-c++ (gcc 9.4.0 "x86_64-conda-linux-gnu-c++ (GCC) 9.4.0")
C++ linker for the host machine: /home/rgommers/anaconda3/envs/scipy-dev/bin/x86_64-conda-linux-gnu-c++ ld.bfd 2.36.1
Host machine cpu family: x86_64
Host machine cpu: x86_64
Compiler for C supports arguments -Wno-unused-but-set-variable: YES 
Library m found: YES
Fortran compiler for the host machine: /home/rgommers/anaconda3/envs/scipy-dev/bin/x86_64-conda-linux-gnu-gfortran (gcc 9.4.0 "GNU Fortran (GCC) 9.4.0")
Fortran linker for the host machine: /home/rgommers/anaconda3/envs/scipy-dev/bin/x86_64-conda-linux-gnu-gfortran ld.bfd 2.36.1
Program cython found: YES (/tmp/build-env-pfwujcvb/bin/cython)
Program pythran found: YES (/tmp/build-env-pfwujcvb/bin/pythran)
Program cp found: YES (/usr/bin/cp)
Program python3 found: YES (/tmp/build-env-pfwujcvb/bin/python)
Found pkg-config: /home/rgommers/anaconda3/envs/scipy-dev/bin/pkg-config (0.29.2)
Library npymath found: YES
Library npyrandom found: YES
Run-time dependency openblas found: YES 0.3.18
Dependency openblas found: YES 0.3.18 (cached)
Program _build_utils/cythoner.py found: YES (/tmp/build-env-pfwujcvb/bin/python /home/rgommers/code/bldscipy/scipy/_build_utils/cythoner.py)
Checking for function "open_memstream" : NO 
Configuring messagestream_config.h using configuration
Checking for size of "void*" : 8
Run-time dependency threads found: YES
Checking for size of "void*" : 8
Dependency threads found: YES unknown (cached)
Compiler for C supports arguments -Wno-unused-but-set-variable: YES (cached)
Compiler for C supports arguments -Wno-unused-but-set-variable: YES (cached)
Compiler for C++ supports arguments -Wno-format-truncation: YES 
Compiler for C++ supports arguments -Wno-class-memaccess: YES 
Build targets in project: 199

SciPy 1.9.0.dev0

  User defined options
    Native files: /home/rgommers/code/bldscipy/.mesonpy-native-file.ini
    debug       : false
    optimization: 2
    prefix      : /home/rgommers/anaconda3/envs/scipy-dev
    strip       : true

Found ninja-1.10.2.git.kitware.jobserver-1 at /tmp/build-env-pfwujcvb/bin/ninja

* Installing packages in isolated environment... (pep621 >= 0.3.0)
* Building sdist...
+ meson setup --native-file=/home/rgommers/code/bldscipy/.mesonpy-native-file.ini -Ddebug=false -Dstrip=true -Doptimization=2 --prefix=/home/rgommers/anaconda3/envs/scipy-dev /home/rgommers/code/bldscipy /home/rgommers/code/bldscipy/.mesonpy-48mc7pbv/build
The Meson build system
Version: 0.62.1
Source dir: /home/rgommers/code/bldscipy
Build dir: /home/rgommers/code/bldscipy/.mesonpy-48mc7pbv/build
Build type: native build
Project name: SciPy
Project version: 1.9.0.dev0
C compiler for the host machine: /home/rgommers/anaconda3/envs/scipy-dev/bin/x86_64-conda-linux-gnu-cc (gcc 9.4.0 "x86_64-conda-linux-gnu-cc (GCC) 9.4.0")
C linker for the host machine: /home/rgommers/anaconda3/envs/scipy-dev/bin/x86_64-conda-linux-gnu-cc ld.bfd 2.36.1
C++ compiler for the host machine: /home/rgommers/anaconda3/envs/scipy-dev/bin/x86_64-conda-linux-gnu-c++ (gcc 9.4.0 "x86_64-conda-linux-gnu-c++ (GCC) 9.4.0")
C++ linker for the host machine: /home/rgommers/anaconda3/envs/scipy-dev/bin/x86_64-conda-linux-gnu-c++ ld.bfd 2.36.1
Host machine cpu family: x86_64
Host machine cpu: x86_64
Compiler for C supports arguments -Wno-unused-but-set-variable: YES 
Library m found: YES
Fortran compiler for the host machine: /home/rgommers/anaconda3/envs/scipy-dev/bin/x86_64-conda-linux-gnu-gfortran (gcc 9.4.0 "GNU Fortran (GCC) 9.4.0")
Fortran linker for the host machine: /home/rgommers/anaconda3/envs/scipy-dev/bin/x86_64-conda-linux-gnu-gfortran ld.bfd 2.36.1
Program cython found: YES (/tmp/build-env-pfwujcvb/bin/cython)
Program pythran found: YES (/tmp/build-env-pfwujcvb/bin/pythran)
Program cp found: YES (/usr/bin/cp)
Program python3 found: YES (/tmp/build-env-pfwujcvb/bin/python)
Found pkg-config: /home/rgommers/anaconda3/envs/scipy-dev/bin/pkg-config (0.29.2)
Library npymath found: YES
Library npyrandom found: YES
Run-time dependency openblas found: YES 0.3.18
Dependency openblas found: YES 0.3.18 (cached)
Program _build_utils/cythoner.py found: YES (/tmp/build-env-pfwujcvb/bin/python /home/rgommers/code/bldscipy/scipy/_build_utils/cythoner.py)
Checking for function "open_memstream" : NO 
Configuring messagestream_config.h using configuration
Checking for size of "void*" : 8
Run-time dependency threads found: YES
Checking for size of "void*" : 8
Dependency threads found: YES unknown (cached)
Compiler for C supports arguments -Wno-unused-but-set-variable: YES (cached)
Compiler for C supports arguments -Wno-unused-but-set-variable: YES (cached)
Compiler for C++ supports arguments -Wno-format-truncation: YES 
Compiler for C++ supports arguments -Wno-class-memaccess: YES 
Build targets in project: 199

SciPy 1.9.0.dev0

  User defined options
    Native files: /home/rgommers/code/bldscipy/.mesonpy-native-file.ini
    debug       : false
    optimization: 2
    prefix      : /home/rgommers/anaconda3/envs/scipy-dev
    strip       : true

Found ninja-1.10.2.git.kitware.jobserver-1 at /tmp/build-env-pfwujcvb/bin/ninja

+ meson dist --no-tests --formats gztar
Created /home/rgommers/code/bldscipy/.mesonpy-48mc7pbv/build/meson-dist/SciPy-1.9.0.dev0.tar.gz
Successfully built scipy-1.9.0.dev0.tar.gz

$ python -m build --sdist --no-isolation
ERROR Missing dependencies:
        numpy==1.21.4; python_version=='3.10' and platform_python_implementation != 'PyPy'

$ python -m build --sdist --no-isolation --skip-dependency-check
* Building sdist...
+ meson setup --native-file=/home/rgommers/code/bldscipy/.mesonpy-native-file.ini -Ddebug=false -Dstrip=true -Doptimization=2 --prefix=/home/rgommers/anaconda3/envs/scipy-dev /home/rgommers/code/bldscipy /home/rgommers/code/bldscipy/.mesonpy-2_x1cita/build
$ ls dist
scipy-1.9.0.dev0.tar.gz

Full output of the above (only irrelevant Cython-related warnings omitted):

$ python -m build --sdist --no-isolation --skip-dependency-check
* Building sdist...
+ meson setup --native-file=/home/rgommers/code/bldscipy/.mesonpy-native-file.ini -Ddebug=false -Dstrip=true -Doptimization=2 --prefix=/home/rgommers/anaconda3/envs/scipy-dev /home/rgommers/code/bldscipy /home/rgommers/code/bldscipy/.mesonpy-2_x1cita/build
The Meson build system
Version: 0.62.1
Source dir: /home/rgommers/code/bldscipy
Build dir: /home/rgommers/code/bldscipy/.mesonpy-2_x1cita/build
Build type: native build
Project name: SciPy
Project version: 1.9.0.dev0
C compiler for the host machine: /home/rgommers/anaconda3/envs/scipy-dev/bin/x86_64-conda-linux-gnu-cc (gcc 9.4.0 "x86_64-conda-linux-gnu-cc (GCC) 9.4.0")
C linker for the host machine: /home/rgommers/anaconda3/envs/scipy-dev/bin/x86_64-conda-linux-gnu-cc ld.bfd 2.36.1
C++ compiler for the host machine: /home/rgommers/anaconda3/envs/scipy-dev/bin/x86_64-conda-linux-gnu-c++ (gcc 9.4.0 "x86_64-conda-linux-gnu-c++ (GCC) 9.4.0")
C++ linker for the host machine: /home/rgommers/anaconda3/envs/scipy-dev/bin/x86_64-conda-linux-gnu-c++ ld.bfd 2.36.1
Host machine cpu family: x86_64
Host machine cpu: x86_64
Compiler for C supports arguments -Wno-unused-but-set-variable: YES 
Library m found: YES
Fortran compiler for the host machine: /home/rgommers/anaconda3/envs/scipy-dev/bin/x86_64-conda-linux-gnu-gfortran (gcc 9.4.0 "GNU Fortran (GCC) 9.4.0")
Fortran linker for the host machine: /home/rgommers/anaconda3/envs/scipy-dev/bin/x86_64-conda-linux-gnu-gfortran ld.bfd 2.36.1
Program cython found: YES (/home/rgommers/anaconda3/envs/scipy-dev/bin/cython)
Program pythran found: YES (/home/rgommers/anaconda3/envs/scipy-dev/bin/pythran)
Program cp found: YES (/usr/bin/cp)
Program python3 found: YES (/home/rgommers/anaconda3/envs/scipy-dev/bin/python3.10)
Found pkg-config: /home/rgommers/anaconda3/envs/scipy-dev/bin/pkg-config (0.29.2)
Library npymath found: YES
Library npyrandom found: YES
Run-time dependency openblas found: YES 0.3.18
Dependency openblas found: YES 0.3.18 (cached)
Program _build_utils/cythoner.py found: YES (/home/rgommers/anaconda3/envs/scipy-dev/bin/python3.10 /home/rgommers/code/bldscipy/scipy/_build_utils/cythoner.py)
Checking for function "open_memstream" : NO 
Configuring messagestream_config.h using configuration
Checking for size of "void*" : 8
Run-time dependency threads found: YES
Checking for size of "void*" : 8
Dependency threads found: YES unknown (cached)
Compiler for C supports arguments -Wno-unused-but-set-variable: YES (cached)
Compiler for C supports arguments -Wno-unused-but-set-variable: YES (cached)
Compiler for C++ supports arguments -Wno-format-truncation: YES 
Compiler for C++ supports arguments -Wno-class-memaccess: YES 
Build targets in project: 199

SciPy 1.9.0.dev0

  User defined options
    Native files: /home/rgommers/code/bldscipy/.mesonpy-native-file.ini
    debug       : false
    optimization: 2
    prefix      : /home/rgommers/anaconda3/envs/scipy-dev
    strip       : true

Found ninja-1.10.2.git.kitware.jobserver-1 at /home/rgommers/anaconda3/envs/scipy-dev/bin/ninja

+ meson dist --no-tests --formats gztar                                                             
Created /home/rgommers/code/bldscipy/.mesonpy-2_x1cita/build/meson-dist/SciPy-1.9.0.dev0.tar.gz
Successfully built scipy-1.9.0.dev0.tar.gz

Conclusion for meson-python: we have 2 ways of generating an sdist, with python -m build with and without isolation. Compared to the setup.py-based sdist method, the meson-python-generated sdists are missing:

• concatenated license files,
• a file name containing the dynamically generated version; the part after dev0+ contains (a) an incrementing number for "commits since last git tag, and (b) the first 7 characters of the current git commit hash
• uncommitted changes, if such changes were made in the repo

Comparison between setup.py and meson-python results:
Are any of these differences important for sdist's?

• The concatenated license files we could perhaps live without, although it is more correct to do the concatenation - but a separate LICENSES_bundled.txt isn't the end of the world.
• We do need the dynamically generated version, because we upload nightly wheels built in CI to https://anaconda.org/scipy-wheels-nightly/scipy/files and the incrementing number is necessary to be able to retrieve the latest nightly.
• Uncommitted changes: for an sdist that is not so important, we could live without it. Reason: sdists as a final artifact are only needed at release time. However, we do need them for wheels (see further down), so if a wheel by default gets built via an sdist, then we need the changes in the sdist too.

Generating a wheel

These are the ways of generating a wheel, and all except the setup.py bdist_wheel apply to both setup.py and meson-python based builds:

• python setup.py bdist_wheel
• pip wheel .
• pip wheel . --no-build-isolation
• python -m build (builds an sdist and then a wheel from the sdist)
• python -m build --wheel (builds a wheel directly, and allows using -Cbuild-dir which helps with caching)
• python -m build --wheel --no-isolation --skip-dependency-check

With setup.py:

$ python setup.py bdist_wheel
$ ls dist
scipy-1.9.0.dev0+2035.7076a90-cp310-cp310-linux_x86_64.whl

$ pip wheel .
Processing /home/rgommers/code/bldscipy
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting numpy>=1.18.5
  Downloading numpy-1.22.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (16.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 16.8/16.8 MB 4.7 MB/s eta 0:00:00
Saved ./numpy-1.22.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Building wheels for collected packages: scipy
  Building wheel for scipy (pyproject.toml) ... 
  Created wheel for scipy: filename=scipy-1.9.0.dev0+2035.7076a90-cp310-cp310-linux_x86_64.whl size=27155214 sha256=5f00c67ac877961a3e3c84974138e1260077ac9940f549b4414495cf66318797
  Stored in directory: /tmp/pip-ephem-wheel-cache-t6yc2x4j/wheels/ac/02/47/5c5589c7e259950e575a7271d3d3625c505da9fba9ad6879c7
Successfully built scipy
$ ls *.whl
$ ls *.whl
numpy-1.22.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
scipy-1.9.0.dev0+2035.7076a90-cp310-cp310-linux_x86_64.whl

$ pip wheel . --no-build-isolation
Processing /home/rgommers/code/bldscipy
  Preparing metadata (pyproject.toml) ... done
Collecting numpy>=1.18.5
  Using cached numpy-1.22.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (16.8 MB)
Saved ./numpy-1.22.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Building wheels for collected packages: scipy
  Building wheel for scipy (pyproject.toml) ... done
  Created wheel for scipy: filename=scipy-1.9.0.dev0+2035.7076a90-cp310-cp310-linux_x86_64.whl size=26948340 sha256=61fd32d04865dd38038cbb69acf21ec6255e7f04421f03591b55e7b8342939e4
  Stored in directory: /tmp/pip-ephem-wheel-cache-8uhol9ge/wheels/ac/02/47/5c5589c7e259950e575a7271d3d3625c505da9fba9ad6879c7
Successfully built scipy
(scipy-dev) rgommers@machine:~/code/bldscipy (main)$ ls *.whl
numpy-1.22.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
scipy-1.9.0.dev0+2035.7076a90-cp310-cp310-linux_x86_64.whl

Note on the above: not sure what's going on there exactly, but --no-build-isolation seems to be partially ignored for pip wheel, no idea why it's messing with numpy there.

$ python -m build
* Creating venv isolated environment...
* Installing packages in isolated environment... (Cython>=0.29.18, numpy; python_version>='3.11', numpy; python_version>='3.8' and platform_python_implementation=='PyPy', numpy==1.18.5; python_version=='3.8' and (platform_machine!='arm64' or platform_system!='Darwin') and platform_machine!='aarch64' and platform_python_implementation != 'PyPy', numpy==1.19.2; python_version=='3.8' and platform_machine=='aarch64' and platform_python_implementation != 'PyPy', numpy==1.19.3; python_version=='3.9' and (platform_machine!='arm64' or platform_system!='Darwin') and platform_python_implementation != 'PyPy', numpy==1.20.0; python_version=='3.8' and platform_machine=='arm64' and platform_system=='Darwin', numpy==1.20.0; python_version=='3.9' and platform_machine=='arm64' and platform_system=='Darwin', numpy==1.21.4; python_version=='3.10' and platform_python_implementation != 'PyPy', pybind11>=2.4.3, pythran>=0.9.12, setuptools<60.0, wheel)
* Getting dependencies for sdist...
* Building sdist...
* Building wheel from sdist
* Creating venv isolated environment...
* Installing packages in isolated environment... (Cython>=0.29.18, numpy; python_version>='3.11', numpy; python_version>='3.8' and platform_python_implementation=='PyPy', numpy==1.18.5; python_version=='3.8' and (platform_machine!='arm64' or platform_system!='Darwin') and platform_machine!='aarch64' and platform_python_implementation != 'PyPy', numpy==1.19.2; python_version=='3.8' and platform_machine=='aarch64' and platform_python_implementation != 'PyPy', numpy==1.19.3; python_version=='3.9' and (platform_machine!='arm64' or platform_system!='Darwin') and platform_python_implementation != 'PyPy', numpy==1.20.0; python_version=='3.8' and platform_machine=='arm64' and platform_system=='Darwin', numpy==1.20.0; python_version=='3.9' and platform_machine=='arm64' and platform_system=='Darwin', numpy==1.21.4; python_version=='3.10' and platform_python_implementation != 'PyPy', pybind11>=2.4.3, pythran>=0.9.12, setuptools<60.0, wheel)

$ ls dist
scipy-1.9.0.dev0+2035.7076a90.tar.gz
scipy-1.9.0.dev0+7076a90.7076a90-cp310-cp310-linux_x86_64.whl

Note that something is going wrong with the wheel filename (commit hash is repeated), that seems to be a minor bug somewhere. More detailed output of the above (note that build shows the raw distutils output (~40,000 lines), that's omitted):

$ python -m build
* Creating venv isolated environment...
* Installing packages in isolated environment... (Cython>=0.29.18, numpy; python_version>='3.11', numpy; python_version>='3.8' and platform_python_implementation=='PyPy', numpy==1.18.5; python_version=='3.8' and (platform_machine!='arm64' or platform_system!='Darwin') and platform_machine!='aarch64' and platform_python_implementation != 'PyPy', numpy==1.19.2; python_version=='3.8' and platform_machine=='aarch64' and platform_python_implementation != 'PyPy', numpy==1.19.3; python_version=='3.9' and (platform_machine!='arm64' or platform_system!='Darwin') and platform_python_implementation != 'PyPy', numpy==1.20.0; python_version=='3.8' and platform_machine=='arm64' and platform_system=='Darwin', numpy==1.20.0; python_version=='3.9' and platform_machine=='arm64' and platform_system=='Darwin', numpy==1.21.4; python_version=='3.10' and platform_python_implementation != 'PyPy', pybind11>=2.4.3, pythran>=0.9.12, setuptools<60.0, wheel)
* Getting dependencies for sdist...
running egg_info
creating scipy.egg-info
writing scipy.egg-info/PKG-INFO
writing dependency_links to scipy.egg-info/dependency_links.txt
writing requirements to scipy.egg-info/requires.txt
writing top-level names to scipy.egg-info/top_level.txt
writing manifest file 'scipy.egg-info/SOURCES.txt'
reading manifest file 'scipy.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files matching '*_subr_*.f' found under directory 'scipy/linalg/src/id_dist/src'
no previously-included directories found matching 'benchmarks/env'
no previously-included directories found matching 'benchmarks/results'
no previously-included directories found matching 'benchmarks/html'
no previously-included directories found matching 'benchmarks/scipy'
no previously-included directories found matching 'doc/build'
no previously-included directories found matching 'doc/source/generated'
warning: no previously-included files matching '*.pyc' found anywhere in distribution
warning: no previously-included files matching '*~' found anywhere in distribution
warning: no previously-included files matching '*.bak' found anywhere in distribution
warning: no previously-included files matching '*.swp' found anywhere in distribution
warning: no previously-included files matching '*.pyo' found anywhere in distribution
adding license file 'LICENSE.txt'
adding license file 'LICENSES_bundled.txt'
writing manifest file 'scipy.egg-info/SOURCES.txt'
* Building sdist...
Running from SciPy source directory.
...
Creating tar archive
removing 'scipy-1.9.0.dev0+2035.7076a90' (and everything under it)
* Building wheel from sdist
* Creating venv isolated environment...
* Installing packages in isolated environment... (Cython>=0.29.18, numpy; python_version>='3.11', numpy; python_version>='3.8' and platform_python_implementation=='PyPy', numpy==1.18.5; python_version=='3.8' and (platform_machine!='arm64' or platform_system!='Darwin') and platform_machine!='aarch64' and platform_python_implementation != 'PyPy', numpy==1.19.2; python_version=='3.8' and platform_machine=='aarch64' and platform_python_implementation != 'PyPy', numpy==1.19.3; python_version=='3.9' and (platform_machine!='arm64' or platform_system!='Darwin') and platform_python_implementation != 'PyPy', numpy==1.20.0; python_version=='3.8' and platform_machine=='arm64' and platform_system=='Darwin', numpy==1.20.0; python_version=='3.9' and platform_machine=='arm64' and platform_system=='Darwin', numpy==1.21.4; python_version=='3.10' and platform_python_implementation != 'PyPy', pybind11>=2.4.3, pythran>=0.9.12, setuptools<60.0, wheel)
* Getting dependencies for wheel...
running egg_info
...
writing manifest file 'scipy.egg-info/SOURCES.txt'
* Installing packages in isolated environment... (wheel)
* Building wheel...
Running from SciPy source directory.
Cythonizing sources
Running scipy/stats/_generate_pyx.py
Successfully built scipy-1.9.0.dev0+2035.7076a90.tar.gz and scipy-1.9.0.dev0+7076a90.7076a90-cp310-cp310-linux_x86_64.whl

$ python -m build --wheel
* Creating venv isolated environment...
* Installing packages in isolated environment... (Cython>=0.29.18, numpy; python_version>='3.11', numpy; python_version>='3.8' and platform_python_implementation=='PyPy', numpy==1.18.5; python_version=='3.8' and (platform_machine!='arm64' or platform_system!='Darwin') and platform_machine!='aarch64' and platform_python_implementation != 'PyPy', numpy==1.19.2; python_version=='3.8' and platform_machine=='aarch64' and platform_python_implementation != 'PyPy', numpy==1.19.3; python_version=='3.9' and (platform_machine!='arm64' or platform_system!='Darwin') and platform_python_implementation != 'PyPy', numpy==1.20.0; python_version=='3.8' and platform_machine=='arm64' and platform_system=='Darwin', numpy==1.20.0; python_version=='3.9' and platform_machine=='arm64' and platform_system=='Darwin', numpy==1.21.4; python_version=='3.10' and platform_python_implementation != 'PyPy', pybind11>=2.4.3, pythran>=0.9.12, setuptools<60.0, wheel)
* Getting dependencies for wheel...
running egg_info
* Installing packages in isolated environment... (wheel)
* Building wheel...
Running from SciPy source directory.
Cythonizing sources

$ ls dist
scipy-1.9.0.dev0+2035.7076a90-cp310-cp310-linux_x86_64.whl

$ python -m build --wheel --no-isolation
ERROR Missing dependencies:
        numpy==1.21.4; python_version=='3.10' and platform_python_implementation != 'PyPy'

$ python -m build --wheel --no-isolation --skip-dependency-check
* Building wheel...
$ ls dist
scipy-1.9.0.dev0+2035.7076a90-cp310-cp310-linux_x86_64.whl

Conclusion for setup.py: we have 3 different packages to generate wheels with (distutils, pip and build), and with the various ways of invoking those, there are ~6 commands of interest (with/without isolation, via sdist or not). Those methods do not all result in equivalent wheels: in particular the numpy version used is quite significant. Also important: none of them are the same as the wheels we release on PyPI. The key difference is that there are no vendored libraries so these wheels are not portable between different machines (or even different environments on the same machine in some cases).

Related conclusion: if sdists are all the same and match the ones we upload to PyPI, and wheels are very different from those we upload to PyPI, there is no direct correspondence between sdist and wheels on PyPI. And this is fine/expected.

For completeness, here are the libraries we vendor into released wheels (example from unpacking from PyPI):

$ ls scipy-1.8.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64/scipy.libs/
libgfortran-040039e1.so.5.0.0
libopenblasp-r0-8b9e111f.3.17.so
libquadmath-96973f99.so.0.0.0

With meson-python:

Note that wheel generation with meson-python with build isolation currently has a problem right now and doesn't work:

ERROR: Could not find a version that satisfies the requirement patchelf-wrapper (from versions: none)
ERROR: No matching distribution found for patchelf-wrapper

Note that this error is unexpected and should be fixed, I filed https://github.com/FFY00/meson-python/issues/52. I will update this section once that is done. For now I'll continue with --no-build-isolation only, which works fine.

$ pip wheel . --no-build-isolation
Processing /home/rgommers/code/bldscipy
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: scipy
  Building wheel for scipy (pyproject.toml) ... done
  Created wheel for scipy: filename=scipy-1.9.0.dev0-cp310-cp310-linux_x86_64.whl size=23844483 sha256=87750bbf0eebfaab7583fea5376bb6afb3635edda35977a4e86e801e299e10ff
  Stored in directory: /tmp/pip-ephem-wheel-cache-mj5tc2hi/wheels/ac/02/47/5c5589c7e259950e575a7271d3d3625c505da9fba9ad6879c7
Successfully built scipy
$ ls *.whl
scipy-1.9.0.dev0-cp310-cp310-linux_x86_64.whl

$ python -m build --wheel --no-isolation
ERROR Missing dependencies:
        numpy==1.21.4; python_version=='3.10' and platform_python_implementation != 'PyPy'
        patchelf-wrapp

This is the same error as when running this wheel build command with setup.py (on the main branch as of today).

$ python -m build --wheel --no-isolation --skip-dependency-check
* Building wheel...
+ meson setup --native-file=/home/rgommers/code/bldscipy/.mesonpy-native-file.ini -Ddebug=false -Dstrip=true -Doptimization=2 --prefix=/home/rgommers/anaconda3/envs/scipy-dev /home/rgommers/code/bldscipy /home/rgommers/code/bldscipy/.mesonpy-1q_dj_tb/build
...
+ meson compile    
[1565/1565] Linking target scipy/sparse/sparsetools/_sparsetools.cpython-310-x86_64-linux-gnu.so
+ meson install --destdir /home/rgommers/code/bldscipy/.mesonpy-1q_dj_tb/install
ninja: Entering directory `/home/rgommers/code/bldscipy/.mesonpy-1q_dj_tb/build'
Installing scipy/_lib/_ccallback_c.cpython-310-x86_64-linux-gnu.so to /home/rgommers/code/bldscipy/.mesonpy-1q_dj_tb/install/home/rgommers/anaconda3/envs/scipy-dev/lib/python3.10/site-packages/scipy/_lib
Stripping target 'scipy/_lib/_ccallback_c.cpython-310-x86_64-linux-gnu.so'.

Copying files to wheel...
[1213/1213] scipy/io/_harwell_boeing/tests/test_hb.pyy
Successfully built scipy-1.9.0.dev0-cp310-cp310-linux_x86_64.whl
$ ls dist
scipy-1.9.0.dev0-cp310-cp310-linux_x86_64.whl

More detailed output:

$ python -m build --wheel --no-isolation --skip-dependency-check
* Building wheel...
+ meson setup --native-file=/home/rgommers/code/bldscipy/.mesonpy-native-file.ini -Ddebug=false -Dstrip=true -Doptimization=2 --prefix=/home/rgommers/anaconda3/envs/scipy-dev /home/rgommers/code/bldscipy /home/rgommers/code/bldscipy/.mesonpy-1q_dj_tb/build
The Meson build system
Version: 0.62.1
Source dir: /home/rgommers/code/bldscipy
Build dir: /home/rgommers/code/bldscipy/.mesonpy-1q_dj_tb/build
Build type: native build
Project name: SciPy
Project version: 1.9.0.dev0
C compiler for the host machine: /home/rgommers/anaconda3/envs/scipy-dev/bin/x86_64-conda-linux-gnu-cc (gcc 9.4.0 "x86_64-conda-linux-gnu-cc (GCC) 9.4.0")
C linker for the host machine: /home/rgommers/anaconda3/envs/scipy-dev/bin/x86_64-conda-linux-gnu-cc ld.bfd 2.36.1
C++ compiler for the host machine: /home/rgommers/anaconda3/envs/scipy-dev/bin/x86_64-conda-linux-gnu-c++ (gcc 9.4.0 "x86_64-conda-linux-gnu-c++ (GCC) 9.4.0")
C++ linker for the host machine: /home/rgommers/anaconda3/envs/scipy-dev/bin/x86_64-conda-linux-gnu-c++ ld.bfd 2.36.1
Host machine cpu family: x86_64
Host machine cpu: x86_64
Compiler for C supports arguments -Wno-unused-but-set-variable: YES 
Library m found: YES
Fortran compiler for the host machine: /home/rgommers/anaconda3/envs/scipy-dev/bin/x86_64-conda-linux-gnu-gfortran (gcc 9.4.0 "GNU Fortran (GCC) 9.4.0")
Fortran linker for the host machine: /home/rgommers/anaconda3/envs/scipy-dev/bin/x86_64-conda-linux-gnu-gfortran ld.bfd 2.36.1
Program cython found: YES (/home/rgommers/anaconda3/envs/scipy-dev/bin/cython)
Program pythran found: YES (/home/rgommers/anaconda3/envs/scipy-dev/bin/pythran)
Program cp found: YES (/usr/bin/cp)
Program python3 found: YES (/home/rgommers/anaconda3/envs/scipy-dev/bin/python3.10)
Found pkg-config: /home/rgommers/anaconda3/envs/scipy-dev/bin/pkg-config (0.29.2)
Library npymath found: YES
Library npyrandom found: YES
Run-time dependency openblas found: YES 0.3.18
Dependency openblas found: YES 0.3.18 (cached)
Program _build_utils/cythoner.py found: YES (/home/rgommers/anaconda3/envs/scipy-dev/bin/python3.10 /home/rgommers/code/bldscipy/scipy/_build_utils/cythoner.py)
Checking for function "open_memstream" : NO 
Configuring messagestream_config.h using configuration
Checking for size of "void*" : 8
Run-time dependency threads found: YES
Checking for size of "void*" : 8
Dependency threads found: YES unknown (cached)
Compiler for C supports arguments -Wno-unused-but-set-variable: YES (cached)
Compiler for C supports arguments -Wno-unused-but-set-variable: YES (cached)
Compiler for C++ supports arguments -Wno-format-truncation: YES 
Compiler for C++ supports arguments -Wno-class-memaccess: YES 
Build targets in project: 199

SciPy 1.9.0.dev0

  User defined options
    Native files: /home/rgommers/code/bldscipy/.mesonpy-native-file.ini
    debug       : false
    optimization: 2
    prefix      : /home/rgommers/anaconda3/envs/scipy-dev
    strip       : true

+ meson compile    
[1565/1565] Linking target scipy/sparse/sparsetools/_sparsetools.cpython-310-x86_64-linux-gnu.so
+ meson install --destdir /home/rgommers/code/bldscipy/.mesonpy-1q_dj_tb/install
ninja: Entering directory `/home/rgommers/code/bldscipy/.mesonpy-1q_dj_tb/build'
Installing scipy/_lib/_ccallback_c.cpython-310-x86_64-linux-gnu.so to /home/rgommers/code/bldscipy/.mesonpy-1q_dj_tb/install/home/rgommers/anaconda3/envs/scipy-dev/lib/python3.10/site-packages/scipy/_lib
Stripping target 'scipy/_lib/_ccallback_c.cpython-310-x86_64-linux-gnu.so'.

Copying files to wheel...
[1213/1213] scipy/io/_harwell_boeing/tests/test_hb.pyy
Successfully built scipy-1.9.0.dev0-cp310-cp310-linux_x86_64.whl

Comparison between setup.py and meson-python results:

We get similar wheels from both build systems. The differences include:

• The included/installed files for meson-python are more correct than for setup.py. For the latter we include a bunch of things that shouldn't be installed, like setup.py files. The main reason for that is that MANIFEST.in is a pain to work with. It doesn't matter too much, but we'd rather install only the files we actually need.
• By default, meson-python strips shared libraries while distutils does not. This means that the wheels produced by meson-python are way smaller, and we don't have to worry about stripping manually as we do now in the scipy-wheels repo.
• The file names of the wheels produced by meson-python currently do not include the dynamically generated version string (same as for sdists).
• The license concatenation that is done in setup.py doesn't happen for meson-python, so LICENSE.txt is the plain BSD-3 license and does not include the content of LICENSES_bundled.txt.
• Any uncommitted changes do not end up in the meson-python generated wheels.

Installing from a repo or an sdist

Installing from a repo or sdist is interesting, because (a) it still invokes the build system, and (b) many users, contributors, and packagers have to do this. I'm not considering installing from wheels, because that's unrelated to build system hooks and a much simpler problem - pypa/installer can be used here in addition to pip; other tools like poetry and pdm will typically invoke pip under the hood to install a wheel.

Ways to install from a repo or an sdist:

• python setup.py install (not recommended, doesn't allow uninstall and may clobber previously installed version)
• pip install .
• pip install scipy --no-binary scipy (from sdist on PyPI)
• pip install <scipy-1.9.0.tar.gz> (from local sdist)
• pip install git+https://github.com/scipy/scipy.git (directly from GitHub)
• python -m build && pip install dist/scipy*.whl
• The --no-build-isolation flavors of all of the pip/build invocations above.

Side note: I'm not considering editable installs here; those are not currently possible with meson-python, but that will come and then pip install . -e will work the same as it currently does for setup.py-based installs. There will not be a direct equivalent for python setup.py develop, because Meson itself always builds out-of-place.

With setup.py:

I will not show the output of each of these methods, but just state that they all work today. Instead, I will explain what the key things to keep in mind are:

• pip install . is the most-often used command to install from a local repo
• all commands to install from a local repo will include uncommitted changes
• python setup.py install does not do any dependency checking/management, so in that way it's similar to --no-build-isolation for pip and --no-isolation for build.
• when omitting --no-build-isolation, the build happens in a fresh virtualenv against the numpy version that is pinned with == in pyproject.toml. The version that is chosen depends on Python version, OS, hardware architecture, and Python interpreter. The actual install_requires for the built SciPy should be >=<numpy-version-at-build-time>, but what actually gets recorded by pip is >=1.18.5 (the lowest supported numpy version today).
• We cannot express all our dependencies (e.g., BLAS/LAPACK, compilers) so the built scipy package depends on whatever was found at build time. If a user removes that BLAS/LAPACK or the compiler runtime, the installed scipy package will break. In practice this is not much of a problem, I cannot remember a bug report about this. Upgrades are typically backwards-compatible.
• Not using --no-build-isolation is typically only doing the right thing for virtualenvs or a standalone Python installed from python.org. It is not great for working in Conda/Nix/Spack/etc. envs, nor in Linux distros - you want to build against the numpy package from those environments. The behavior of --no-build-isolation is therefore very important.

With meson-python:

$ pip install .  # TODO, runs into the patchelf-wrapper issue

$ pip install . --no-build-isolation
Processing /home/rgommers/code/bldscipy
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: scipy
  Building wheel for scipy (pyproject.toml) ... done
  Created wheel for scipy: filename=scipy-1.9.0.dev0-cp310-cp310-linux_x86_64.whl size=23844482 sha256=18c2e0e7e9ab7a140a2f2d573af3b4001432cc7c6aa1c038541508d221067fc9
  Stored in directory: /tmp/pip-ephem-wheel-cache-2twxir85/wheels/ac/02/47/5c5589c7e259950e575a7271d3d3625c505da9fba9ad6879c7
Successfully built scipy
Installing collected packages: scipy
Successfully installed scipy-1.9.0.dev0

$ pip install . --no-use-pep517
Processing /home/rgommers/code/bldscipy
ERROR: Disabling PEP 517 processing is invalid: project specifies a build backend of mesonpy in pyproject.toml

Added --no-use-pep517 for completeness. This does not work, on purpose: Pip disabled it because it's likely user error for non-setuptools backends. That said, that is a guess of course, and it would have worked for SciPy's 1.9.0 release, since we ship with a distutils build system as a fallback. This shows that that fallback cannot be invoked directly; instead the user must build from the repo with python setup.py bdist_wheel && pip install dist/scipy*.whl. It's of course a little questionable that an explicit --no-use-pep517 fails based on content in the pyproject.toml file that we explicitly ask it to ignore ... but that ship has sailed, let's not worry about it. See this comment and the following ones on gh-15476 for more details on this.

$ python -m build --wheel --no-isolation --skip-dependency-check && pip install dist/scipy*.whl
...
$ pip list
...
scipy                         1.9.0.dev0

Comparison between setup.py and meson-python results:

Overall behavior is similar (aside from build performance:) ); turning build isolation on/off is the most important factor in behavior. The differences noted for wheel builds apply similarly to installs from source (repo or sdist).

Update on recommendations for changes

Now that I've written all this, let me get back to my "tentative conclusions" in the comment higher up.

Tentative conclusion 1: meson-python should include changes to files tracked in git but not committed, that's better all around.

This one I'm very confident about. It will be very confusing to continue ignoring local changes that aren't committed, plus it leads to serious issues when trying to apply patches (both for us when building release wheels, and for any downstream packagers - they all carry patches). I don't think there's a serious case to make for keeping the current behavior, the upside is very minor and the problems it creates are real.

EDIT: I opened https://github.com/FFY00/meson-python/issues/53 for this.

Tentative conclusion 2: SciPy sdists on PyPI should no longer contain numpy== pins, to keep sdists usable with pip install for all packaging systems.

This one I'm a lot less confident about after writing all this. There's actually 3 flavors to consider:

1. Leaving everything as it is today, including numpy== pins in main.
2. Only adding numpy== pins in maintenance branches, not in main.
3. Not adding numpy== pins anywhere.

@FFY00 first dropped these pins for main in gh-15476 because the pins were causing problems for CI. This can be worked around in other ways though, namely by consistently using --no-build-isolation with pip and --no-isolation --skip-dependency-check with build. One note if you're trying to test this: pip 22.1 (latest as of today) contains a serious regression for --no-build-isolation that will be reverted, so please ensure you're not testing with that version.

I'm still convinced that (2) isn't great: numpy pins should not be different between main and maintenance/* branches, that will lead to annoying changes in CI config files and behavior between them.

For option (3), the pros and cons are:

• Pro: no need to use --skip-dependency-check with build.
• Con: isolated builds will default to most recent numpy release, which then don't work with any older installed numpy.
• (I thought there were other reasons this was necessary/important in main, but I don't see them anymore).

So tl;dr - I think we can stay with numpy== pins in pyproject.toml in all branches. The main thing we should document well is our recommended use of pip and build. I think those are:

• for users wanting to install a development version from source in an environment: pip install . --no-build-isolation (this will work for any kind of dev env)
• for users wanting to install a release from source in an environment: pip install . if it's a virtualenv, pip install . --no-build-isolation otherwise
• for packagers: pip install . --no-build-isolation -vv (they may already have turned off build isolation globally anyway, e.g. https://github.com/conda/conda-build/blob/400fde1da22f8665b1dddc1286a4c21a5f5cd45a/conda_build/build.py#L2594-L2600)
• for us to build sdist's in CI: both python -m build --sdist and python -m build --sdist --no-isolation --skip-dependency-check (they're basically the same, just check they both work)
• for us to build wheels in CI: python -m build --wheel --no-isolation --skip-dependency-check (cannot use build isolation, we need to test against multiple numpy versions)
• for us to directly install and test: pip install . --no-build-isolation

I will update this PR to follow these recommendations, and see if it then all works as expected or not.

[rgommers]

Ugh, the 32-bit Linux job is a real pain. It downloads openblas through tools/openblas_support.py (which are the binaries built at https://github.com/MacPython/openblas-libs/), and those contain headers and a CMake support file, but not the pkg-config files we need.

The other custom openblas build we have, in .github/workflows/windows.yml (built with the MinGW toolchain by Matthew) do include pkg-config files.

I think I'll give up on that job for now. Once Ubuntu Jammy (22.04 LTS) has 32-bit Docker images available we can simply use those; that release has OpenBLAS 3.20.

[rgommers]

Okay, this is finally looking more or less happy. It's not mergeable yet, there are a couple of things left to do:

1. Fix meson-python for the patchelf-wrapper issue and cut a new release
2. Update this PR to meson-python 0.5.0`
3. Update this PR to use pip install . (so without build isolation) and ensure that that works.
4. Deal with building release wheels, either in the build-pypi-artifacts.py script here or in CI. The latter is only possible if https://github.com/FFY00/meson-python/issues/53 is addressed in time.

[eli-schwartz]

I have been mulling over this for a bit, and... it seems that the main problem, really, is tied to using meson dist in the build backend to create sdists. There are now multiple pep517 build backends for Meson which are doing the same thing, so I figure it's a good idea to ask "why".

So what's happening?

meson dist is the builtin Meson mechanism for creating release artifacts. It has some nifty functionality similar to what autotools make distcheck does -- particularly, it will build a presumably complete release, then do an end-to-end test and see if that release can be built on its own. It also only initially works from inside a git repo -- you cannot meson dist from a tarball you downloaded and extracted.

The above is true not just for SciPy, but for any complex package. So why do meson dist, meson-python and flit rely on extracting dist/sdist from committed changes only? The main reason is to avoid producing unreproducible/incorrect tarballs at release time. A good goal, but also very easy to avoid when you build your release artifacts in CI (which serious projects should do anyway).

That is, if you'll excuse me, a very... pythonic... way of thinking. Not everyone agrees that serious projects should do releases via CI, particularly if they need to do things like PGP-sign those releases (technically possible with PyPI, but the website will do its utmost to hide the signature files, so even the people who bothered with them, generally got the hint being messaged by PyPA, and stopped).

It also lets you trivially test that patches pass a full distcheck, and git bisect them too.

But also -- it makes it very easy to manage. Files not checked into git are likely not important to be distributed. Meson can also use gitattributes to remove files (like .github/) that don't get distributed, and it does this by using git archive under the hood. No need to invent MANIFEST.in :D but it does require using git to export the right files, which means uncommitted changes on disk are not going to be noticed.

Meson does provide functionality in case what you want to distribute isn't exactly what is in the state of the repository. You can use meson.add_dist_script() to run scripts that post-process the in-process distributable tarball. This may mean running autoreconf -fi for a program that supports both meson and autotools, or perhaps in SciPy's case swapping out the pyproject.toml file.

Alternatively, you may want to do without meson dist, for example by following flit which lists files tracked by git but copies them live, including uncommitted changes as long as those files are, in fact, tracked.

...

What would be the best thing to use? I don't know, there are pros and cons to a bunch of different approaches. The thing that most stands out to me, however, is not about what makes the best sdist. It's what you do with an sdist.

python -m build tries to build an sdist from an sdist, and then uses the built sdist in order to build a wheel. This is an... interesting... approach which I'm not sure I agree with; it's essentially reimplementing the concept of a distcheck (with the purpose of discovering the same type of issues that a meson dist already discovers), but divorcing it from the concept of doing that test from inside the git repository. Perhaps it's an okay approach to take for python -m build by assuming that the only people using it are package authors running from a git repository, and anyone else will manually remember to tell "build" to "not do that".
Interestingly, pip does not have this issue, but there's an open ticket asking for it: pypa/pip#10746

Either way, this bears repeating:

meson dist cannot be run recursively. It creates an output artifact that doesn't support meson dist and assumes you are happy with the one you've got.

This has been considered an acceptable tradeoff in the initial Meson implementation, which had a target audience of project maintainers cutting new releases, but does not fit well with the idea that end users might be expected to do this in order to install the software. End users should not be running maintainer-dist processes unexpectedly.

I don't know if the answer is to "not use meson dist" or "python packaging ecosystem software should not try to build sdists when installing".

[eli-schwartz]

Assuming no changes in the build backend or meson, the following should be more or less workable:

• use meson.add_dist_script() and commit it to git, for any cases where you need the sdist to differ from the contents of the latest commit
• recommend that people either use pip or python -m build -w to build wheels, and never invoke build without -w
• cross fingers that pip does not decide to do the source-to-sdist-to-wheel story some time in the future

[rgommers]

Thanks for the detailed thoughts @eli-schwartz

That is, if you'll excuse me, a very... pythonic... way of thinking. Not everyone agrees that serious projects should do releases via CI, particularly if they need to do things like PGP-sign those releases

I wasn't saying "upload to PyPI straight from CI". We actually do things like PGP signing and adding checksums of release artifacts to release notes. But if you do everything locally, you can get subtle bugs like "new release manager was on a different OS, and now file permissions changed, and we only noticed once the bug reports started rolling in" (this one is not made up unfortunately).

for example by following flit which lists files tracked by git but copies them live, including uncommitted changes as long as those files are, in fact, tracked.

Ah, I didn't realize this - that's actually the behavior I want as well.

[eli-schwartz]

I wasn't saying "upload to PyPI straight from CI".

(I have seen lots of people that do, sorry for the confusion.)

Ah, I didn't realize this - that's actually the behavior I want as well.

The relevant flit code for comparison is here:
https://github.com/pypa/flit/blob/048c87c380ac41efc4b26222114e54f6581c64f6/flit/sdist.py#L159-L177
Using any VCS implementation from https://github.com/pypa/flit/tree/048c87c380ac41efc4b26222114e54f6581c64f6/flit/vcs

[rgommers]

use meson.add_dist_script() and commit it to git, for any cases where you need the sdist to differ from the contents of the latest commit

This is actually useful for the "release sdist" part. We could indeed use it to amend the license file. For anything else that may go via sdist, like a packager applying a patch, it doesn't help.

I like the "steal code from Flit" option instead of using meson dist here.

[eli-schwartz]

And just to clarify -- I do believe that meson dist made a reasonable design decision, and I'm hesitant to change it, but I don't think it's the only reasonable design decision (different projects do have different needs), and indeed sometime in the future if/when (likely a post-py3.11 world) I design a pep517 build backend as part of Meson itself, it's not unlikely I would propose a different decision than the meson dist one.

It's something I'd need to seriously consider, based on how people are using it and what the Python ecosystem wants to use it for.
Fortunately, I get to punt that decision down the road while we wait for better stdlib support, and observe how third-party backends evolve before committing to an approach in Meson that might be a lot harder to change without backwards-compatibility concerns.

[rgommers]

If CI passes I will remove this script completely for now.

[FFY00]

There are some failures on the tests that I cannot tell if were caused by the build system change or something else, and linting errors on build-pypi-artifacts.py.

[rgommers]

It looks like this uncovered an unrelated bug in the recently merged optimize.direct, on 32-bit Linux only:

ImportError: dynamic module does not define module export function (PyInit__direct)

The only other issue is a random failure that is unrelated:

FAILED scipy/sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_PROPACK::test_svd_random_state_3[None]

[eli-schwartz]

It looks like this uncovered an unrelated bug in the recently merged optimize.direct, on 32-bit Linux only:

Is this just gh-16230 or... ?

EDIT: wait, _directmodule.c, py3.extension_module('_directmodule',, from ._directmodule import direct as _direct, but...

... cannot find "PyInit__direct"?

[rgommers]

It looks like this uncovered an unrelated bug in the recently merged optimize.direct, on 32-bit Linux only:

Is this just gh-16230 or... ?

It shouldn't be, because I just rebased on main and my branch does have PyMODINIT_FUNC. But there's an oddity that I'm trying to figure out: it's looking for PyInit__direct while the init routine is PyInit__directmodule.

[rgommers]

This is now good to go. Last call before merge, please have a look today if you want.

[blink1073]

Congratulations @rgommers and @FFY00, this has been an amazing lift.

[tylerjereddy]

Ok, I read through the entire diff again and it seems reasonable enough. I also read through most of the comments (90 %, ok maybe 80 % of the content). I've made a note to consult the release process adjustments for 1.9.0--those changes look solid, at least in theory.

Do I understand all the points in the discussion above? No. Does it look like carefully crafted work to improve our build system? Yes.

Let's see how it goes! Merging, with individual commits preserved, thanks.

[leofang]

Do I understand all the points in the discussion above? No. Does it look like carefully crafted work to improve our build system? Yes.

+100! Great work, Ralf and all!