Support disentangled native and python configuration

[SomeoneSerge]

Context

Hi! Currently the conventional way of building a projected backed by meson-python appears to be:

• run pip or other PEP-517 front-end,
• then rely on meson-python to run meson setup and guess the correct flags,
• ...
• wait for the control flow to get from meson back to meson-python, and for the latter to produce the wheel

This works great, and provides users with the simple pip install /path/to/repo workflow, which is very handy. However, when there are native build options to choose (e.g. BLAS implementations), meson being hidden behind a PEP-517 front-end is a hindrance. For example, rather than having an option to explicitly run meson setup build/ -Dblas=mkl -Dlapack=mkl $mesonFlags (retaining the full control to pass the correct native configuration flags) prior to running pip to produce the .dist-info and a .whl, users are as of now encouraged to rely on custom PEP-517 front-ends (pypa/build) and special flags to pass the meson options through layers of python glue: #54, scipy/scipy#17244 (comment). Note that even with this escape hatch, users do not have full control over the flags passed to meson, and thus are at the mercy of several layers of complex python-related logic owned by many different parties (e.g. sysconfig) to configure the native build correctly

I also imagine it is desirable in the long term that meson acquire the ability to generate wheels directly. I think that to accommodate such possibility in the future we'd need to work now towards separating the native build configuration (meson setup ..., cmake -S ... -B ..., etc) from the .dist-info+.whl generation and sdist verification logic (pip wheel ...), and we'd need to discourage users from entangling the two.

What could be done now

Seeing how the pip front-end already allows to pass --config-settings builddir=build, I imagine the first step could be to introduce a way to defuse the --reconfigure:

meson-python/mesonpy/__init__.py

Line 716 in 78def82

reconfigure = self._build_dir.joinpath('meson-private/coredata.dat').is_file()

This could be a baseline way of achieving a two-stage meson setup ... ; pip wheel ... build right now. Assuming, of course, that meson-python were interested in keeping this flow "supported"

---

What do you think?

[SomeoneSerge]

Briefly skimming through #321, it seems that sysconfig also makes cross-compilation harder?

[rgommers]

@SomeoneSerge do you actually have a need for the request of a separate meson setup invocation? E.g., a non-Python component being involved somehow? Having to write -C-Dblas=xxx rather than -Dblas=xxx isn't really enough of a reason for us to worry about supporting this.

it seems that sysconfig also makes cross-compilation harder?

That seems like an unrelated remark. Yes, that is the case. It has little to do with meson-python specifically, it is simply the case that Python itself has very poor support for cross-compilation. sysconfig is a major culprit indeed.

[eli-schwartz]

The problem with sysconfig is that there aren't alternatives. The state of the art in cross compilation is passing an undocumented variable to tell sysconfig to report information from a data file in another python. This works on some operating systems and is ignored on others.

There is a CPython meta issue open regarding changing the sysconfig API to make it more accommodating of what people need.
python/cpython#103480

[SomeoneSerge]

@rgommers I'm sorry, I think I've completely failed to communicate my motivation. I didn't realize you're maintaining meson-python as well either. I thought I'm starting a new conversation from the clean slate, rather than asking the same thing from the same person twice, as if arrogantly dismissing the previous response. Although I have been dismissive anyway, and I apologize for that. I also only invest a limited time a day in writing, so if I post something that isn't well received I'll try to iterate on it later.

Let's get back to the beginning. My original problem was to configure the BLAS implementation used by scipy in nixpkgs (DISCLAIMER: I do not in any way represent the community here, I'm a user, and if I'm being a moron that's entirely on me): NixOS/nixpkgs#206866. As scipy is a PEP-517/pyproject.toml-based project, nixpkgs builds it using a generic method, which amounts to pip wheel followed by pip install. This is expected to work (and it does) as pip is merely a front-end. I am now experimenting with adding the pypa/build front-end so that we can pass "-Csetup-args=-Dblas=..." "-Csetup-args=-Dlapack=...": NixOS/nixpkgs#230131. This is absolutely not an issue neither for me nor (I suppose) for nixpkgs: one can just choose the "right" front-end, and it doesn't cost a thing to anyone.

But, in principle, I think what is happening here is that scipy effectively acquires a "dependency" on a particular PEP-517 front-end. While harmless, this doesn't sounds like a desirable thing either.

The previous paragraph should be a synchronization point, because if we disagree there, the rest probably doesn't make sense either.

If we do agree there: my hypothesis is that the reason pypa/build becomes a "dependency" is because meson-python takes on some of the responsibilities of meson:

• ✅ We had a way to integrate meson projects with PEP-517 front-ends
• ✅ We found that sometimes we want to configure meson options
• ✅ Which means: we want some control over the underlying "native build configuration tool" (meson)
• ❔ But the native configuration happens as part of python configuration and cannot be controlled by the user directly
• ❔ The reaction was to forward options through a PEP-517 front-end, then through meson-python and to meson
• ❔ Some front-ends do not yet allow passing more than one option
• ❔ The next step could be to try and extend the standard so that all frontends provide means to pass multiple flags to the back-end
• ❔ This approach looks very similar to that of setuptools-based projects, which run CMake from their setup.py file. For example, pytorch's build script seems to consult a number of environment variables that are then passed on to CMake. Of course their situation is more problematic, because they usually hard-code flags and paths and do it wrong. Meson-python is much better at choosing the flags than a randomly picked setuptools-based native project. But there's always a risk of breaking some use-case, and for no reason other than fusing native configuration into the python build
• ❓ What if we didn't enforce that native configuration runs as part of python configuration?

The reason I referred to sysconfig was because I assumed it's what meson-python uses to infer the libpython-related flags, installation locations, et cetera. With just a brief search, I could locate the responsible code in meson-python, so I won't try to speculate on that

I also presumed that explicitly drawing a line between the tools that produce python metadata (.dist-info) and package wheels, and the tools that manage native configuration would have been in line with the goal (at least I think it was a long-term goal for meson at some point) of producing python wheels directly with meson. Again, end of speculation

@rgommers I hope I'm making at least some sense now. Do you think I still have a chance to extract an actionable issue from this?

@eli-schwartz Thank you for the link! I see they're also addressing some issues I thought I was experiencing with CMake. Personally, I'm not currently invested in nor am familiar with cross-compilation, but I imagine that having customization points to override sysconfig's choices on the meson/CMake level can prove handy. For example, unless I'm mistaken, in nixpkgs all of soabi names, site-packages and installation paths can be evaluated well before the build begins, and if one was really insisting on cross-compiling a native python extension and sysconfig for some reason didn't support it, they could probably just write these flags out explicitly (not that this would be encouraged of course)

[rgommers]

I am now experimenting with adding the pypa/build front-end so that we can pass "-Csetup-args=-Dblas=..." "-Csetup-args=-Dlapack=..."

Thanks for the detailed write-up @SomeoneSerge! I'll reply in more detail later, but a quick note: from pip 23.1 onwards, this exact same "-Csetup-args=-Dblas=..." "-Csetup-args=-Dlapack=..." will work with pip as well. So there's no dependency on a particular frontend I believe, since pip and pypa/build are the only two frontends that really matter for producing a wheel.

[eli-schwartz]

For the record, meson-python primarily just runs meson and expects meson to take responsibility for compiling with the right compiler, emitting the correct soabi extensions, and even placing files in the right site-packages directory. It then hooks in at the end and converts the files meson installed into a wheel file, and adds .dist-info metadata on top.

There's no technical reason why you can't install a project such as SciPy with

meson setup builddir &&
ninja -C builddir &&
meson install -C builddir

and in fact SciPy's dev.py script for handling various developer tasks and providing a convenient developer environment for SciPy, uses essentially this. It doesn't use pip or build or even meson-python.

The only thing you'd be missing compared to building with pip or build, is the .dist-info metadata, which is admittedly important for pip interoperability, although nix, dpkg, rpm, pacman, portage, xbps, conda... all don't really care.

It is a long-term goal of meson to incorporate this functionality natively, and also add a native pep517 build backend. Part of this experiment is meson-python itself, which currently iterates relatively rapidly and is discovering lots of edge cases, and also has some dedicated functionality that is probably out of scope for meson itself (things like patchelf and shared library rebundling? The proposals around pin-compatible dependencies?).

I would like to see a world where a python package can be installed with either python -m build && python -m installer dist/*.whl or meson setup builddir && ninja -C builddir && meson install -C builddir and produce byte-identical results. I think we will get there. But that day is not today.

[eli-schwartz]

A bit of an aside, but I just recently merged a new meson feature that lets it handle all the bytecode compilation that pip / installer also do.

[rgommers]

A few more thoughts:

But, in principle, I think what is happening here is that scipy effectively acquires a "dependency" on a particular PEP-517 front-end. While harmless, this doesn't sounds like a desirable thing either.

The previous paragraph should be a synchronization point, because if we disagree there, the rest probably doesn't make sense either.

I agree with you. It would be very undesirable if SciPy's use of Meson would force using a particular frontend. If that is the case, it should be addressed.

In this particular case, we actually did address it, because it was indeed very annoying. I actually got PEP 517 updated with language that specifies how to handle duplicate keys, and then we updated the pip implementation. It's possible that there may be other such "surprises" in the future, because the config_settings interface is not very battle-tested yet I'd say. Fixing that properly will be the right thing to do though, rather than working around it in meson-python. Because scikit-build-core and other build backends will eventually run into the same problems.

Regarding cross-compilation, I think we're in a similar situation. That's a longer-term effort, but it should be fixed properly in the places where we have issues now (sysconfig, meson at least), so that meson-python remains a simple glue layer that mostly passes through configuration between pip&co and meson.

Regarding the original request here: I think we'd prefer not to support configuring a build dir first and then invoking meson-python to use that build dir. It's hard to test, and given that we do have a need to pass on some build flags, we do need to reconfigure and that's then always going to be more fragile than passing all options in one go.

A downside of that that I've already seen in conda-forge as well is that it requires to forward the relevant arguments that a distro is trying to pass to Meson directly. E.g.: https://github.com/conda-forge/scipy-feedstock/blob/1cb2d28ad746de4ee696757fbfc402c909db6ba4/recipe/build.sh#L17-L23

That seems like a reasonable price to pay though.

[SomeoneSerge]

Thank you so much for the explanation! Now that I have a better idea of how much effort has already been invested in making the pip feed-forward approach work well, and how distributions seem to be cool about it after all, it's hard to disagree that it only makes sense that meson-python limit support to that

given that we do have a need to pass on some build flags, we do need to reconfigure and that's then always going to be more fragile than passing all options in one go

Well, more of a comment than a suggestion or request, but what I'd like to be able to do in this situation is to manually specify these flags meson-python wants, and then have pip wheel/python -m build assert that the flags I used were "right". With all your comments in mind, I am now reflecting if this really just doesn't belong in meson-python

I need some more time to wrap my head around all this, but I'll probably end up closing this issue

[rgommers]

Cc @domenkozar, who brought up the issue of running meson with Nix helper functions before invoking meson-python earlier today (perhaps not completely coincidentally?).

[jameshilliard]

There's no technical reason why you can't install a project such as SciPy with

Yeah, so for buildroot I just converted scipy to a native meson package since meson-python doesn't appear to allow passing the correct meson setup args needed for cross compilation to meson.

I guess ideally there would be a way to disable all the broken(for us when cross compiling) setup args guessing meson-python does and have it just pass all the correct setup args from our meson infrastructure through meson-python to meson so that the meson build works like normal meson builds.

[dnicolodi]

I guess ideally there would be a way to disable all the broken(for us when cross compiling) setup args guessing meson-python does and have it just pass all the correct setup args from our meson infrastructure through meson-python to meson so that the meson build works like normal meson builds.

meson-python does not pass any setup argument to meson, except the ones documented here https://meson-python.readthedocs.io/en/stable/explanations/default-options.html#explanations-default-options and none of these is the result of guessing. All of these parameters can be overwritten, except for --native-fille that can only be appended to. However, the only content of the native file is the location of the pyhton interpreter, which is also not the result of guessing, but it is the interpreter that is executing the build. This is mandated by how building Python wheels is specified by the Python ecosystem.

Can you please be more specific about which settings meson-python sets incorrectly?

[jameshilliard]

All of these parameters can be overwritten, except for --native-fille that can only be appended to.

We don't normally set a native file when cross compiling with meson, only a cross file. I noticed meson-python has some logic for cross files but not sure if that will cause issues.

Can you please be more specific about which settings meson-python sets incorrectly?

Well some of the default options are normally variables we set ourselves for meson builds I guess.

[dnicolodi]

I noticed meson-python has some logic for cross files but not sure if that will cause issues.

meson-python do not generate a cross file unless compiling on macOS and the $ARCHFLAGS environment variable is set.

Well some of the default options are normally variables we set ourselves for meson builds I guess.

None of the default options affect cross compilation. Unless you can be more specific, there is nothing we can do to help you. The strong language you used for factually wrong statements is even more irritating when paired with the vague nature of your complains.

[rgommers]

I suspect it's the same as the conda-forge case linked above? Packaging systems typically set a bunch of defaults centrally, from max number of CPU cores to use to forcing release builds to adding certain compiler flags. They tend to do that with environment variables, and some are universally accepted (e.g., CFLAGS) while some others are per build system. For Meson, those env vars then have to be transformed into CLI arguments, because by design Meson and meson-python accept only a limited set of env vars.

Doing the env-vars-to-cli-args transformation is a little tedious, but once it's done everything is less magical and build logs are more complete about what is actually being used, so I'm still reasonably happy with the trade-offs.

[dnicolodi]

@rgommers My interpretation of this statement

I guess ideally there would be a way to disable all the broken (for us when cross compiling) setup args guessing meson-python does and have it just pass all the correct set

is very different from yours. However, we didn't get any detail about what are the wrong arguments meson-python allegedly passes to meson. I tend to believe that the issue is a misunderstanding of how meson-python is to be used.

Unless you find something actionable on the meson-python side in this issue, I'm going to close it.

[rgommers]

Unless you find something actionable on the meson-python side in this issue, I'm going to close it.

I agree - I re-read this whole issue top to bottom, and there's valuable content/context here, but nothing that needs to change in meson-python itself at the moment. So let's close this. If anyone has a more concrete targeted proposal for a change that is desirable, please open a new issue.

The original topic about passing config parameters and a concern about the interface being specific to pypa/build was addressed, and is covered well in the docs: https://meson-python.readthedocs.io/en/latest/how-to-guides/meson-args.html. And the tracking issue for cross-compilation is gh-321.