setup.py --with-openmp option not compatible with PEP517 (pyproject.toml) build

[drew-parsons]

Bug Description

setup.py manages a --with-openmp build option to provide openmp support.

pyproject.toml was introduced in QuTiP 4.6.0 to allow PEP517 build without (directly) using setup.py.

But the --with-openmp option that previously worked with a setup.py build does not seem to be compatible with a PEP517 build.

The problem shows up in the Debian build of the qutip package. Debian provides its own tool (pybuild from the dh-python package) to manage the build of python packages. PEP517 is relatively new, so by default pybuild uses setup.py if it is available. pybuild will use PEP517 instead, if the pybuild-plugin-pyproject package is installed and pyproject.toml is available.

Without pybuild-plugin-pyproject installed, the Debian build was invoking

/usr/bin/python3.10 setup.py build --with-openmp

using setup.py and specifying --with-openmp. A successful build log for this method can be found at https://buildd.debian.org/status/fetch.php?pkg=qutip&arch=amd64&ver=4.6.2-3&stamp=1641397514&raw=0

With pybuild-plugin-pyproject installed, the Debian build invokes a wheel build (command line given below). But --with-openmp causes the build to fail.

Code to Reproduce the Bug

python3.10 -m build --skip-dependency-check --no-isolation --wheel --outdir /build/qutip/.pybuild/cpython3_3.10_qutip --with-openmp

Code Output

usage: python -m build [-h] [--version] [--sdist] [--wheel] [--outdir OUTDIR] [--skip-dependency-check] [--no-isolation] [--config-setting CONFIG_SETTING] [srcdir]
python -m build: error: unrecognized arguments: --with-openmp

Expected Behaviour

The PEP517 build should be able to be configured to support openmp.

Your Environment

QuTiP Version:      4.6.2  (trying to build 4.7.0)
Numpy Version:      1.21.5
Scipy Version:      1.8.0
Cython Version:     0.29.28
Matplotlib Version: 3.5.1
Python Version:     3.10.4
Number of CPUs:     4
BLAS Info:          OPENBLAS
OPENMP Installed:   True  (from setup.py build, not PEP517 build)
INTEL MKL Ext:      False
Platform Info:      Linux (x86_64)
Installation path:  /usr/lib/python3/dist-packages/qutip

Additional Context

PEP517 support is new for Debian. It is possible the pybuild build tool needs to be adapted to handle setup.py options like --with-openmp in the context of PEP517. But I've tried to pose the issue here in terms of the underlying python3 build commands without needing to consider pybuild as such.

Is there a different way in which --with-openmp should be configured when using a PEP517 build?

[drew-parsons]

python3 -mbuild has a -C option (--config-setting), seehttps://pypa-build.readthedocs.io/en/latest/. It sounds like it could be used for build configuration options like qutip's openmp support.

It doesn't seem to work with the current handling in setup.py however. Permutations of -C--with-openmp, -Copenmp, -Copenmp=1 enable a succcessful qutip PEP517 build, but qutip.about() then reports

OPENMP Installed:   False

indicating the option is ignored.

[hodgestar]

I have found out how to use this after reading the source:

python -m build -n -x --wheel --config-setting="--global-option=--with-openmp"

[hodgestar]

One can read a little bit of somewhat useful information in PEP517 -- https://peps.python.org/pep-0517/#config-settings. I tried --build-option instead of --global-option, but that didn't work.

[hodgestar]

I've added a note to the source on this in #1978, so I'll close this issue once that is merged.

[drew-parsons]

Not sure if it's that simple.

--config-setting="--global-option=--with-openmp"

actually gives me a build error:

$ python3.10 -m build --skip-dependency-check --no-isolation --wheel --config-setting="--global-option=--with-openmp"
* Building wheel...
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/pep517/in_process/_in_process.py", line 363, in <module>
    main()
  File "/usr/lib/python3/dist-packages/pep517/in_process/_in_process.py", line 345, in main
    json_out['return_val'] = hook(**hook_input['kwargs'])
  File "/usr/lib/python3/dist-packages/pep517/in_process/_in_process.py", line 261, in build_wheel
    return _build_backend().build_wheel(wheel_directory, config_settings,
  File "/usr/lib/python3/dist-packages/setuptools/build_meta.py", line 230, in build_wheel
    return self._build_with_temp_dir(['bdist_wheel'], '.whl',
  File "/usr/lib/python3/dist-packages/setuptools/build_meta.py", line 211, in _build_with_temp_dir
    sys.argv = (sys.argv[:1] + setup_command +
TypeError: can only concatenate list (not "str") to list

ERROR Backend subproccess exited when trying to invoke build_wheel

The other method

--config-setting="--build-option=--with-openmp"

appears to work in the sense that it builds successfully. But after installing the package it reports that openmp is still missing

$ python3 -c "import qutip; qutip.about()"

QuTiP: Quantum Toolbox in Python
================================
Copyright (c) QuTiP team 2011 and later.
Current admin team: Alexander Pitchford, Nathan Shammah, Shahnawaz Ahmed, Neill Lambert, Eric Giguère, Boxi Li, Jake Lishman and Simon Cross.
Board members: Daniel Burgarth, Robert Johansson, Anton F. Kockum, Franco Nori and Will Zeng.
Original developers: R. J. Johansson & P. D. Nation.
Previous lead developers: Chris Granade & A. Grimsmo.
Currently developed through wide collaboration. See https://github.com/qutip for details.

QuTiP Version:      4.7.0
Numpy Version:      1.21.5
Scipy Version:      1.8.1
Cython Version:     0.29.30
Matplotlib Version: 3.5.2
Python Version:     3.10.6
Number of CPUs:     8
BLAS Info:          OPENBLAS
OPENMP Installed:   False
INTEL MKL Ext:      False
Platform Info:      Linux (x86_64)
Installation path:  /usr/lib/python3/dist-packages/qutip
================================================================================
Please cite QuTiP in your publication.
================================================================================
For your convenience a bibtex reference can be easily generated using `qutip.cite()`
``

[drew-parsons]

--config-setting=--global-option=--with-openmp is still not working with git head (with #1978 merged). This is with

• python 3.10.6-1
• python3-build 0.7.0-3
• python3-setuptools 59.6.0-1.2
• python3-wheel 0.37.1-2
• gcc 12.1.0-7

[hodgestar]

@drew-parsons Is this still an issue for the Debian package? I'm not quite sure how to reproduce my side, but I am cutting the 4.7.1 release soon, so this would be a good time to address this if we can.

[drew-parsons]

Looks like it's resolved now (unless documentation should be updated on how to use the --with-openmp option with the new build method. I'm not sure if end users should now be using wheels builds).

Both --config-setting="--global-option=--with-openmp" and --config-setting="--build-option=--with-openmp" achieve a wheels (PEP517) build and python -c "import qutip; qutip.about()" reports OPENMP Installed: True for both python3.10 and python 3.11.

--global-option gets a warning message:

$ python3.10 -m build --skip-dependency-check --no-isolation --wheel --outdir /projects/misc/build/qutip/.pybuild/cpython3_3.10_qutip --config-setting="--global-option=--with-openmp"
* Building wheel...
/usr/lib/python3/dist-packages/setuptools/build_meta.py:307: SetuptoolsDeprecationWarning: 
            The arguments ['--with-openmp'] were given via `--global-option`.
            Please use `--build-option` instead,
            `--global-option` is reserved to flags like `--verbose` or `--quiet`.
            
  warnings.warn(msg, SetuptoolsDeprecationWarning)
...

So it's saying we should be using --build-option not --global-option