MAINT: 1.9.2 backports #17132

tylerjereddy · 2022-10-02T20:49:27Z

Try to get ready for SciPy 1.9.2 and release of Python 3.11 (and other) wheels via cibuildwheel infrastructure. Merge conflicts related to build system infrastructure were extreme--any PR with a backport label that modified code that doesn't even exist on the maintenance branch had its backport label stripped and a comment added to the PR explaining why.

In cases where I saw that I could cleanly apply a patch to the build system, but the context in flanking code had changed radically, I applied the patch nonetheless--for example, many meson data structures were using things like py3_dep on maintenance branch but neither before nor after on the labeled backport PRs--in these cases I apply the patch directly and assume that the context change is safe (otherwise the burden of reinterpretation of the code would be huge).

Also, remember that this branch still uses do.py not dev.py, to give you an idea of how much stuff has drifted..

Backports currently included (those excluded should have a recent comment from me, I'll go back and check notifications to see what else needs to be backported... there will be more..):

Co-authored-by: Julien Jerphanion <git@jjerphan.xyz> Co-authored-by: Pamphile Roy <roy.pamphile@gmail.com> Co-authored-by: Thomas J. Fan <thomasjpfan@gmail.com>

We're interested in moving to the modern `cibuildwheel` machinery for the release/wheel build process, following NumPy/pandas, etc. [skip actions] [skip circleci] [skip ci] [skip azp]

Closes scipygh-16935

…dims=False` (scipy#16954)

This resulted in `{py_purelib}/` entries in `<build-dir>/meson-info/intro-install_plan.json`, and there should be zero of those. [skip azp] [skip circle]

Issue materialized with MSVC as: ``` cl : Command line error D8021 : invalid numeric argument '/Wno-cpp' ``` [skip actions] [skip circle]

[skip circle]

[skip actions] [skip circle]

If the detection happens via CMake, that results in an upper-case result for `blas.name()`. So deal with any case, and also that of using `-Dblas=mkl_rt`. Note that this does not fix things for using CMake, because that defaults to the ILP64 interface (xref scipygh-16988). Add notes on what is left for MKL support Also add g77 ABI wrapper and fix a typo in f2py signature file in `_propack` Note that this is an incomplete fix, because PROPACK is in a bad state, as discussed in the issue linked in the code comment.

[ci skip]

Update the cibuildwheel version used in the wheels.yml configuration from 2.9.0 to 2.10.1. This is mainly usefull for the updated CPython 3.11 version (from 3.11.0rc1 to v3.11.0rc2). [wheel build]

Resolve the large precision error of truncnorm.logcdf with upper tail argument. mass_case_central of _log_gauss_mass was inaccurate due to catastrophic cancellation in logsumexp.

…tion

FIX: ensure a hold on GIL before raising warnings/errors

Require "meson-python>=0.9.0", "Cython>=0.29.32", "pybind11>=2.10.0" and "pythran>=0.12.0" to build SciPy. These updated build-system requirements allow for consistent builds. The updated meson-python and Cython helps with Python 3.11 builds. Probably a backport candidate to the 1.9 branch if Python 3.11 wheels are to be released on this branch. (note that build-system requirements are only needed to build SciPy from source, not to run it after a regular pip install, and thus can be set fairly aggressive). [wheel build]

doc/release/1.9.2-notes.rst

scipy/optimize/meson.build

scipy/integrate/meson.build

mdhaber · 2022-10-03T00:49:45Z

@tylerjereddy gh-16814 and gh-17055 just merged, can we add them here? Shall I open a PR against your branch, push here, or let you do it?

* clean up some of my own mistakes from merge conflict resolution in the build system

tylerjereddy · 2022-10-04T02:32:38Z

Eh, there are still meson build issues, but I've cleaned things up a little at least. I'm still not fully convinced on the safety of the backports vs. the degree of change around those backports, but I'll try to take another look tomorrow. Some pyproject.toml stuff may need to be reverted as well, wheel venv issue is familiar...

rgommers · 2022-10-04T10:24:59Z

FAILED ../../build-install/lib/python3.10/site-packages/scipy/linalg/tests/test_lapack.py::test_gges_tgexc[float32]
FAILED ../../build-install/lib/python3.10/site-packages/scipy/linalg/tests/test_decomp.py::TestQZ::test_qz_single

That requires backporting the relevant fix from gh-17057 (EDIT: done).

The doc build is failing on a single Matplotlib deprecation warning for set_tight_layout. Not sure why it's not erroring on main, but that one can be solved by pinning Matplotlib or updating the code with the suggested replacement. EDIT: see gh-17144

I will have a look at the build failures and push fixes for them.

This will solve the failure of the "build from sdist" job in Azure.

(cherry picked from commit 076a1f0)

…tive byte order. (scipy#17035) (cherry picked from commit 29ec80b)

rgommers · 2022-10-04T11:13:46Z

Eh, there are still meson build issues, but I've cleaned things up a little at least. I'm still not fully convinced on the safety of the backports vs. the degree of change around those backports, but I'll try to take another look tomorrow.

I think it's pretty close now. It is quite a bit of churn, but I don't see a good alternative for Python 3.11 unless we pull the 1.10.0 release way forward - which is likely a lot more work.

See scipygh-16949

See scipygh-16926 (cherry picked from commit 81f4a4c)

There's a PendingDeprecationWarning for this introduced in Matplotlib 3.6 (released 16 Sep 2022). It should be fine to depend on latest Matplotlib in this one example. [skip azp] [skip actions]

Brings these more in line with `main`

[wheel build]

rgommers · 2022-10-04T13:05:36Z

Okay, all the regular CI should work now, and I also pushed a change to CI triggering for wheels.yml, so the wheel builders are running now on this PR when you add [wheel build] to the commit message.

If this is all green, I think the PR is ready to merge, and after that it needs one more commit to set the version number to 1.9.2 and push the tag.

tylerjereddy · 2022-10-04T15:09:27Z

Interesting, did you fix the MacOS Python 3.11 certificate issue somehow or did that just go away with a dependency update somewhere?

rgommers · 2022-10-04T15:14:06Z

Interesting, did you fix the MacOS Python 3.11 certificate issue somehow or did that just go away with a dependency update somewhere?

See #17132 (comment)

rgommers

Even aarch64 on QEMU seems happy. The one thing that this PR does not yet do is uncomment the Upload wheels section of wheels.yml. I think that can be done separately - or in this PR without re-testing. Let's just confirm that the code is identical there between main and 1.9.x.

mdhaber · 2022-10-04T18:46:57Z

Can we get these two bug fixes in? #17132 (comment)

tylerjereddy · 2022-10-04T19:34:46Z

@mdhaber I may do those separately but will try to get them in--the diff here is getting close to 4,000 lines. Let me try to skim this PR over one more time then I'll probably merge it based on Ralf's recommendation/fixes.

mdhaber · 2022-10-04T20:15:19Z

Great, thanks!

Micky774 and others added 21 commits October 2, 2022 13:37

FIX: Updated dtype resolution in _stack_along_minor_axis (scipy#16628)

12327ba

Co-authored-by: Julien Jerphanion <git@jjerphan.xyz> Co-authored-by: Pamphile Roy <roy.pamphile@gmail.com> Co-authored-by: Thomas J. Fan <thomasjpfan@gmail.com>

ENH: cibuildwheel infrastructure (scipy#16842)

0204a3d

We're interested in moving to the modern `cibuildwheel` machinery for the release/wheel build process, following NumPy/pandas, etc. [skip actions] [skip circleci] [skip ci] [skip azp]

MAINT: minimize, restore squeezed ((1.0))

461b992

Cast linear_sum_assignment to PyCFunction

705ee88

BLD: use compiler flags in a more portable way.

da01ada

Closes scipygh-16935

MAINT: stats.mode: fix bug with axis!=1, nan_policy='omit', `keep…

670782a

…dims=False` (scipy#16954)

BLD: fix usage of get_install_data, which defaults to purelib

0df827a

This resulted in `{py_purelib}/` entries in `<build-dir>/meson-info/intro-install_plan.json`, and there should be zero of those. [skip azp] [skip circle]

DOC: Update numpy supported versions for 1.9.2

08cc962

BUG: missed one more gcc-specific flag, clean it up

74d8dfe

Issue materialized with MSVC as: ``` cl : Command line error D8021 : invalid numeric argument '/Wno-cpp' ``` [skip actions] [skip circle]

BLD: compensate for funky ifort name mangling on Windows

b18c9be

[skip circle]

BLD: use native threads on Windows, not pthreads

6ee0091

[skip actions] [skip circle]

TST: cibuildwheel test manylinux_aarch64

0fd485e

DOC: fix a formatting issue in the building FAQ doc page

466cb94

[ci skip]

CI: Update cibuildwheel to 2.10.1

9447b60

Update the cibuildwheel version used in the wheels.yml configuration from 2.9.0 to 2.10.1. This is mainly usefull for the updated CPython 3.11 version (from 3.11.0rc1 to v3.11.0rc2). [wheel build]

BUG: Fix numerical precision error of truncnorm.logcdf

7c97930

Resolve the large precision error of truncnorm.logcdf with upper tail argument. mass_case_central of _log_gauss_mass was inaccurate due to catastrophic cancellation in logsumexp.

MAINT: stats.truncnorm.logcdf/logsf: fix another catastropic cancella…

b04b7eb

…tion

FIX: ensure a hold on GIL before raising warnings/errors (scipy#17096)

e079338

FIX: ensure a hold on GIL before raising warnings/errors

TST: stats.studentized_range: fix incorrect test

58c8b79

DOC: update 1.9.2 relnotes.

9c923e6

tylerjereddy added the maintenance Items related to regular maintenance tasks label Oct 2, 2022

tylerjereddy added this to the 1.9.2 milestone Oct 2, 2022

tylerjereddy requested review from person142, peterbell10, perimosocordiae, larsoner, ilayn, andyfaff and ev-br as code owners October 2, 2022 20:49

tylerjereddy requested a review from rgommers as a code owner October 2, 2022 20:49

tylerjereddy commented Oct 2, 2022

View reviewed changes

doc/release/1.9.2-notes.rst Show resolved Hide resolved

tylerjereddy commented Oct 2, 2022

View reviewed changes

scipy/optimize/meson.build Outdated Show resolved Hide resolved

tylerjereddy commented Oct 2, 2022

View reviewed changes

scipy/integrate/meson.build Show resolved Hide resolved

MAINT: PR 17132 revisions

f3ca38d

* clean up some of my own mistakes from merge conflict resolution in the build system

rgommers and others added 3 commits October 4, 2022 12:37

BLD: remove Meson pin, newer versions have fixes we need

d885c55

This will solve the failure of the "build from sdist" job in Azure.

BLD: add _USE_MATH_DEFINES for all Cython-generated code

dc1ae86

(cherry picked from commit 076a1f0)

BUG: sparse: Avoid creating a view when ensuring that an array has na…

35138da

…tive byte order. (scipy#17035) (cherry picked from commit 29ec80b)

rgommers added 3 commits October 4, 2022 14:53

TST: mark float32 gges and qz tests as knownfail

54e27b3

See scipygh-16949

TST: mark one linalg.solve_discrete_are as knownfail

50c85e6

See scipygh-16926 (cherry picked from commit 81f4a4c)

DOC: replace set_tight_layout with set_layout_engine in example

5a793e9

There's a PendingDeprecationWarning for this introduced in Matplotlib 3.6 (released 16 Sep 2022). It should be fine to depend on latest Matplotlib in this one example. [skip azp] [skip actions]

rgommers force-pushed the treddy_192_backports branch from 40fccfa to 9e4289a Compare October 4, 2022 12:56

rgommers added 2 commits October 4, 2022 15:03

CI: update wheel builder triggers

014ecf6

Brings these more in line with `main`

BLD: set version to 1.9.2.dev0 (and trigger wheel build CI)

2bc973a

[wheel build]

rgommers force-pushed the treddy_192_backports branch from 9e4289a to 2bc973a Compare October 4, 2022 13:03

rgommers approved these changes Oct 4, 2022

View reviewed changes

tylerjereddy merged commit bcfce27 into scipy:maintenance/1.9.x Oct 5, 2022

tylerjereddy deleted the treddy_192_backports branch October 5, 2022 02:38

tylerjereddy mentioned this pull request Oct 5, 2022

MAINT: more 1.9.2 backports prep #17150

Merged

EwoutH mentioned this pull request Oct 7, 2022

Python 3.11 manylinux wheels #16851

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MAINT: 1.9.2 backports #17132

MAINT: 1.9.2 backports #17132

tylerjereddy commented Oct 2, 2022 •

edited

mdhaber commented Oct 3, 2022 •

edited

tylerjereddy commented Oct 4, 2022 •

edited

rgommers commented Oct 4, 2022 •

edited

rgommers commented Oct 4, 2022

rgommers commented Oct 4, 2022

tylerjereddy commented Oct 4, 2022

rgommers commented Oct 4, 2022

rgommers left a comment

mdhaber commented Oct 4, 2022

tylerjereddy commented Oct 4, 2022

mdhaber commented Oct 4, 2022

MAINT: 1.9.2 backports #17132

MAINT: 1.9.2 backports #17132

Conversation

tylerjereddy commented Oct 2, 2022 • edited

mdhaber commented Oct 3, 2022 • edited

tylerjereddy commented Oct 4, 2022 • edited

rgommers commented Oct 4, 2022 • edited

rgommers commented Oct 4, 2022

rgommers commented Oct 4, 2022

tylerjereddy commented Oct 4, 2022

rgommers commented Oct 4, 2022

rgommers left a comment

Choose a reason for hiding this comment

mdhaber commented Oct 4, 2022

tylerjereddy commented Oct 4, 2022

mdhaber commented Oct 4, 2022

tylerjereddy commented Oct 2, 2022 •

edited

mdhaber commented Oct 3, 2022 •

edited

tylerjereddy commented Oct 4, 2022 •

edited

rgommers commented Oct 4, 2022 •

edited