MAINT: prepare for SciPy 1.8.0rc4

[tylerjereddy]

• backport the two merged PRs with a backport label

• this needs to fix the remaining conda-forge PROPACK issues

• @mckib2 I don't see a submodule change here? was the maintenance branch already ahead of main in that regard? Not sure I fully trust current submodule settings/situation, so good to double check we have the right submodule commit in use here (seems to be the latest one you can get locally).

• cc @h-vetinari

• I'm still slightly confused about how PROPACK related issues could remain at this point--my objective was to turn off (skip) all propack related tests in rc3 with the env variable--I still think some are not getting skipped properly as I noted in the rc3 backport PR

[tylerjereddy]

For the submodule business, it looks like that snuck in with the REL commit before this to bump to rc4, which wasn't really intentional (I thought we added a guard to make this harder to do...):

Anyway, the net result there is acceptable I suppose.

[tylerjereddy]

Things I still plan to check here:

• scipy/sparse/linalg/tests/test_propack.py tests need to be skipping when USE_PROPACK is not set; I was convinced this was true already for test_svds, but never really understood why the skip was supposed to happen for test_propack.py, and indeed the segfault on OSX conda-forge seemed to happen in that set of tests (the MKL backport may solve this, but is orthogonal to the intention--those tests shouldn't be running at all unless the user opts in to use PROPACK at runtime); I'm almost certain that I'm going to restore the skip I wanted to add in the rc3 backport PR here: MAINT: release branch PROPACK switch (default off) #15432 (using the environment variable, NOT the import check since we can always import)
• .gitmodules submodule update machinery is consistent with main (more PROPACK patches may come for Windows support on maintenance branch, having this work reliably would be useful)

For remaining failures on conda-forge:

• the first point above and the MKL patch here should prevent the segfaults in two ways for OSX + MKL
• for Windows, conda-forge needs to turn off pythran, we gave up on trying to support binaries with pythran on Windows in the short term with distutils mysteries
• for their aarch64 linux failure, it is a single test failure I haven't seen before, maybe they can vendor a patch to skip one test

[tylerjereddy]

This does what I want now re: PROPACK skipping. python runtests.py -t "scipy/sparse/linalg/tests/test_propack.py" -- -rsx doesn't run a single test when USE_PROPACK is not set. It properly runs 40 tests when USE_PROPACK is set.

I double checked the diff here--looks right to me. If CI passes, I will merge and proceed with the rc4 release.

[tylerjereddy]

Ok, two timeouts in CI are just the usual for this maintenance branch, so let's merge and hope we don't need a fifth rc.

[tylerjereddy]

1.8.0rc4 wheel builds are underway; I'm a little concerned about some activity I saw in the NumPy wheel repo related to MacOS Azure changes--perhaps there's a decent chance I'm going to have replicate the NumPy wheels patches as well.. (cc @charris ?).

This is the patch I may need to "mimic" for our wheels: MacPython/numpy-wheels#149

[tylerjereddy]

well, the Azure MacOS jobs seem to be passing.. so maybe SciPy is "ok" for this for now...

[h-vetinari]

So the mechanism is now working, but I'm now wondering if the environment variable shouldn't be called SCIPY_USE_PROPACK rather than USE_PROPACK...? Too late for 1.8.0, but just thinking out loud...

[tylerjereddy]

more encouragement to fix the problem and delete the variable entirely I suppose