MAINT: wheels 1.8.1 prep

[tylerjereddy]

• restore Pythran for Windows builds to see
  if we are good to go there (if so, we can close
  BUG: cl.exe used instead of clang-cl.exe for Windows pythran building #162 as well)

• bump BUILD_COMMIT to point to the latest
  relevant maintenance branch--this should also
  tell me if anything strange is happening with
  things that may not be pinned since the 1.8.0 rel

[tylerjereddy]

All Linux test runs on Azure and Travis (various platforms) suffer a single test failure:

___________________________ test_valid_scipy_version ___________________________
[gw0] linux -- Python 3.9.7 /venv/bin/python
/venv/lib/python3.9/site-packages/scipy/_build_utils/tests/test_scipy_version.py:18: in test_valid_scipy_version
    assert_(res is not None, scipy.__version__)
E   AssertionError: 1.8.1.dev0+0.
        dev_suffix = '(\\.dev0\\+.+([0-9a-f]{7}|Unknown))'
        res        = None
        version_pattern = '^[0-9]+\\.[0-9]+\\.[0-9]+(|a[0-9]|b[0-9]|rc[0-9])'

[tupui]

All Linux test runs on Azure and Travis (various platforms) suffer a single test failure:

___________________________ test_valid_scipy_version ___________________________
[gw0] linux -- Python 3.9.7 /venv/bin/python
/venv/lib/python3.9/site-packages/scipy/_build_utils/tests/test_scipy_version.py:18: in test_valid_scipy_version
    assert_(res is not None, scipy.__version__)
E   AssertionError: 1.8.1.dev0+0.
        dev_suffix = '(\\.dev0\\+.+([0-9a-f]{7}|Unknown))'
        res        = None
        version_pattern = '^[0-9]+\\.[0-9]+\\.[0-9]+(|a[0-9]|b[0-9]|rc[0-9])'

I think it is missing a pipe after Unknown so: dev_suffix = '(\.dev0\+.+([0-9a-f]{7}|Unknown|))'. Otherwise it's one or the other and does not work if empty. (I used https://regexr.com/ to see how to make it match)

[tylerjereddy]

I think something is wrong with the wheel building somehow. If I look at the final wheel name produced in Linux CI there is a weird version number with dev0_0.-.. in various places

/opt/python/cp310-cp310/lib/python3.10/site-packages/pkg_resources/__init__.py:123: PkgResourcesDeprecationWarning: 1.8.1.dev0-0. is an invalid version and will not be supported in a future release
Copying SciPy.egg-info to build/bdist.linux-x86_64/wheel/SciPy-1.8.1.dev0_0.-py3.10.egg-info
INFO:auditwheel.main_repair:
Fixed-up wheel written to /tmp/tmp.U2x1LMBydU/SciPy-1.8.1.dev0_0.-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Finishing: Build wheel

For a MacOS job it looks normal:

delocate-wheel /Users/runner/work/1/s/wheelhouse/SciPy-1.8.1.dev0+2313.a6a2fe5-cp310-cp310-macosx_10_9_x86_64.whl

The Linux job I checked uses a crazy-new version of setuptools, so as a first pass I'll try applying the same version constraints as are used in the pyproject.toml to the manual setuptools install for Linux.

[charris]

The version problem looks familiar, NumPy CI had similar problems with some tests that used docker images and I'm sort of expecting them for 1.22.4. See https://github.com/numpy/numpy/pull/21350/files for an example. This may need fixing in multibuild if that is what you are using. @matthew-brett Thoughts?

EDIT: https://github.com/numpy/numpy/pull/21368/files is another example.

[charris]

Another option might be to pin the manylinux docker version. I think the problem started upstream a couple of days ago, but didn't test that solution because the other fixes worked.

[tylerjereddy]

Crazy, maybe related to this Git vulnerability: https://github.blog/2022-04-12-git-security-vulnerability-announced/

Maybe I should look into pinning the docker version if I can do that without too much hassle using an env variable or similar, without adjusting upstream.

[charris]

@tylerjereddy That git change sounds right. I did check git releases and didn't see that, but the announcement timing is right.

[charris]

There is a DOCKER_TEST_IMAGE that can be set. If you go to https://quay.io/repository/pypa/manylinux2014_i686?tab=tags&tag=latest, you can find the docker images and if you hit the download button on the right of a version you can get something like docker pull quay.io/pypa/manylinux2014_i686:2022-04-03-da6ecb3 which can be edited to get the pinned version. I recall doing that once, but don't remember the details or if it worked. Images for the other manylinux versions can also be found if you edit the link above.

[charris]

I opened a multibuild issue for this: multi-build/multibuild#470

[tylerjereddy]

Thanks Chuck, yeah I'm just working on other stuff at the moment, but I'll think about some of your suggestions.

[tylerjereddy]

Ok, let's see what happens with a pin to a March 31 version of the manylinux2014 images.

[tylerjereddy]

Hmm, I suspect this may need to be fixed upstream--for example, not only is there an issue using the hardcoded docker image, but the version number is already corrupted from the "build" of the wheel before DOCKER_TEST_IMAGE even becomes relevant. There is also the DOCKER_IMAGE env variable.

[tylerjereddy]

Well, another option may be to try to stick the git command in some appropriate location(s) in i.e., config.sh custom funcs.

[mattip]

• The linux builds on ARM64 as well as githbu actions are emitting UserWarning: The version specified ('1.8.1.dev0+0.') is an invalid version, this may not work as expected with newer versions of setuptools, pip, and PyPI. Please see PEP 440 for more details.. Then a test to check the version is failing.

[tylerjereddy]

I've cut the CI down to a single Linux Azure entry while I debug with some of the recent suggestions.

[tylerjereddy]

I can reproduce locally if I run python setup.py install when I'm a user that is not the same as the user who owns the scipy directory I'm working in.

And I can also reproduce that git config --global --add safe.directory /home/treddy/github_projects/dummy/scipy does absolutely nothing to help:

cat scipy/version.py:

# THIS FILE IS GENERATED FROM SCIPY SETUP.PY
short_version = '1.8.1'
version = '1.8.1'
full_version = '1.8.1.dev0+0.'
git_revision = ''
commit_count = '0'
release = False

if not release:
    version = full_version

Why isn't the command helping? Not sure, and amazingly, it doesn't even help if I add the option on both users involved. Only running as the true/genuine owning user works for me at the moment.

[henryiii]

For a quick fix, I think the safe directory can be *? You can also see my comments in the other issues, namely you can set GIT_DIR manually.

[tylerjereddy]

Alright, Henry's code change suggestion seems to help locally, so now temporarily pointing this PR at my fork and a commit that modifies setup.py per scipy/scipy#16139, to see if the single Azure job currently set to run will pass.

[tylerjereddy]

Given the success of Henry's suggestion, I've effectively restored the original diff that simply points to the latest version of the maintenance branch and restores Pythran usage on Windows. I'll let this run overnight--fingers crossed.

[tylerjereddy]

the one windows failure is sporadic, restarted

[tylerjereddy]

The single test failure in 1/5 Appveyor builds is just the E PermissionError: [WinError 5] Access is denied, and while it persisted after a restart, I've seen that test fail sporadically for like year I think, so I'm going to merge this and move on. When doing the actual release I have to restart the failing job(s) until they pass but not point doing that here.