python3.pkgs.scipy: 1.10.1 -> 1.11.1 & more #239969

doronbehar · 2023-06-26T16:25:34Z

Description of changes

Update some dependencies of scipy, and added:

Cross compilation support.
Support for overriding blas - close python3Packages.scipy can't be build with other blasProvider than openBlas #206866.

Things done

doronbehar · 2023-06-27T10:02:15Z

It compiles, and I even added blas and lapack overriding support as requested at #206866 . Currently I test cross compilation of it using:

nix build -Lf. pkgsCross.armv7l-hf-multiplatform.python3.pkgs.scipy

doronbehar · 2023-06-27T11:22:02Z

Tested both armv7l-hf-multiplatform.python3.pkgs.scipy and aarch64-multiplatform.python3.pkgs.scipy and they both compile and seem to generate correct binaries.

FRidh

Great work getting this going!

I'm of the opinion we should not be overriding the build phase. Adding pipBuildFlags is a small change now. The downside of course is the mass rebuild making it a bit time consuming to test.

pkgs/development/python-modules/scipy/default.nix

doronbehar · 2023-06-28T07:51:18Z

The pipBuildFlags thing works. Next step is to introduce meson-python.setupHooks.

pkgs/development/python-modules/scipy/default.nix

doronbehar · 2023-06-29T08:21:43Z

Now using meson.setupHook and mesonFlags, the expression looks much cleaner, though still there is no meson-python.setupHook. I also added a better comment why we don't use --config-settings=setup-args and instead why we run meson setup manually. Once pip 23.1 or above lands in Nixpkgs, it shouldn't be hard to create meson-python.setupHook that will set pipBuildFlags with proper --config-settings=setup-args arguments.

@FRidh or @SomeoneSerge if you'd approve the changes, I can squash some of the commits and make the history cleaner as well.

pkgs/development/libraries/xsimd/default.nix

pkgs/development/python-modules/Cython/default.nix

pkgs/development/python-modules/meson-python/default.nix

pkgs/development/python-modules/scipy/default.nix

Diff: serge-sans-paille/pythran@0.11.0...0.13.1

Changelog: https://github.com/mesonbuild/meson-python/blob/0.13.1/CHANGELOG.rst

Diff: scipy/scipy@v1.10.1...v1.11.1 Significant changes to expression: - Use `pypaBuildHook` to build it. - Use `fetchFromGitHub` instead of `fetchPypi`. - Support cross compilation using `pypaBuildHook` and `$mesonFlags`. - Add a `passthru.updateScript` that downloads and updates hashes for datasets needed for tests. - Cleanup: * Remove unneeded anymore `setupPyBuildFlags`. * Remove unnecessary `sed` command from preConfigure that fixed a Numpy issue in `setup.py` * Use `blas` and `lapack` directly, relying on them being overriden by the user in all of Python's packages' set - see commit 213d084 .

doronbehar · 2023-07-18T11:45:29Z

I rebased the git history a bit, with almost a single commit per package. If there are no objections, with your permissions I'd like to merge this today / tomorrow.

SuperSandro2000 · 2023-07-18T12:57:32Z

I would like to cherry-pick this into python-updates unless there are review comments left we need to consider.

SuperSandro2000 · 2023-07-18T16:28:53Z

Thanks, cherry-picked into #244135

mweinelt · 2023-07-30T01:22:14Z

This broke scipy on x86_64-darwin. PTAL. cc #245935

https://hydra.nixos.org/build/229710588
https://hydra.nixos.org/build/229711355

doronbehar · 2023-07-30T06:46:34Z

This broke scipy on x86_64-darwin. PTAL. cc #245935

hydra.nixos.org/build/229710588
hydra.nixos.org/build/229711355

@mweinelt these links don't seem related to scipy, putting here a link that seems more relevant to me:

https://hydra.nixos.org/eval/1797964?filter=scipy&compare=1797817&full=#tabs-now-fail

doronbehar · 2023-07-30T06:50:59Z

A fix is possible at #246152 , will need to wait for ofborg to check it...

mweinelt · 2023-07-30T11:20:22Z

Please recheck the build steps for the linked builds. They are very much related.

https://hydra.nixos.org/log/cwpjczjxjav1kyq5p1ilav93rkl0gd3a-python3.10-scipy-1.11.1.drv

doronbehar · 2023-07-30T12:51:51Z

Please recheck the build steps for the linked builds. They are very much related.

hydra.nixos.org/log/cwpjczjxjav1kyq5p1ilav93rkl0gd3a-python3.10-scipy-1.11.1.drv

I see. I'm still somewhat confused about the logs there (why is this aarch64 log not failing like the others?). But in any case, in #246152 as well, the build fails like you linked, and I have no idea how to fix that issue :/. It's also hard to debug this without a real darwin machine. Maybe contact upstream?

rgommers · 2023-07-30T13:15:12Z

Maybe contact upstream?

I'm already paying attention here.

In file included from scipy/stats/_stats_pythran.cpp:34:
In file included from /nix/store/97hrm3g9b47ishjflg1nhrp07sfnvijl-python3.10-pythran-0.13.1/lib/python3.10/site-packages/pythran/pythonic/include/numpy/ceil.hpp:8:
In file included from /nix/store/97hrm3g9b47ishjflg1nhrp07sfnvijl-python3.10-pythran-0.13.1/lib/python3.10/site-packages/pythran/xsimd/xsimd.hpp:55:
/nix/store/97hrm3g9b47ishjflg1nhrp07sfnvijl-python3.10-pythran-0.13.1/lib/python3.10/site-packages/pythran/xsimd/arch/xsimd_scalar.hpp:474:16: error: use of undeclared identifier '__exp10f'
        return __exp10f(x);
               ^
/nix/store/97hrm3g9b47ishjflg1nhrp07sfnvijl-python3.10-pythran-0.13.1/lib/python3.10/site-packages/pythran/xsimd/arch/xsimd_scalar.hpp:478:16: error: use of undeclared identifier '__exp10'; did you mean 'exp10'?
        return __exp10(x);
               ^~~~~~~
               exp10

It looks like the problem is that xsimd is too old:

Run-time dependency xsimd found: YES 9.0.1

If you upgrade xsimd to 11.0 the problem should disappear.

doronbehar · 2023-07-30T13:21:40Z

Thanks @rgommers I updated #246152 to include that update. Let's see if our darwin CI says it's OK.

doronbehar requested a review from FRidh June 26, 2023 16:25

github-actions bot added the 6.topic: python label Jun 26, 2023

ofborg bot added 10.rebuild-darwin: 501+ 10.rebuild-darwin: 5001+ 10.rebuild-linux: 501+ 10.rebuild-linux: 5001+ labels Jun 26, 2023

doronbehar force-pushed the pkg/scipy branch from ea8c861 to 54079dc Compare June 27, 2023 09:47

doronbehar marked this pull request as ready for review June 27, 2023 11:20

FRidh reviewed Jun 28, 2023

View reviewed changes

pkgs/development/python-modules/scipy/default.nix Outdated Show resolved Hide resolved

pkgs/development/python-modules/scipy/default.nix Outdated Show resolved Hide resolved

pkgs/development/python-modules/scipy/default.nix Outdated Show resolved Hide resolved

This was referenced Jun 28, 2023

pip-build-hook.sh: Support $pipBuildFlags #240288

Closed

WIP: python3Packages.scipy: allow overriding BLAS #230131

Draft

doronbehar requested review from jonringer and mweinelt as code owners June 28, 2023 07:50

github-actions bot added the 8.has: documentation label Jun 28, 2023

ofborg bot requested review from FRidh and tobim June 28, 2023 08:52

SomeoneSerge reviewed Jun 28, 2023

View reviewed changes

pkgs/development/python-modules/scipy/default.nix Outdated Show resolved Hide resolved

doronbehar force-pushed the pkg/scipy branch from 23ad648 to 1e5a4a1 Compare June 29, 2023 08:21

SuperSandro2000 reviewed Jun 29, 2023

View reviewed changes

FRidh requested changes Jun 30, 2023

View reviewed changes

pkgs/development/python-modules/scipy/default.nix Show resolved Hide resolved

pkgs/development/python-modules/scipy/default.nix Outdated Show resolved Hide resolved

pkgs/development/python-modules/scipy/default.nix Outdated Show resolved Hide resolved

doronbehar changed the title ~~python3.pkgs.scipy: 1.10.1 -> 1.11.0 & more~~ python3.pkgs.scipy: 1.10.1 -> 1.11.1 & more Jul 12, 2023

doronbehar force-pushed the pkg/scipy branch from 5da4bd9 to cdc3a4b Compare July 12, 2023 13:01

doronbehar requested review from SomeoneSerge and FRidh July 12, 2023 13:01

ofborg bot added 10.rebuild-darwin-stdenv 8.has: package (new) labels Jul 12, 2023

rgommers reviewed Jul 14, 2023

View reviewed changes

pkgs/development/python-modules/scipy/default.nix Outdated Show resolved Hide resolved

doronbehar mentioned this pull request Jul 15, 2023

Update request: python3Packages.pip 23.0.1 → 23.1 #227364

Closed

1 task

doronbehar added 9 commits July 18, 2023 14:31

doc/python: Demonstrate how to override the blas implementation

213d084

xsimd: Fix cross compilation

959919e

pip-build-hook.sh: Support $pipBuildFlags

f1fdb90

python3.pkgs.pypaBuildHook: init

434e978

python3.pkgs.pythran: 0.11.0 -> 0.13.1

3aa7291

Diff: serge-sans-paille/pythran@0.11.0...0.13.1

python3.pkgs.meson-python: 0.12.1 -> 0.13.1

976f3cf

Changelog: https://github.com/mesonbuild/meson-python/blob/0.13.1/CHANGELOG.rst

python3.pkgs.meson-python: populate python build flags with $mesonFlags

6361504

python3.pkgs.scipy: Add doronbehar as maintainer

02b8fe6

doronbehar force-pushed the pkg/scipy branch from a008a8a to 1b10073 Compare July 18, 2023 11:44

SuperSandro2000 added this to In progress in Python batch upgrade via automation Jul 18, 2023

This was referenced Jul 18, 2023

python311Packages.meson-python: 0.12.1 -> 0.13.2, python311Packages.python-rtmidi: 1.4.9 -> 1.5.4, ledfx: 2.0.67 -> 2.0.69 #240791

Closed

Python updates 2023-07-18 #244135

Merged

SuperSandro2000 closed this Jul 18, 2023

Python batch upgrade automation moved this from In progress to Done Jul 18, 2023

doronbehar mentioned this pull request Jul 20, 2023

Add pipInstallFlags to all pip install invocations in python hooks #244444

Closed

12 tasks

doronbehar mentioned this pull request Jul 30, 2023

python3.pkgs.scipy: Set HOME to a writeable directory #246152

Closed

12 tasks

doronbehar mentioned this pull request Aug 1, 2023

python3.pkgs.scipy ignores when tests failed #246453

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

python3.pkgs.scipy: 1.10.1 -> 1.11.1 & more #239969

python3.pkgs.scipy: 1.10.1 -> 1.11.1 & more #239969

doronbehar commented Jun 26, 2023 •

edited

doronbehar commented Jun 27, 2023

doronbehar commented Jun 27, 2023

FRidh left a comment

doronbehar commented Jun 28, 2023

doronbehar commented Jun 29, 2023

doronbehar commented Jul 18, 2023

SuperSandro2000 commented Jul 18, 2023

SuperSandro2000 commented Jul 18, 2023 •

edited

mweinelt commented Jul 30, 2023 •

edited

doronbehar commented Jul 30, 2023 •

edited

doronbehar commented Jul 30, 2023

mweinelt commented Jul 30, 2023 •

edited

doronbehar commented Jul 30, 2023

rgommers commented Jul 30, 2023

doronbehar commented Jul 30, 2023

python3.pkgs.scipy: 1.10.1 -> 1.11.1 & more #239969

python3.pkgs.scipy: 1.10.1 -> 1.11.1 & more #239969

Conversation

doronbehar commented Jun 26, 2023 • edited

Description of changes

Things done

doronbehar commented Jun 27, 2023

doronbehar commented Jun 27, 2023

FRidh left a comment

Choose a reason for hiding this comment

doronbehar commented Jun 28, 2023

doronbehar commented Jun 29, 2023

doronbehar commented Jul 18, 2023

SuperSandro2000 commented Jul 18, 2023

SuperSandro2000 commented Jul 18, 2023 • edited

mweinelt commented Jul 30, 2023 • edited

doronbehar commented Jul 30, 2023 • edited

doronbehar commented Jul 30, 2023

mweinelt commented Jul 30, 2023 • edited

doronbehar commented Jul 30, 2023

rgommers commented Jul 30, 2023

doronbehar commented Jul 30, 2023

doronbehar commented Jun 26, 2023 •

edited

SuperSandro2000 commented Jul 18, 2023 •

edited

mweinelt commented Jul 30, 2023 •

edited

doronbehar commented Jul 30, 2023 •

edited

mweinelt commented Jul 30, 2023 •

edited