Python: build wheels in separate derivation

[FRidh]

Describe the bug

Perform wheel building and wheel installing in two separate derivations.

Additional context

Builds of Python wheels typically have only very few dependencies. When installing the wheel, wrapping needs to be done and propagated build inputs are added to the store resulting in a much larger closure.

By putting the building in a separate derivation, we can avoid rebuilds which is especially interesting for larger builds.

Related issue for tests #26400

[FRidh]

After 22.05 I want to pick up #102613 again and merge that. Once that is done, I will split the building and testing. buildPython* will then get two new parameters, separateBuild and separateCheck.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nixpkgss-current-development-workflow-is-not-sustainable/18741/75

[SomeoneSerge]

Is it the intention to make all python builds follow the "anything-to-wheel, wheel to /nix/store" path?
What about format = "other"?

[FRidh]

in mk-python-derivation.nix the src there would point to a new derivation in case separateBuild == true; and format == pyproject. We could also allow setuptools for it but we should be migrating to pyproject. Tricky part is filtering the parameters to be passed on.

Non-wheel builds are rare so I think it's best to keep them in the same derivation and indeed use wheels as artifacts to pass on.

[SomeoneSerge]

flit should be easy too

[FRidh]

flit should be easy too

I highly doubt we have any flit packages left that are not yet compatible with pyproject and pip. We should get rid of that one.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/python-propagatedbuildinputs-could-we-do-better/29035/5

[yajo]

FWIW today I found another use case for this: python circular dependencies.

Here you have an issue from the past where this was a problem for nix: mkdocstrings/mkdocstrings#376

Today I'm hitting it in other project.

The truth is that it's not a problem for python resolvers because they split the build dependencies from the runtime ones. Thus, for building, it won't have circular dependencies; it's usually just setuptools, hatchling, poetry-core, etc. However, for runtime, circulars aren't a problem because all is already built, so you just have to put them all together in the same env.

I guess splitting build and runtime deps in nixpkgs just like pypi does would let us safely support circular dependencies.

[adisbladis]

A few thoughts:

• This likely has massive positive performance impact if done right.

We could possibly get rid of the idiom of packageOverrides.
python.override { packageOverrides = ...; } is very expensive because it rebootstraps the python set every time.

We'd also not have to rebuild just because runtime deps change. This would mean far less rebuilds for Python going forward.

• It probably means giving up on python3Packages.requests giving you a usable package

Shells looking like

let
  pkgs = import <nixpkgs> {};
  inherit (pkgs) python3Packages;
in
pkgs.mkShell {
  packages = [
    python3Packages.requests
  ];
}

are common, but are relying on a bad practice.

The mechanism that makes requests actually importable in this case is setup hook propagation adding requests to $PYTHONPATH.
$PYTHONPATH is problematic for many reasons and composes poorly. This is already an anti-pattern.
When splitting the wheel build it also implies removing the setup hook from the separated derivation, breaking this workflow.

Things like withPackages could walk the list of required python modules in the Nix evaluation instead

• Unifying of dependencies/optional-dependencies
  Python runtime inputs no longer being build inputs means that we would want to unify the structure of Python dependencies:

  passthru.dependencies = [
    brotlicffi
    certifi
    charset-normalizer
    idna
    urllib3
  ];

  passthru.optional-dependencies = {
    security = [];
    socks = [
      pysocks
    ];
    use_chardet_on_py3 = [
      chardet
    ];
  };

[SuperSandro2000]

see #102613