python3.pkgs.buildPythonPackage: Separate runtime & build time dependencies

[adisbladis]

Description of changes

See #272178 for more context.

Dependency separation

Adds a separation between Python runtime dependencies, Python build-time dependencies & non-Python dependencies arguments to buildPythonPackage.
It's important to have these distinctions so we can solve our long standing issues re dependency propagation in Python.
With this change in place it's possible to implement a solution for #170577.

It's similar to #102613 (a subset, really) but with the notable changes that:

• Uses PEP-621 standard names/idioms
  WIP: Python: wrap and patch using requiredPythonModules instead of propagatedBuildInputs #102613 introduces the argument requiredPythonModules, but with Python metadata being standardised I'd like to be more closely aligned with that.
• Only implements the API that enables changing the behaviour
  It's probably going to take a while to refactor all of nixpkgs to use the appropriate arguments.
  This change is purposely minimal in the hopes that we can move forward with it more quickly.

For now dependencies are always mapped onto propgatedBuildInputs, so there is no change in behaviour in the immediate term.
2 packages are migrated as a PoC, one is using format = "pyproject" and the other is setuptools.

Interface changes

This also unifies regular dependencies with optional dependencies:

{
  passthru.optional-dependencies.socks = [ pysocks ];
  nativeBuildInputs = [ poetry-core ];
  propagatedBuildInputs = [ requests ];
}

becomes

{
  optional-dependencies.socks = [ pysocks ];
  build-system = [ poetry-core ];
  dependencies = [ requests ];
}

which looks much more like what a standard pyproject.toml file contains.
I think the interface change alone is a pretty good improvement.

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

Priorities

Add a 👍 reaction to pull requests you find important.

[jonringer]

I wonder if it would be better to just expose a new mkDerivation helper (e.g. buildPyprojectPackage). I think within this context, it's simpler to expose the PEP-621terminology without having to compound it with the legacy behavior of buildPythonPackage.

[adisbladis]

I wonder if it would be better to just expose a new mkDerivation helper (e.g. buildPyprojectPackage). I think within this context, it's simpler to expose the PEP-621terminology without having to compound it with the legacy behavior of buildPythonPackage.

As discussed on Matrix I don't like adding another function because:

• The naming is already confusing between buildPythonPackage/buildPythonApplication

  This would be even more murky with a third name thrown in the mix.

• This PR doesn't semantically change much, it just allows you to speak about Python dependencies specifically as opposed to explaining the Nix semantics explicitly like propagatedBuildInputs.

  buildPythonPackage has no real notion of Python dependencies which is a bit silly when you think about it.

• Python builds are converging on pyproject

  PEP-517 style builds are becoming the default in the Python ecosystem. It would be unfortunate if our default Python builder wasn't aligned with the default Python package formats.

• Optional dependencies

  Optional dependencies are currently "supported" by buildPythonPackage by the ad-hoc use of passthru.
  This PR makes optional-dependencies (setuptools calls this extras) an explicit concept known to buildPythonPackage.
  These are not new concepts, PEP-621 just gave them better more clear naming.

[SuperSandro2000]

• The naming is already confusing between buildPythonPackage/buildPythonApplication

💯
I have done a lot with nixpkgs python and it is still hard for me to remember which I am supposed to use.

---

Also if we can get away with not fragmenting things and just shove this under the current system without breaking everything then I would be super happy.

[FRidh]

The main topic here is the separation. We need that, without a doubt.

Then there is the naming. This PR introduces non-standard names for Nixpkgs, but names that are the standard in Python packaging. While I've long held the opinion that we should have names that are similar to what is used elsewhere in Nixpkgs, this builder is used by other higher-level tools such as dream2nix and pdm2nix, which would preferably work with an interface more closely aligned to Python packaging, and for direct users of this function it is I think also more convenient.

I'd say let's go ahead with this.

[natsukium]

I have no particular opinion on the naming and have left some comments about the document.

Then, not just for this PR, but I would like to see a section like the changelog.
Since not everyone follows these discussions and changes, it would be easier to catch up if we had something to list all the changes.

[adisbladis]

Rebased, refactored & fixed comments.

[mweinelt]

Looks very straightforward. I like it.