Trying to fix the setuptools fallout

[K900]

TODO list (ongoing):

• fix package name normalization
• fix override updating script to not screw with existing overrides with version constraints

[cpcloud]

@K900 Thanks for looking into this. I pushed a commit to the PR to format the code to get more CI to run.

[cpcloud]

It looks like the pluggy dependency for hatchling isn't found because the hatchling version requires pluggy 1.0.0 but 0.13.1 is found.

[cpcloud]

I'm trying this against a project I work on with complex dependencies (https://github.com/ibis-project/ibis) and this PR appears to still infinitely recurse, but in a way that isn't detectable until after evaluation has started (and eventually fails with a stack overflow), as opposed to poetry2nix/master in which the recursion is detected during evaluation and a somewhat useful error message is reported.

[K900]

There's definitely new circular dependencies in there somewhere, we'll have to minimize it to find out.

[cpcloud]

Using poetry<1.2, setuptools is never included in a package's dependencies in poetry.lock, whereas with poetry==1.2 now packages that have a (likely incorrect) runtime dependency on setuptools have that dependency listed in the lock file.

Removing those dependencies' dependency on setuptools in the lockfile and removing setuptools itself from the lockfile allows evaluation to proceed.

[cpcloud]

I have upstream'd a few PRs to help address the incorrect setuptools dependency in the ecosystem:

one to Toblerity/Fiona likely to be merged: Toblerity/Fiona#1139
one to ipdb unlikely to be merged: gotcha/ipdb#254
one to numba, not merged but addressed in another open PR: numba/numba#8366

[K900]

I'm still not sure why we're looping on setuptools at all tbh, especially now that there's no implicit dependency on it.

[cpcloud]

I've pushed a commit to bring back the from/until version bounds on a couple of the build system overrides, since that was causing failures due to bringing in incompatible versions of transitive dependencies.

[cpcloud]

I haven't yet been able isolate a smaller reproducible example than what's in ibis-project/ibis, but it appears there may be a loop in python-remove-tests-dir-hook?

When I run

nix build -f . -vvv --dry-run 2>&1 | tee /tmp/output.txt

on my cpcloud/use-our-nixpkgs branch of ibis-project/ibis

I see this output repeated ad infinitum:

instantiated 'python-remove-tests-dir-hook' -> '/nix/store/i1x50xj78xyv9zra1qpjb5dp42jqd7aj-python-remove-tests-dir-hook.drv'
instantiated 'python-setup-hook.sh' -> '/nix/store/bcjff6dn4gp7pf0knnja9ax4qrll78d5-python-setup-hook.sh.drv'
instantiated 'Python-3.10.7.tar.xz' -> '/nix/store/ap22y5ilipad2bhs555rldnir8hki1j2-Python-3.10.7.tar.xz.drv'
instantiated 'python3-3.10.7' -> '/nix/store/c5rlaqdfymd3j5v41rh98pjcv3dx3sh7-python3-3.10.7.drv'
instantiated 'hook' -> '/nix/store/z7br4z8im93lckrpxgwbrscfhwyh0ic8-hook.drv'
instantiated 'python-remove-tests-dir-hook' -> '/nix/store/i1x50xj78xyv9zra1qpjb5dp42jqd7aj-python-remove-tests-dir-hook.drv'
instantiated 'python-setup-hook.sh' -> '/nix/store/bcjff6dn4gp7pf0knnja9ax4qrll78d5-python-setup-hook.sh.drv'
instantiated 'Python-3.10.7.tar.xz' -> '/nix/store/ap22y5ilipad2bhs555rldnir8hki1j2-Python-3.10.7.tar.xz.drv'
instantiated 'python3-3.10.7' -> '/nix/store/c5rlaqdfymd3j5v41rh98pjcv3dx3sh7-python3-3.10.7.drv'
instantiated 'hook' -> '/nix/store/z7br4z8im93lckrpxgwbrscfhwyh0ic8-hook.drv'

Add this bit to the poetry-overrides doesn't seem to help:

  poetry-core = super.poetry-core.overridePythonAttrs (_: {
    dontUsePythonRemoveTestsDir = true;
  });

[cpcloud]

@K900 Hope you don't mind, but I pushed what I think is a minimal test case project: it only depends on setuptools. This is sufficient to cause the infinite recursion at eval time.

[K900]

Please continue, I probably won't have much more time to look into this in the next few days.

[K900]

This should probably be extended to cover all the build systems we support...

[cpcloud]

Yeah, that makes sense. I'd like to see if I can get to the bottom of the reason why the infinite recursion is happening before doing that though.

One downside to this approach is that the version of whatever the build system is always pinned to whatever version/commit nixpkgs is pinned to. For setuptools that probably doesn't matter much, but for the newer build systems I'm not so sure.

[cpcloud]

I was able to get the test case I added to pass by always using super.setuptools if it's included as a dependency.

I'll admit I'm not entirely sure why this works.

My hunch is that there are two different versions of setuptools in the closure: one coming from super, and one coming from the poetry lock file and they are colliding somehow.

@adisbladis Thoughts on this?

[K900]

I don't think that's avoidable, and it might be better than getting whatever random thing we end up with in the lockfile.

[cpcloud]

Hm, that's a good point.

Separately, I tried this branch against ibis-project/ibis and it works.

[roberth]

I've blocked a nixops_unstable update because of this issue, but it's not clear to me what's happened around setuptools. Is it because of a poetry version update? (A PR description or issue link would have clarified this)

How do you think this situation should/will affect the poetry2nix-based packages in the upcoming NixOS release?

[K900]

Nixpkgs' Python hooks used to propagate setuptools, but they don't anymore, so all the things that implicitly expected it to be there exploded.

[cpcloud]

There's a second issue here too: poetry < 1.2 automatically removed setuptools from dependencies, whereas 1.2 doesn't resulting in infinite recursion.

[talyz]

@roberth NixOS/nixpkgs#185815 is the breaking change.

[PedroRegisPOAR]

Maybe I have found an reproducer

1. From poetry2nix master:

nix run --no-write-lock-file github:nix-community/poetry2nix#poetry -- --version

The locked URL:

# To get the locked URL run:
# nix flake metadata --no-write-lock-file github:nix-community/poetry2nix
nix run --no-write-lock-file github:nix-community/poetry2nix/c2ee667d2fd19f70a9ac256b81f2cfc248c4f04d#poetry -- --version

Results in:

warning: not writing modified lock file of flake 'github:nix-community/poetry2nix':
• Added input 'flake-utils':
    'github:numtide/flake-utils/c0e246b9b83f637f4681389ecabcb2681b4f3af0' (2022-08-07)
• Added input 'nixpkgs':
    'github:NixOS/nixpkgs/2380304ae4975ef0d6ba13b64f990e94e8c5fdbb' (2022-10-04)
error: builder for '/nix/store/avk4c6r2y4mixriryam1rgrrbkzfivmi-python3.10-packaging-21.3.drv' failed with exit code 2;
       last 10 log lines:
       >   File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
       >   File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
       >   File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
       >   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
       >   File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
       >   File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
       >   File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked
       > ModuleNotFoundError: No module named 'setuptools'
       > 
       > 
       For full logs, run 'nix log /nix/store/avk4c6r2y4mixriryam1rgrrbkzfivmi-python3.10-packaging-21.3.drv'.
error: 1 dependencies of derivation '/nix/store/wb17wpkm1p8w00cajp960w8yinwh75xy-python3.10-poetry-1.2.0.drv' failed to build

2. From K900/poetry2nix/hack

nix run --no-write-lock-file github:K900/poetry2nix/hack#poetry -- --version

Results in:

warning: not writing modified lock file of flake 'github:K900/poetry2nix/hack':
• Added input 'flake-utils':
    'github:numtide/flake-utils/c0e246b9b83f637f4681389ecabcb2681b4f3af0' (2022-08-07)
• Added input 'nixpkgs':
    'github:NixOS/nixpkgs/2380304ae4975ef0d6ba13b64f990e94e8c5fdbb' (2022-10-04)
Poetry (version 1.2.0)

So, it works!

The locked URL:

# To get the locked URL run:
# nix flake metadata --no-write-lock-file github:K900/poetry2nix/hack
nix run --no-write-lock-file github:K900/poetry2nix/fc3b734766d49695b34259d68e823f3baa732c07#poetry -- --version

Extra metadata

nix run nixpkgs#nix-info -- --markdown

• system: "x86_64-linux"
• host os: Linux 5.15.0-48-generic, Ubuntu, 22.04.1 LTS (Jammy Jellyfish), nobuild
• multi-user?: yes
• sandbox: yes
• version: nix-env (Nix) 2.10.2
• channels(vagrant): "nixpkgs"
• nixpkgs: /home/vagrant/.nix-defexpr/channels/nixpkgs

About the machine that it was tested (an Ubuntu Vagrant virtual machine):

nix run nixpkgs#neofetch -- --json

Outputs:

{
    "OS": "Ubuntu 22.04.1 LTS x86_64",
    "Host": "KVM/QEMU (Standard PC (i440FX + PIIX, 1996) pc-i440fx-7.0)",
    "Kernel": "5.15.0-48-generic",
    "Uptime": "7 mins",
    "Packages": "626 (dpkg), 44 (nix-user), 3 (snap)",
    "Shell": "bash 5.1.16",
    "Resolution": "1024x768",
    "Terminal": "/dev/pts/0",
    "CPU": "Intel (Haswell, no TSX, IBRS) (4) @ 3.392GHz",
    "GPU": "Cirrus Logic QEMU Virtual Machine",
    "Memory": "331MiB / 968MiB",
    "Version": "7.1.0"
}

[K900]

Yes, that reproduces the missing setuptools issue, but not the infinite recursion issue.

[cpcloud]

I guess the question is whether the solution to fix the infinite recursion is the correct one. I'm happy to add the other build systems to the check if people think the approach is the way forward.

[vale981]

fwiw, it still seems like there is an issue here when attempting to run packages, instead of just building an env for them.
i'm in a situation where the nix shell works fine, but nix run . ... does not
( -- edit: i think i'm wrong; i solved my problem by adding setuptools = "*" as a dep. )

A lot of packages don't add setuptools to their runtime dependencies even though they depend on it (typically pkg_resources).

Is there a workaround for injecting this dependency other than editing poetry.lock?

[K900]

Yes, via the overrides mechanism.

[vale981]

so i feared :P