InvalidVersion exception when invalid version used on Setuptools 66

[jstucke]

setuptools version

66.0.0

Python version

3.8

OS

Ubuntu 20.04

Additional environment information

only happening when not running inside a venv

Description

Trying to install certain pip packages like e.g. ssdeep results in an pkg_resources.extern.packaging.version.InvalidVersion exception. This seems to be related to this bug where certain ubuntu/debian packages install python packages (in this case python3-distro-info and python-debian) with versions that don't conform to PEP 440

Expected behavior

With setuptools <66.0.0 this does not cause an error

How to Reproduce

1. docker run -it --rm --entrypoint=bash ubuntu:focal
2. apt update && apt install python3 python3-pip python3-distro-info python-debian libfuzzy-dev -y
3. python3 -m pip install -U setuptools pip wheel
4. python3 -m pip install ssdeep

Output

python3 -m pip install --user ssdeep --no-cache-dir
Collecting ssdeep
  Downloading ssdeep-3.4.tar.gz (110 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 110.8/110.8 kB 3.6 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [28 lines of output]
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-install-3dwjl5j0/ssdeep_f2bd2f0d9f384d9a92bf4f0f6f3090d3/setup.py", line 108, in <module>
          setup(
        File "/home/vagrant/.local/lib/python3.8/site-packages/setuptools/__init__.py", line 86, in setup
          _install_setup_requires(attrs)
        File "/home/vagrant/.local/lib/python3.8/site-packages/setuptools/__init__.py", line 80, in _install_setup_requires
          dist.fetch_build_eggs(dist.setup_requires)
        File "/home/vagrant/.local/lib/python3.8/site-packages/setuptools/dist.py", line 874, in fetch_build_eggs
          resolved_dists = pkg_resources.working_set.resolve(
        File "/home/vagrant/.local/lib/python3.8/site-packages/pkg_resources/__init__.py", line 815, in resolve
          dist = self._resolve_dist(
        File "/home/vagrant/.local/lib/python3.8/site-packages/pkg_resources/__init__.py", line 844, in _resolve_dist
          env = Environment(self.entries)
        File "/home/vagrant/.local/lib/python3.8/site-packages/pkg_resources/__init__.py", line 1044, in __init__
          self.scan(search_path)
        File "/home/vagrant/.local/lib/python3.8/site-packages/pkg_resources/__init__.py", line 1077, in scan
          self.add(dist)
        File "/home/vagrant/.local/lib/python3.8/site-packages/pkg_resources/__init__.py", line 1096, in add
          dists.sort(key=operator.attrgetter('hashcmp'), reverse=True)
        File "/home/vagrant/.local/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2631, in hashcmp
          self.parsed_version,
        File "/home/vagrant/.local/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2678, in parsed_version
          self._parsed_version = parse_version(self.version)
        File "/home/vagrant/.local/lib/python3.8/site-packages/pkg_resources/_vendor/packaging/version.py", line 266, in __init__
          raise InvalidVersion(f"Invalid version: '{version}'")
      pkg_resources.extern.packaging.version.InvalidVersion: Invalid version: '0.23ubuntu1'
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

[nachovizzo]

I'm also facing this error

[jaraco]

This issue is by design (#2497). Non-conformant versions have been discouraged since ~2014, deprecated in packaging since late 2020 and deprecated in Setuptools since Oct 2021.

Projects should adapt to provide conformant versions (i.e. 0.23+ubuntu1) or environments that rely on such packages should pin to Setuptools < 66. These changes are necessary to keep Setuptools healthy and aligned with the goals of the packaging ecosystem.

Unfortunately, deprecation and removal is the main way I know to inform downstream consumers of breaking changes like these.

If the disruption is too great or the workarounds aren't suitable, this project may consider backing out the changes for another limited period, but only as a mitigation measure. Please first explore mitigation measures, such as pinning setuptools in the build-system.requires in the projects or work with the pip team to allow specifying constraints on build system requirements.

I'm also facing this error

Please use the thumbs-up on the original post to register that this issue also affects your project.

[yishaibeeri]

Quick note that in some cases internal dependencies several steps down the chain from the package I'm using are bringing in the new version of setuptools, making it hard to impossible to pin setuptools as a user who's running pip install. It also means install workflows that worked yesterday are broken today, even when pinning all my requirements.txt rows.
This means wide impact without an available workaround for users.
Am I missing an available workaround?
Please consider coordinating the rollout of this change with the pip team (et al) to allow for effective pinning

[karlicoss]

I'm getting this issue when I'm using setup.py with https://github.com/pypa/setuptools_scm
I have the following configuration:

        use_scm_version={
            'version_scheme': 'python-simplified-semver',
            'local_scheme': 'dirty-tag',
        },
        setup_requires=['setuptools_scm'],

With setuptools 66.0.0, I get

...
           self._parsed_version = parse_version(self.version)
        File "/usr/local/lib/python3.10/dist-packages/pkg_resources/_vendor/packaging/version.py", line 266, in __init__
          raise InvalidVersion(f"Invalid version: '{version}'")
      pkg_resources.extern.packaging.version.InvalidVersion: Invalid version: '1.1build1'
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

When I roll back the setuptools version to 65.7.0, it works as expected (e.g. my package version ends up as 1.0.20220516.dev56)

[jstucke]

Am I missing an available workaround?

If I am not mistaken, using a virtual environment should work (since all the packages with offending versions seem to come via apt or equivalent). If the package you are trying to install or its dependencies do not conform to PEP 440, this will of course not help.
Also uninstalling the system packages (python3-distro-info, python-debian, etc.) seems to work but they are probably needed by some other packages. python3-distro-info for example is used by update-manager-core, which is in turn used by update-manager, which is used by ubuntu-desktop, so it might not be feasible.

[kchomski]

I have the same issue with setuptools==65.5.1.
When tagging and trying to install a local project I get:

pkg_resources.extern.packaging.version.InvalidVersion: Invalid version: '1.23.0-test.1.2.3'

$ pip list | grep setuptools
setuptools                65.5.1

1.23.0-test.1.2.3 is a valid semver 2.0.0 tag.

EDIT: this happens inside venv (unlike OP)

$ which python
/c/USERS/foobar/.virtualenvs/venv-3.8/Scripts/python

EDIT 2:
Seems like an issue on my end - after checking PEP440 I learned that it's not fully compatible with Semantic Versioning:

Semantic versions containing a hyphen (pre-releases - clause 10) or a plus sign (builds - clause 11) are not compatible with this PEP and are not permitted in the public version field.

[jstucke]

Since the error message is somewhat cryptic (it would help if it also printed out the offending package in addition to the version), I dug around a bit to help match the version from the error message to the package that caused them:

• python-debian (is probably irrelevant because Python2.7 only)
  • 0.18ubuntu0.18.04.1 bionic
  • 0.1.36ubuntu1 focal
• python3-distro-info
  • 0.23ubuntu1 focal
  • 1.1build1 jammy
  • 1.1ubuntu1 kinetic
• python3-reportbug
  • 7.5.3~deb10u1 buster
  • 7.1.8ubuntu1 bionic
  • 7.6.0ubuntu1 focal
  • 11.4.1ubuntu1 jammy
  • 11.5.0ubuntu1 kinetic/lunar

[wijjj]

For those that for some reason cannot use the suggested fix (i.e. installing things with pip install --use-pep5171), version 67.5.1 implements a workaround for the specific problem described by @regebro:

Setuptools should simply not fail to install a package because an already installed package has a version number it doesn't like.

Notes

* Non-PEP 440 compliant versions are still deprecated and no-longer supported when building packages.
  Projects that don't comply with PEP 440 are _urged_ to change.
  Developers that support the idea that general projects should not necessarily be subject to PEP 440 are encouraged to create a discussion in the [Python discourse for packaging](https://discuss.python.org/c/packaging/14).
  
  * Setuptools issue tracker is not the best place to discuss this because it involves the entire packaging ecosystem and the standards that `setuptools` and `pip` are supposed to follow.

* Using the `--use-pep517` flag in pip is still the recommended solution[1](#user-content-fn-1-2e03857a76ce8c336e8d18daffbbd541).

Tips

* Package developers are encouraged to run tests/automated builds that don't hide deprecation warnings (e.g. `python -X dev -m build`, `python -X dev -m pip -v` and [`pytest` warning filters](https://docs.pytest.org/en/7.1.x/how-to/capture-warnings.html))

* Package developers can make setuptools less verbose by setting [build's `--config-setting`](https://pypa-build.readthedocs.io/en/latest/#python--m-build---config-setting), e.g. `python -m build -C--quiet`.

Based on the reproducer reported in #3772 (comment) by @kiorky, we can see 67.5.1 working for the complex scenario described above:

> docker run --rm -e DEBIAN_FRONTEND=noninteractive -it ubuntu:bionic@sha256:4a45212e9518f35983a976eead0de5eecc555a2f047134e9dd2cfc589076a00d bash

apt update && apt install -q -y virtualenv python3-distro-info python3-distutils

mkdir -p /tmp/mypkg
cd /tmp/mypkg
echo "aaa=1" > mymod.py
cat <<EOF > setup.py
from setuptools import setup
setup(
    name="mypkg",
    version='1.0',
    py_modules=['mymod']
)
EOF

## An older version will cause the error:
virtualenv --system-site-packages /tmp/.venv1
/tmp/.venv1/bin/python -m pip install -U 'pip==23.0.1' 'setuptools==67.4.0'
/tmp/.venv1/bin/python -m pip install -e .
# ...
# File "/tmp/.venv1/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2694, in parsed_version
#   raise packaging.version.InvalidVersion(f"{str(ex)} {info}") from None
# pkg_resources.extern.packaging.version.InvalidVersion: Invalid version: '0.23ubuntu1' (package: distro-info)

## 67.5.1 will not cause the error:
virtualenv --system-site-packages /tmp/.venv2
/tmp/.venv2/bin/python -m pip install -U 'pip==23.0.1' 'setuptools==67.5.1'
/tmp/.venv2/bin/python -m pip -v install -e .
# Successfully installed mypkg-1.0

Footnotes

1. Note that this is going to be the default in [`pip` 23.1](https://github.com/pypa/pip/blob/main/src/pip/_internal/utils/deprecation.py#L164-L175). See [pypa/pip#8559](https://github.com/pypa/pip/issues/8559) for details. [leftwards_arrow_with_hook](#user-content-fnref-1-2e03857a76ce8c336e8d18daffbbd541) [leftwards_arrow_with_hook2](#user-content-fnref-1-2-2e03857a76ce8c336e8d18daffbbd541)

thx that worked for me!

[julian-klode]

One issue I see is that this is only a warning in dist.py, so it builds fine with wrong version numbers and then errors out at runtime, like this should be erroring out hard in

https://github.com/pypa/setuptools/blob/main/setuptools/dist.py#L568

but then only issue a warning when you use pkg_resources to find plugins or something.

Such that if you have broken packages installed, unrelated software does not break, but you cannot build broken versions anymore.

[bdrung]

The initial bug (faulty versions of python3-distro-info and python-debian) is finally fixed in Ubuntu >= 20.04 (focal).

[akors]

So does this mean that I am now forever stuck at setuptools<66 ?

My project uses several dependencies that would still work (if it weren't for this change), but they are abandoned. Replacing them is not really an option, and forking is highly undesirable.

There are many old packages out there that are still useful, but do not conform to PEP 440. Is using a newer setuptools mutually exclusive with being able to use the heritage of the python ecosystem?

[jaraco]

So does this mean that I am now forever stuck at setuptools<66 ?

Yes, I think so. Either that or patching the offending packages' versions prior to install.

I wouldn't be opposed to someone supplying a plugin for setuptools that restores the prior lenient behavior on newer Setuptools versions, so that users had to explicitly opt into the known unsupported state and Setuptools doesn't have to own the legacy support. I'm uncertain if Setuptools currently provides a supported hook where such a plugin could attach or if that approach would even be helpful to the affected environments.

[Greg7000]

When I am in a context where I face this error and still want a custom tag. I use the '+' sign (which you can see from jaraco comment.**

For example, lets say I want mycode-1.0<CUSTOM_TAG>

Assuming I want my custom tag to be win_bob

I set my version to mycode-1.0+win_bob

Which is then interpreted as mycode-1.0+win.bob. For my needs, this is acceptable.

Check https://peps.python.org/pep-0440/#local-version-segments for more information.

[RyanPaulMcKenna]

Switching to setuptools 65 solved it for me, now I have new error.
pip install setuptools==65