`import pkg_resources` triggers a DeprecationWarning on Python 3.9 #368

alexhenrie · 2020-12-03T23:43:38Z

I'm pretty sure this is a bug because I don't think pkg_resources is deprecated in its entirety.

$ PYTHONWARNINGS=error python3 -c 'import pkg_resources'
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/packaging/version.py", line 57, in parse
    return Version(version)
  File "/usr/lib/python3.9/site-packages/packaging/version.py", line 298, in __init__
    raise InvalidVersion("Invalid version: '{0}'".format(version))
packaging.version.InvalidVersion: Invalid version: '/usr/lib/python3.9/site'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python3.9/site-packages/pkg_resources/__init__.py", line 3239, in <module>
    def _initialize_master_working_set():
  File "/usr/lib/python3.9/site-packages/pkg_resources/__init__.py", line 3222, in _call_aside
    f(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/pkg_resources/__init__.py", line 3251, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/lib/python3.9/site-packages/pkg_resources/__init__.py", line 558, in _build_master
    ws = cls()
  File "/usr/lib/python3.9/site-packages/pkg_resources/__init__.py", line 551, in __init__
    self.add_entry(entry)
  File "/usr/lib/python3.9/site-packages/pkg_resources/__init__.py", line 607, in add_entry
    for dist in find_distributions(entry, True):
  File "/usr/lib/python3.9/site-packages/pkg_resources/__init__.py", line 2056, in find_on_path
    path_item_entries = _by_version_descending(filtered)
  File "/usr/lib/python3.9/site-packages/pkg_resources/__init__.py", line 2026, in _by_version_descending
    return sorted(names, key=_by_version, reverse=True)
  File "/usr/lib/python3.9/site-packages/pkg_resources/__init__.py", line 2024, in _by_version
    return [packaging.version.parse(part) for part in parts]
  File "/usr/lib/python3.9/site-packages/pkg_resources/__init__.py", line 2024, in <listcomp>
    return [packaging.version.parse(part) for part in parts]
  File "/usr/lib/python3.9/site-packages/packaging/version.py", line 59, in parse
    return LegacyVersion(version)
  File "/usr/lib/python3.9/site-packages/packaging/version.py", line 127, in __init__
    warnings.warn(
DeprecationWarning: Creating a LegacyVersion has been deprecated and will be removed in the next major release

The text was updated successfully, but these errors were encountered:

alexhenrie · 2020-12-04T00:36:19Z

Cross-posted to pypa/setuptools#2466 because I'm not sure whether the bug is in packaging or in setuptools.

brettcannon · 2020-12-04T01:48:52Z

Without knowing what version is being attempted to be constructed it's hard to know if it's a bug. But it is possible that pkg_resources is specifying a version format that only works with LegacyVersion.

uranusjr · 2020-12-04T01:58:28Z

One problem with pkg_resources is that it traverses through all the installed packages in your environment eagerly, on import time, so this would be triggered if you have any such packages installed, even if the package is not used at all. It would be best if you could run pip list or something similar to find out what the offending package is, and try to (or tell maintainers of the package to) fix it.

As for a fix… setuptools would need to provide additional handling to avoid constructing LegacyVersion instances, since it is going away at some point in the future. Or, since you’re on Python 3.9, it would be best to switch to importlib.metadata instead, which is better maintained.

jaraco · 2020-12-20T16:39:26Z

The only reason importlib.metadata doesn't fall prey to this issue is because it doesn't attempt to provide packages sorted by version.

By dropping support for LegacyVersion, this project is suggesting that Setuptools too should drop support for such versions. One way this could happen is Setuptools could drop support for sorting versions. Another could be to drop support for packages that have these versions. Does the packaging project have a design in mind for what should happen with projects and environments that contain packages like these? I should think that before deprecating the structures for managing these versions, the project should first move to discourage or disallow the presence of such packages. Or perhaps instead, the PyPA would like to allow these types of non-standard versions, but the tooling will simply stop supporting them.

di · 2020-12-20T23:33:00Z

The only reason importlib.metadata doesn't fall prey to this issue is because it doesn't attempt to provide packages sorted by version.

I'm not a setuptools maintainer, could you explain why setuptools is trying to do this? What's the use case?

One way this could happen is Setuptools could drop support for sorting versions. Another could be to drop support for packages that have these versions.

Legacy versions are just sorted lexicographically -- since they don't have a structure there's no other way to sort them. I think there's probably a middleground here where setuptools can continue to sort all versions (if it needs to) and just not try to turn invalid versions into legacy versions.

I should think that before deprecating the structures for managing these versions, the project should first move to discourage or disallow the presence of such packages.

Legacy versions are already prohibited on PyPI and have been for quite some time.

Anyone using them is either a) living so far in the past that they can't expect modern tooling to be compatible or b) doing some weird stuff that's probably well outside the use cases we're able to support.

jaraco · 2020-12-21T03:43:49Z

The only reason importlib.metadata doesn't fall prey to this issue is because it doesn't attempt to provide packages sorted by version.

I'm not a setuptools maintainer, could you explain why setuptools is trying to do this? What's the use case?

I'm not sure. I removed that sorting behavior and the tests still pass, so any use case isn't captured by the tests. It's possible this behavior was present to give precedence to the latest versions of a package when multiple versions of the same package were installed.

Tracing the commits, I found pypa/setuptools@fb1867e and pypa/setuptools@e0805ec introduced the sorting behavior. Those commits led me to pypa/setuptools#629. My suspicions about the motivations were correct. I also notice that I allowed the change without tests and at the same time warned that these changes might be optimized away, so maybe that's the best approach at this point.

One way this could happen is Setuptools could drop support for sorting versions. Another could be to drop support for packages that have these versions.

Legacy versions are just sorted lexicographically -- since they don't have a structure there's no other way to sort them. I think there's probably a middleground here where setuptools can continue to sort all versions (if it needs to) and just not try to turn invalid versions into legacy versions.

This isn't quite correct. LegacyVersions are sorted using the rules that Setuptools used to use to sort versions. It does honor decimal separators.

>>> import packaging.version
>>> lv = packaging.version.LegacyVersion('3.10_4')
>>> lv._key
(-1, ('00000003', '00000010', '*_', '00000004', '*final'))
>>> lv > packaging.version.LegacyVersion('3.2')
True

I should think that before deprecating the structures for managing these versions, the project should first move to discourage or disallow the presence of such packages.

Legacy versions are already prohibited on PyPI and have been for quite some time.

Anyone using them is either a) living so far in the past that they can't expect modern tooling to be compatible or b) doing some weird stuff that's probably well outside the use cases we're able to support.

Sounds good. I've filed pypa/setuptools#2497 to enact that change.

uranusjr · 2020-12-21T05:18:06Z

Anyone using them is either a) living so far in the past that they can't expect modern tooling to be compatible or b) doing some weird stuff that's probably well outside the use cases we're able to support.

What counts as “weird stuff” is fuzzy, however. Both importlib.metadata and pkg_resources inspect things in site-packages directories, and those locations are used by more than Python packaging tools. This DeprecationWarning pops up when those tools use a different version scheme from PEP 440’s, which is not a “far in the past” thing because they never had a need to change.

Those other tools should ultimately modernise, of course, but the particular way packaging is doing this makes the migration path less obvious. Maybe packaging can raise an alternative warning subclass for setuptools to catch, so it has a chance to detect the deprecation in a structured way and emit a more user-friendly warning.

Also, would it be a good idea to expose a base class/protocol for downstream tools to inherit from, so tools can easier re-implement (a part of) LegacyVersion when they really need to deal with non-PEP-440 version formats?

di · 2020-12-21T21:42:14Z

This isn't quite correct. LegacyVersions are sorted using the rules that Setuptools used to use to sort versions. It does honor decimal separators.

Sorry, my mistake, that's what I get trying to respond from memory alone 🙂

bnavigator · 2021-02-03T22:27:50Z

Without knowing what version is being attempted to be constructed it's hard to know if it's a bug. But it is possible that pkg_resources is specifying a version format that only works with LegacyVersion.

pkg_resources is passing all parts of the package name, possibly including some directory paths to packaging.version.parse() and just treats everything that is not a Version as LegacyVersion: pypa/setuptools#2466 (comment), pypa/setuptools#2466 (comment)

Caused by pypa/packaging#368

#1045) * Temporary workaround to pip install -e issues * Filter out version warnings Caused by pypa/packaging#368 * Temporarily limit xarray to <0.17

* grid.__getitem__(key) now returns u.Quantity instead of xarray.Dataset The goal of the grids module is for users to not have to be familiar with xarray to use it. Returning a u.Quantity is more useful. * Added require_quantities() method to grids * Temporary workaround to pip install -e issues * Filter out version warnings Caused by pypa/packaging#368 * Temporarily limit xarray to <0.17 * Revert "Temporarily limit xarray to <0.17" This reverts commit 4f64afe. * xarraydev tests * Changes to proton radiography module because __getitem__ now returns u.Quantity * pre-commit * Swatted one bug (add_quantities removing ax0,ax1,ax2 from dataset for non-uniform grids) * Fixed issue setting quantities? * Create 1027.breaking.rst * Update 1027.breaking.rst * Update 1027.breaking.rst * Refactored one check in `proton_radiography.py` to use new `grid.require_quantities` method Co-authored-by: Dominik Stańczak <stanczakdominik@gmail.com>

x-yuri · 2021-10-14T06:33:43Z

One problem with pkg_resources is that it traverses through all the installed packages in your environment eagerly, on import time, so this would be triggered if you have any such packages installed, even if the package is not used at all. It would be best if you could run pip list or something similar to find out what the offending package is, and try to (or tell maintainers of the package to) fix it.

@uranusjr Either I'm going to say something silly, or you're missing the point. It fails for any package including pip. Take a look at _by_version():

https://github.com/pypa/setuptools/blob/v58.2.0/pkg_resources/__init__.py#L2027

When it receives pip-21.3.dist-info, it calls packaging.version.parse() with pip, 21.3, and .dist-info. Only the second one returns a Version instance. Sounds like a bug in setuptools.

To reproduce:

$ docker run --rm -it -e PYTHONWARNINGS=always python:3.10.0-alpine3.14 pip --version
/usr/local/lib/python3.10/site-packages/pip/_vendor/packaging/version.py:111: DeprecationWarning: Creating a LegacyVersion has been deprecated and will be removed in the next major release
  warnings.warn(
/usr/local/lib/python3.10/site-packages/pip/_vendor/packaging/version.py:111: DeprecationWarning: Creating a LegacyVersion has been deprecated and will be removed in the next major release
  warnings.warn(
/usr/local/lib/python3.10/site-packages/pip/_vendor/packaging/version.py:111: DeprecationWarning: Creating a LegacyVersion has been deprecated and will be removed in the next major release
  warnings.warn(
/usr/local/lib/python3.10/site-packages/pip/_vendor/packaging/version.py:111: DeprecationWarning: Creating a LegacyVersion has been deprecated and will be removed in the next major release
  warnings.warn(
/usr/local/lib/python3.10/site-packages/pip/_vendor/packaging/version.py:111: DeprecationWarning: Creating a LegacyVersion has been deprecated and will be removed in the next major release
  warnings.warn(
/usr/local/lib/python3.10/site-packages/pip/_vendor/packaging/version.py:111: DeprecationWarning: Creating a LegacyVersion has been deprecated and will be removed in the next major release
  warnings.warn(
/usr/local/lib/python3.10/site-packages/pip/_vendor/packaging/version.py:111: DeprecationWarning: Creating a LegacyVersion has been deprecated and will be removed in the next major release
  warnings.warn(
/usr/local/lib/python3.10/site-packages/pip/_vendor/packaging/version.py:111: DeprecationWarning: Creating a LegacyVersion has been deprecated and will be removed in the next major release
  warnings.warn(
/usr/local/lib/python3.10/site-packages/pip/_vendor/packaging/version.py:111: DeprecationWarning: Creating a LegacyVersion has been deprecated and will be removed in the next major release
  warnings.warn(
/usr/local/lib/python3.10/site-packages/pip/_vendor/packaging/version.py:111: DeprecationWarning: Creating a LegacyVersion has been deprecated and will be removed in the next major release
  warnings.warn(
/usr/local/lib/python3.10/site-packages/pip/_vendor/packaging/version.py:111: DeprecationWarning: Creating a LegacyVersion has been deprecated and will be removed in the next major release
  warnings.warn(
/usr/local/lib/python3.10/site-packages/pip/_vendor/packaging/version.py:111: DeprecationWarning: Creating a LegacyVersion has been deprecated and will be removed in the next major release
  warnings.warn(
/usr/local/lib/python3.10/site-packages/pip/_internal/locations/_distutils.py:9: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
  from distutils.cmd import Command as DistutilsCommand
/usr/local/lib/python3.10/distutils/command/install.py:13: DeprecationWarning: The distutils.sysconfig module is deprecated, use sysconfig instead
  from distutils.sysconfig import get_config_vars
pip 21.2.4 from /usr/local/lib/python3.10/site-packages/pip (python 3.10)

And it's so since 2016-11-04. LegacyVersion was deprecated on 2020-10-21.

uranusjr · 2021-10-16T03:07:54Z

??? I am not even sure why you want to tag me, or why you are even commenting here. This is an entire different problem, and you already concluded setuptools should fix it, and you are commenting in a repository called packaging?

x-yuri · 2021-10-16T03:38:47Z

Are you sure the problem is entirely different? It was cross-posted to setuptools. And while investigating I saw people here and there saying that LegacyVersion should be dropped. Which might be the case, but it doesn't seem like the reason behind the DeprecationWarning. In this issue noone mentioned that the problem is with setuptools. The reason I mentioned you is because you clearly stated that this is caused by "obsolete" (?) packages. But it can be reproduced in a pristine environment. Maybe to an extent because we kind of met before :) At least it looks like the issue here can be closed. Unless you want it to be about dropping LegacyVersion. And when I created my post, I was less sure about the cause. Since then I made more investigation and an edit. Sorry to bother :(

pradyunsg · 2021-10-16T07:10:14Z

Closing since pypa/setuptools#2466 covers this, and we've had one awkward social interaction already. :)

alexhenrie mentioned this issue Dec 4, 2020

import pkg_resources triggers a DeprecationWarning on Python 3.9 pypa/setuptools#2466

Closed

brettcannon added bug packaging.version labels Dec 4, 2020

yan12125 mentioned this issue Dec 10, 2020

Avoid using LegacyVersion from packaging pypa/setuptools#2478

Closed

2 tasks

domdfcoding mentioned this issue Dec 10, 2020

pip CLI emits DeprecationWarning due to use of LegacyVersion pypa/pip#9250

Closed

jaraco mentioned this issue Dec 21, 2020

Remove support for LegacyVersion pypa/setuptools#2497

Closed

bnavigator mentioned this issue Feb 3, 2021

test_import_without_warning fails with packaging >= 20.5 dask/fastparquet#558

Open

StanczakDominik added a commit to StanczakDominik/PlasmaPy that referenced this issue Feb 26, 2021

Filter out version warnings

89ad360

Caused by pypa/packaging#368

StanczakDominik mentioned this issue Feb 26, 2021

Temporary workarounds to pip install -e, packaging, xarray 0.17 issues PlasmaPy/PlasmaPy#1045

Merged

1 task

jpmckinney mentioned this issue May 13, 2021

Using csvkit with Python3.9 produces DeprecationWarning's wireservice/csvkit#1112

Closed

frenck mentioned this issue Aug 22, 2021

InvalidVersion on importing ytmusicapi AppDaemon/appdaemon#1336

Closed

This was referenced Aug 22, 2021

This is a Python issue, there are multiple issues created around different Python projects about this. AppDaemon/appdaemon#1339

Closed

Replace usage of pkg_resources with importlib_resources sigma67/ytmusicapi#223

Closed

pradyunsg closed this as completed Oct 16, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`import pkg_resources` triggers a DeprecationWarning on Python 3.9 #368

`import pkg_resources` triggers a DeprecationWarning on Python 3.9 #368

alexhenrie commented Dec 3, 2020

alexhenrie commented Dec 4, 2020

brettcannon commented Dec 4, 2020

uranusjr commented Dec 4, 2020

jaraco commented Dec 20, 2020

di commented Dec 20, 2020

jaraco commented Dec 21, 2020

uranusjr commented Dec 21, 2020

di commented Dec 21, 2020

bnavigator commented Feb 3, 2021 •

edited

x-yuri commented Oct 14, 2021 •

edited

uranusjr commented Oct 16, 2021

x-yuri commented Oct 16, 2021

pradyunsg commented Oct 16, 2021

import pkg_resources triggers a DeprecationWarning on Python 3.9 #368

import pkg_resources triggers a DeprecationWarning on Python 3.9 #368

Comments

alexhenrie commented Dec 3, 2020

alexhenrie commented Dec 4, 2020

brettcannon commented Dec 4, 2020

uranusjr commented Dec 4, 2020

jaraco commented Dec 20, 2020

di commented Dec 20, 2020

jaraco commented Dec 21, 2020

uranusjr commented Dec 21, 2020

di commented Dec 21, 2020

bnavigator commented Feb 3, 2021 • edited

x-yuri commented Oct 14, 2021 • edited

uranusjr commented Oct 16, 2021

x-yuri commented Oct 16, 2021

pradyunsg commented Oct 16, 2021

`import pkg_resources` triggers a DeprecationWarning on Python 3.9 #368

`import pkg_resources` triggers a DeprecationWarning on Python 3.9 #368

bnavigator commented Feb 3, 2021 •

edited

x-yuri commented Oct 14, 2021 •

edited