Breaking change in 7e5bae4 (importlib_metadata 5) #409
Comments
|
seconded. i don't have specific details like @inno but i am able to trace this back to upgrading to 5.0.0. the package that triggered the error was for the time being i have downgraded to <5.0.0 |
|
Breaks flake8 too, which uses |
|
…though it looks like that was intentional looking at 7e5bae4, which removed a previously-deprecated interface and corresponding warnings silencer specifically for flake8. Ah. Right. This is the whole SelectableGroups debacle I'd managed to forget. :-( |
|
That's correct. Support for dict-based access was deprecated in Is it possible the usage in stevedore didn't trigger the deprecation warnings? |
The corresponding warnings silencer for flake8 was removed in #319. The only reason it shows up in that specific commit is because I made these removal commits against v4.4.0 to ensure that I wasn't removing anything that wasn't part of Python 3.10. The actual diff for this change can be seen in #405. It's my understanding that later versions of flake8 provide a workaround for this issue but also that flake8 pins against and older version of importlib_metadata, so shouldn't get this behavior anyway. Any idea why an older version of flake8 is ending up with a brand new version of importlib_metadata? |
Other things in the dep chain need a newer importlib_metadata, so pip walks back the flake8 versions until it finds flake8 3.9.2 which is the last release that includes If I manually pin to flake8 >=4.0.0, the first release which includes the importlib_metadata pin, then pip instead walks back the versions of other things to a point where the version of importlib_metadata they need is compatible with the older version flake8 needs. (I'm not sure yet if this is acceptable or not for our usage, but at least it's something.) |
Oh, interesting. That makes sense now that you explain it.
I'd expect that to be the case, as I do believe that flake8 has removed the usage of the deprecated code paths in late releases, so they probably can remove the pin and help ease the pain.
That also seems like a reasonable workaround. |
|
By the way, if it would help to yank the release and delay the break for a day or a week or a month to stop the bleeding, that’s certainly possible. |
Workaround for poetry + flake8 + python 3.7+: This pin fixed the resolution for meflake8 = [
{ version = "*", python = "^3.7" },
{ version = ">=5", python = "^3.8" },
]I truly don't know why the resolution has these issues. One possibility I haven't ruled out: https://github.com/PyCQA/flake8/blob/e94ee2b5f1801354b940cfe830b9160852915aec/setup.cfg#L45 |
With the release of importlib_metadata 5.0 deprecation of an endpoint has caused stevedore to fail (at least in python 3.7). For now, I'm pinning importlib-metadata to < 5.0. See: * python/importlib_metadata#409 * https://stackoverflow.com/questions/73929564/entrypoints-object-has-no-attribute-get-digital-ocean
|
Unfortunately importlib_metadata is used deep within many automated systems (build systems, containers, etc...), so the deprecation warnings are likely unseen by most and ignored by many others. It's super easy to see "this functionality is deprecated" and treat it as "I won't use it more in the future" instead of "I will stop using it now". On that note, it might be helpful to have messaging regarding a date or version when breaking changes are coming. That way folks can plan for it, especially for a commonly used package. |
|
I really appreciate the feedback.
That's useful feedback. I've been operating under a less rigorous versioning strategy and I've only seen a few examples of projects that have adopted a rigorous strategy for breaking changes (Python recently with the two-version deprecation period, and pip with the breaking changes annually). I've found that different breaking changes have different blast radii, so it's difficult to devise a specific strategy that accommodates the variety of impacts. I try to ascertain the level of impact and use that to devise a timeline for deprecation/removal. Often, I will put a specific date in the code "not to be removed before YYYY-MM-DD" to provide downstream consumers a minimum time. I have found that attempting to say specifically when or in what version a breaking change will occur is folly and often gets missed. I didn't put a particular date on this change in part because the deprecation itself was so impactful, I'd expected the major consumers to have already addressed the issue early and the 16 month delay gave sufficient time for systems to upgrade. I do think it would have helped to put a minimum date on the deprecation warning. I'll do that for the future and possibly remaining deprecation warnings.
That's an interesting perspective. I've never taken that perspective. Whenever I've encountered a If some maintainers hold the opinion that I could imagine some projects employing a Per the Python deprecation policy, this behavior is expected to change in Python 3.12 as well, which is why I wanted to get the backport change out sooner, so users would have a chance to adapt where they have more control over the behavior (pinning dependencies, adding backports, etc).
Good point. If there's a specific example where the directly-affected project (the project using the interface) did not have visibility to the deprecation warning, I'd like to investigate that, because that's an assumption I rely on to enact these changes. I know other projects downstream of the direct-consuming projects will be affected, and I rely on the direct consumers to take action when deprecations affect their projects and their users. My current understanding is that the big projects affected are openstack (stevedore) and flake8, both projects with which I engaged directly at the time of the deprecation (or prior to it) and proposed multiple fixes to address the breakage early. If any project had reached out and requested that the removal be delayed, I certainly would have considered that and likely built it into the approach. Stability and predictability are important, but if a project chooses to ignore the warnings, they do so at their own risk (true of any project). |
…interface. Summary: Temporarily peg importlib_metadata to <5.0 due do breaking changes (python/importlib_metadata#409) for CPython 3.7 The right fix is to use `metadata.entry_points(group=group, name=name)`, however typechecker does not recognize the changes (python/importlib_metadata@7e5bae4) Differential Revision: D40053701 fbshipit-source-id: 2cc6adab47945334104c150af005057845839b73
|
fwiw, Celery is now tracking this - celery/celery#7783 - which was why i came to the issue in this repo in the first place. i understand that the change to 5.x was semantically a breaking change. on the other hand, because my project isn't using poetry (😭) i didn't even know i was using |
…interface. (#608) Summary: Pull Request resolved: #608 Temporarily peg importlib_metadata to <5.0 due do breaking changes (python/importlib_metadata#409) for CPython 3.7 The right fix is to use `metadata.entry_points(group=group, name=name)`, however typechecker does not recognize the changes (python/importlib_metadata@7e5bae4) Reviewed By: d4l3k Differential Revision: D40053701 fbshipit-source-id: 9860594c5283ce3f24f7284dff98447aa2df0604
The biggest impact of this change is on Python 3.7 where we were seeing
CI test failures.¹ Those now pass again. The story is complicated.
Let's start with the facts.
The recent release of importlib-metadata 5.0.0 removed a previously
deprecated entry points API.²
This API is used by flake8.
flake8 4.0.0 onwards declares (more or less):
importlib-metadata <4.3; python_version < "3.8"
…while prior versions, flake8 3.9.2 and earlier, declared:
importlib-metadata; python_version < "3.8"
Note that flake8 3.9.2 and earlier still require in practice the upper
bound on the importlib-metadata version, they just don't express it in
package metadata. (This is what's known as foreshadowing…)
Meanwhile, Sphinx 4.4.0 onwards declares (more or less):
importlib-metadata >=4.4; python_version < '3.10'
Note that on Python 3.7 and earlier, the latest versions of both flake8
and Sphinx have conflicting version requirements for importlib-metadata.
So when Pip on 3.7 goes to install both, it has to resolve the issue by
downgrading either flake8 or Sphinx until it finds a dependency
solution. For whatever reason, it choose to keep the newer versions of
Sphinx and importlib-metadata and downgrade flake8. So it walks back in
flake8 versions until it eventually finds 3.9.2 without the
importlib-metadata version pin. Pip is satisfied, but although Pip
can't know it, flake8 won't work with importlib-metadata 5.0.0.
By pinning flake8 ≥4.0.0, we're hinting to Pip that only those versions
accurately declare their dependencies given the current situation. Pip
ultimately can use this better dependency information solve the
importlib-metadata conflict by walking back to a version of Sphinx
≤4.4.0 (which it turns out, don't use importlib-metadata at all).
The flake8 breakage doesn't arise on Python 3.6 because support for that
Python version was dropped by importlib-metadata ≥4.9.0, so the removed
API in 5.0.0 isn't seen. Without our flake8 pin, Pip still resolves the
conflict by walking flake8 back to 3.9.2 and keeping the latest Sphinx
plus importlib-metadata 4.8.3. While that importlib-metadata version is
declared as unsupported in later version of flake8, it seems to work
fine in practice. ¯\_(ツ)_/¯
On Python ≥3.8, the importlib-metadata conflict doesn't exist,
presumably because flake8 uses the importlib.metadata standard library
instead.
We'll still build our production docs on RTD with the latest version of
Sphinx, because we use Python 3.10 there, and the downgrading of Sphinx
is limited to 3.7.
(If some of this is making your head ring but also ringing some bells,
this isn't the first time importlib-metadata's development choices and
flake8's development choices have had fallout for us.³)
¹ python/importlib_metadata#409 (comment)
² python/importlib_metadata#405
³ "dev: Ignore a deprecation warning generated by flake8's usage of importlib_metadata" (1a04901)
|
If I have this code: what is the recommended way to adjust that with |
|
Ah, I think I figured it out! https://importlib-metadata.readthedocs.io/en/latest/using.html#entry-points The documentation was clear. =) |
|
I really appreciate that |
Closes #1550 Related to python/importlib_metadata#409 `entry_points()` does not return dict-like objects anymore. Switch to using `.select()`. `.select()` has been around since `importlib-metadata` 3.6, which is our lowest supported version.
I'm pretty sure you can simplify this to: Because |
CI jobs `profile-windows-*`, `kombu` and `celery` are failing due to the release of `importlib_metadata==5.0.0` which removed deprecated shims, resulting in the following exception: ``` AttributeError: 'EntryPoints' object has no attribute 'get' ``` We fix this by pinning the dependency for kombu and celery. In the case of the Windows tests, we address it by upgrading the Windows image to a version that includes Python 3.10 rather than Python 3.7 which removes the need to install the dependency when installing tox. - [Example log of failing job](https://app.circleci.com/pipelines/github/DataDog/dd-trace-py/21679/workflows/2a7606b5-9ca8-4212-9b02-053665de0198/jobs/1477147) - python/importlib_metadata#409 (cherry picked from commit c2b416e) Co-authored-by: Tahir H. Butt <tahir.butt@datadoghq.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
|
tuve el mismo problema en este caso con celery
esto se soluciona instalando en el proyecto la version de importlib-metadata==4.13.0 |
|
This was very painful but we are getting over it now. |
A Debian Buster-based zuul-jobs test started failing when using stestr
recently [1]. Upon further investigation, this is a Python 3.7
environment which is affected by a recent breaking change to
importlib_metadata.
It seems stevedore worked around this with
Ib9c2b0a14edea91e97d122d2ac93b650029f918e, which was released with
3.5.1 -- but I was still seeing the issue.
Upon further investigation, the "real_groups" dict being returned here
with importlib-metadata 4.12.0 is in buckets by group, e.g.
{'group.one': [EntryPoint(name='foo', ... , group='group.one'),
EntryPoint(name='bar', ... , group='group.one')],
'group.two': [EntryPoint(name='moo', ... , group='group.two'),
EntryPoint(name='goo', ... , group='group.two')],
}
This current code seems to return a dict with entry-points by thier
name, e.g.
{'foo': EntryPoint(name='foo', ... , group='group.one),
'bar': EntryPoint(name='bar', ... , group='group.one),
'moo': EntryPoint(name='moo', ... , group='group.two),
'goo': EntryPoint(name='goo', ... , group='group.two)
}
This reorgansies the fixup routine to put entry-points in a bucket by
their group. With this change, stestr is again finding it's command
plugins.
[1] mtreinish/stestr#336
[2] python/importlib_metadata#409
Change-Id: I3496ab1dfa312b1098a869cdfd9a0c6f81653b28
* Update stevedore from branch 'master'
to 5189992d719ad15e0e3504947895bf5ba9dc7a1d
- Order old importlib-metadata results by group
A Debian Buster-based zuul-jobs test started failing when using stestr
recently [1]. Upon further investigation, this is a Python 3.7
environment which is affected by a recent breaking change to
importlib_metadata.
It seems stevedore worked around this with
Ib9c2b0a14edea91e97d122d2ac93b650029f918e, which was released with
3.5.1 -- but I was still seeing the issue.
Upon further investigation, the "real_groups" dict being returned here
with importlib-metadata 4.12.0 is in buckets by group, e.g.
{'group.one': [EntryPoint(name='foo', ... , group='group.one'),
EntryPoint(name='bar', ... , group='group.one')],
'group.two': [EntryPoint(name='moo', ... , group='group.two'),
EntryPoint(name='goo', ... , group='group.two')],
}
This current code seems to return a dict with entry-points by thier
name, e.g.
{'foo': EntryPoint(name='foo', ... , group='group.one),
'bar': EntryPoint(name='bar', ... , group='group.one),
'moo': EntryPoint(name='moo', ... , group='group.two),
'goo': EntryPoint(name='goo', ... , group='group.two)
}
This reorgansies the fixup routine to put entry-points in a bucket by
their group. With this change, stestr is again finding it's command
plugins.
[1] mtreinish/stestr#336
[2] python/importlib_metadata#409
Change-Id: I3496ab1dfa312b1098a869cdfd9a0c6f81653b28
A Debian Buster-based zuul-jobs test started failing when using stestr
recently [1]. Upon further investigation, this is a Python 3.7
environment which is affected by a recent breaking change to
importlib_metadata.
It seems stevedore worked around this with
Ib9c2b0a14edea91e97d122d2ac93b650029f918e, which was released with
3.5.1 -- but I was still seeing the issue.
Upon further investigation, the "real_groups" dict being returned here
with importlib-metadata 4.12.0 is in buckets by group, e.g.
{'group.one': [EntryPoint(name='foo', ... , group='group.one'),
EntryPoint(name='bar', ... , group='group.one')],
'group.two': [EntryPoint(name='moo', ... , group='group.two'),
EntryPoint(name='goo', ... , group='group.two')],
}
This current code seems to return a dict with entry-points by thier
name, e.g.
{'foo': EntryPoint(name='foo', ... , group='group.one),
'bar': EntryPoint(name='bar', ... , group='group.one),
'moo': EntryPoint(name='moo', ... , group='group.two),
'goo': EntryPoint(name='goo', ... , group='group.two)
}
This reorgansies the fixup routine to put entry-points in a bucket by
their group. With this change, stestr is again finding it's command
plugins.
[1] mtreinish/stestr#336
[2] python/importlib_metadata#409
Change-Id: I3496ab1dfa312b1098a869cdfd9a0c6f81653b28
(cherry picked from commit 5189992)
A Debian Buster-based zuul-jobs test started failing when using stestr
recently [1]. Upon further investigation, this is a Python 3.7
environment which is affected by a recent breaking change to
importlib_metadata.
It seems stevedore worked around this with
Ib9c2b0a14edea91e97d122d2ac93b650029f918e, which was released with
3.5.1 -- but I was still seeing the issue.
Upon further investigation, the "real_groups" dict being returned here
with importlib-metadata 4.12.0 is in buckets by group, e.g.
{'group.one': [EntryPoint(name='foo', ... , group='group.one'),
EntryPoint(name='bar', ... , group='group.one')],
'group.two': [EntryPoint(name='moo', ... , group='group.two'),
EntryPoint(name='goo', ... , group='group.two')],
}
This current code seems to return a dict with entry-points by thier
name, e.g.
{'foo': EntryPoint(name='foo', ... , group='group.one),
'bar': EntryPoint(name='bar', ... , group='group.one),
'moo': EntryPoint(name='moo', ... , group='group.two),
'goo': EntryPoint(name='goo', ... , group='group.two)
}
This reorgansies the fixup routine to put entry-points in a bucket by
their group. With this change, stestr is again finding it's command
plugins.
[1] mtreinish/stestr#336
[2] python/importlib_metadata#409
Change-Id: I3496ab1dfa312b1098a869cdfd9a0c6f81653b28
(cherry picked from commit 5189992)
(cherry picked from commit 93f1e09)
A Debian Buster-based zuul-jobs test started failing when using stestr
recently [1]. Upon further investigation, this is a Python 3.7
environment which is affected by a recent breaking change to
importlib_metadata.
It seems stevedore worked around this with
Ib9c2b0a14edea91e97d122d2ac93b650029f918e, which was released with
3.5.1 -- but I was still seeing the issue.
Upon further investigation, the "real_groups" dict being returned here
with importlib-metadata 4.12.0 is in buckets by group, e.g.
{'group.one': [EntryPoint(name='foo', ... , group='group.one'),
EntryPoint(name='bar', ... , group='group.one')],
'group.two': [EntryPoint(name='moo', ... , group='group.two'),
EntryPoint(name='goo', ... , group='group.two')],
}
This current code seems to return a dict with entry-points by thier
name, e.g.
{'foo': EntryPoint(name='foo', ... , group='group.one),
'bar': EntryPoint(name='bar', ... , group='group.one),
'moo': EntryPoint(name='moo', ... , group='group.two),
'goo': EntryPoint(name='goo', ... , group='group.two)
}
This reorgansies the fixup routine to put entry-points in a bucket by
their group. With this change, stestr is again finding it's command
plugins.
[1] mtreinish/stestr#336
[2] python/importlib_metadata#409
Change-Id: I3496ab1dfa312b1098a869cdfd9a0c6f81653b28
(cherry picked from commit 5189992)
(cherry picked from commit 93f1e09)
(cherry picked from commit 6c9978a)
A Debian Buster-based zuul-jobs test started failing when using stestr
recently [1]. Upon further investigation, this is a Python 3.7
environment which is affected by a recent breaking change to
importlib_metadata.
It seems stevedore worked around this with
Ib9c2b0a14edea91e97d122d2ac93b650029f918e, which was released with
3.5.1 -- but I was still seeing the issue.
Upon further investigation, the "real_groups" dict being returned here
with importlib-metadata 4.12.0 is in buckets by group, e.g.
{'group.one': [EntryPoint(name='foo', ... , group='group.one'),
EntryPoint(name='bar', ... , group='group.one')],
'group.two': [EntryPoint(name='moo', ... , group='group.two'),
EntryPoint(name='goo', ... , group='group.two')],
}
This current code seems to return a dict with entry-points by thier
name, e.g.
{'foo': EntryPoint(name='foo', ... , group='group.one),
'bar': EntryPoint(name='bar', ... , group='group.one),
'moo': EntryPoint(name='moo', ... , group='group.two),
'goo': EntryPoint(name='goo', ... , group='group.two)
}
This reorgansies the fixup routine to put entry-points in a bucket by
their group. With this change, stestr is again finding it's command
plugins.
[1] mtreinish/stestr#336
[2] python/importlib_metadata#409
Change-Id: I3496ab1dfa312b1098a869cdfd9a0c6f81653b28
(cherry picked from commit 5189992)
(cherry picked from commit 93f1e09)
(cherry picked from commit 6c9978a)
(cherry picked from commit 1c12706)
* fix(ci): pin importlib_metadata<5.0.0 (#4257) CI jobs `profile-windows-*`, `kombu` and `celery` are failing due to the release of `importlib_metadata==5.0.0` which removed deprecated shims, resulting in the following exception: ``` AttributeError: 'EntryPoints' object has no attribute 'get' ``` We fix this by pinning the dependency for kombu and celery. In the case of the Windows tests, we address it by upgrading the Windows image to a version that includes Python 3.10 rather than Python 3.7 which removes the need to install the dependency when installing tox. - [Example log of failing job](https://app.circleci.com/pipelines/github/DataDog/dd-trace-py/21679/workflows/2a7606b5-9ca8-4212-9b02-053665de0198/jobs/1477147) - python/importlib_metadata#409 (cherry picked from commit c2b416e) # Conflicts: # .circleci/config.yml * fix conflicts Co-authored-by: Tahir H. Butt <tahir.butt@datadoghq.com> Co-authored-by: Munir Abdinur <munir_abdinur@hotmail.com>
|
5.0.0 removed SelectableGroups which Redash 10 depends on See python/importlib_metadata#409
I think defining a specific version (maybe in addition to the date) would be more helpful. Unfortunately, a lot of python packages (including Celery, which caused a problem in my case) don't specify a maximum version/upper bound for their dependencies, but such warnings could help. (These would even help their users, i.e. us, if they don't include it regardless.) The problem is that even if these dependent packages get updated before the change, if they don't use a max version in their pin and their users have to pin an earlier version of their package it will still break for them when you release a new version. Now I understand that this is mostly their fault, but my gut feeling is that stating a version number (as most projects I know do, BTW) will make it easier for them to handle it. Maybe even be a hint to define that pin... |
entry_pointspreviously returned a dict-like object and other projects have taken advantage of this functionality (stevedore, for example, iterates via.items()). Asentry_pointsnow returns a list-like object, it breaks these previously working dict-like methods.Issue appears with this change: 7e5bae4
The text was updated successfully, but these errors were encountered: