Loosen importlib-metadata version requirement slightly?

[justinmayer]

When adding the importlib-metadata dependency version, it's not clear to me whether Python-Markdown needs that version to be >=4.4 in order to function properly. If so, please feel free to close this issue with my apologies for the noise.

If, on the other hand, Python-Markdown also works just fine with, say, importlib-metadata 4.2, could the version requirement be loosened a bit?

As Pelican's project lead, I've been busily updating many Pelican plugins' development dependencies to Flake8 4.0, but since it requires importlib-metadata <4.3, and since Python-Markdown 3.3.6 requires importlib-metadata >= 4.4, this combination leads to a dependency resolution error and is wreaking havoc in our CI systems.

I would really rather avoid pinning Markdown or Flake8 in dozens of repositories in order to avoid this problem, particularly if loosening this requirement slightly in Python-Markdown will neatly avoid it. Would that be possible?

[mitya57]

It looks like the functionality we need was first introduced in importlib_metadata 3.6:
https://importlib-metadata.readthedocs.io/en/latest/history.html#v3-6-0

However, there were some internal changes in between 3.6 and 4.4, and only version 4.4 is fully compatible with importlib.metadata from Python 3.10 standard library:
https://github.com/python/importlib_metadata#compatibility

After reading python/importlib_metadata#319, PyCQA/flake8#101, PyCQA/flake8@975a3f45334861eb#commitcomment-51876533 and PyCQA/flake8#1438 linked by you, it seems to me that flake8 is doing the wrong thing here.

We could temporarily loosen the dependency to give flake8 time to adapt to the new version of importlib_metadata. But if they reject pull requests that implement such adaptation and don't give any explanation, sooner or later we will have a conflict anyway.

Anyway, I won't mind any resolution of this problem, and I'll leave the final decision to @waylan.

[justinmayer]

There's no question that Flake8 should eventually stop pinning importlib.metadata to <4.3 — I have long contended that specifying upward bounds can cause these and other problems. Hynek touches on those problems as well in one particular section of his excellent article on versioning.

But nearly as problematic is specifying minimum dependency versions that don't represent the true minimum versions needed to function properly. The likelihood of dependency resolution conflicts goes down dramatically when everyone chooses permissive minimum dependency versions.

Flake8's maintainers have their reasons for the upward bound, and those reasons must be significant enough to bear the heat for doing so, so I'd rather side-step the question of whether their position is justified. While Flake8 is the genesis for this particular issue, it could have been (and could be in the future) another project that needs to temporarily pin a dependency, and we can all collectively make the Python ecosystem a more stable and resilient place by being as permissive as possible when specifying minimum dependency versions.

[waylan]

only version 4.4 is fully compatible with importlib.metadata from Python 3.10 standard library:
https://github.com/python/importlib_metadata#compatibility

This was the basis for the current requirement. In other words, the developers of importlib_metadata are telling us that we need version 4.4 to be compliant with Python 3.10. I am not interested in experimenting with earlier versions to see if maybe we can get away with using them.

[waylan]

But nearly as problematic is specifying minimum dependency versions that don't represent the true minimum versions needed to function properly. The likelihood of dependency resolution conflicts goes down dramatically when everyone chooses permissive minimum dependency versions.

That is a valid point and I generally try to avoid defining either minimum and maximum versions for dependencies (which has been highly criticized by some users who insist we should always pin to a specific version). However, sometimes a minimum is required as a new feature/API won't work with previous versions. That is the case here. That said, it seems that the API was first implemented against an early (alpha) build of Python 3.10 and was updated across multiple releases after that. It would appear that Flake8 locked in on one of those early releases. However, importlib_metadata finally settled on version 4.4 as the minimum required to match the API and behavior of Python 3.10. Therefore, 4.4 is the minimum version.

Now, it is possible that the specific API which we use was solidified in an earlier release and other APIs were in flux in the later releases. But I don't know that for certain and am not going to try to work that out. Therefore, we are sticking with importlib_metadata's advertised minimum version of 4.4. If Flake8 wants to be stubborn and refuse to move forward, then that is on them.

As an aside, we use Flake8 for our own code in this project. However, we are running it in a separate virtual environment (via tox) where we only install flake8, but not our own package. Therefore, we can run Flake8 against our code without encountering the conflict. We've been doing it this way for years and never noticed that the conflict existed.