importlib-metadata bumped to >=4.4 in release version 3.3.5 - compatibility breaking change

[ilCatania]

As part of the changes released with Markdown version 3.3.5 last week, importlib-metadata was bumped to >=4.4:

install_requires=["importlib-metadata>='4.4';python_version<'3.10'"],

and a reference to the group kwarg was added in the code:

INSTALLED_EXTENSIONS = metadata.entry_points(group='markdown.extensions')

These are compatibility breaking changes and are breaking downstream projects. In my case I am running into compilation issues when picking up Markdown as part of apache-airflow==2.1.0. Why was this not released as 4.0.0? What options do I have? Thank you very much for your help!

[mitya57]

Are you suggesting us to bump version to 4.0.0 just because of an extra dependency?

This dependency is quite widespread, comes from Python developers and I don't see why having it may be a problem. Maybe you have the same issue as #1200, i.e. conflict with pyflakes? In this case I would suggest pinning Markdown to an earlier version until pyflakes bug is fixed.

[ilCatania]

Exactly because importlib-metadata is a widely used library, bumping its version to 4.x will break several downstream builds.

As far as I can tell, importlib-metadata was already a dependency before Markdown version 3.3.5 and in fact has been a dependency for the last two years, only it didn't have a version specified. Markdown 3.3.4 works just fine with importlib-metadata==1.7.0 but 3.3.5 breaks awfully upon initialization unless importlib-metadata version is bumped. Here's my error when I just try to import a package inside my virtual environment, without having made any code changes on my side:

venv/lib/python3.7/site-packages/airflow/www/utils.py:22: in <module>
    import markdown
venv/lib/python3.7/site-packages/markdown/__init__.py:29: in <module>
    from .core import Markdown, markdown, markdownFromFile  # noqa: E402
venv/lib/python3.7/site-packages/markdown/core.py:26: in <module>
    from . import util
venv/lib/python3.7/site-packages/markdown/util.py:88: in <module>
    INSTALLED_EXTENSIONS = metadata.entry_points(group='markdown.extensions')
   TypeError: entry_points() got an unexpected keyword argument 'group'

From the user's perspective, the expectation with projects using semver is that any potentially breaking change will cause a major version bump. My suggestion would be to either:

1. release a new major version, or
2. loosen the importlib-metadata version requirement, wrap the INSTALLED_EXTENSIONS statements in a try/except block and handle the case where an earlier version of importlib-metadata is installed

Hope this makes sense, please let me know if I can provide more examples. Thank you

[mitya57]

Can you explain why upgrading importlib-metadata is a problem?

As far as I can tell, its API is backwards compatible, so even if some other part of your project calls it without group argument, that code will continue to work after upgrade.

[ilCatania]

@mitya57 I am not explicitly importing any version of importlib-metadata, but the dependencies in my environment (mainly apache-airflow) explicitly request importlib-metadata~=1.7 so I am not at liberty to just upgrade it.

[facelessuser]

1. For Python Markdown, this was an implementation detail, and we can not anticipate that a project would be locked to some old dependency and not allowed to upgrade. There was nothing breaking in our API, so we did not view it as a breaking change. Sure, some people could argue that it is breaking for them maybe now, but we cannot anticipate everything. Maybe in the future, such a change could be considered breaking, and I'm sure we'll take that under consideration.
2. It's already released, so changing it to 4.0 is not practical. It is easy to argue in hindsight that maybe the change was bigger than it was, but it is already released, and people are using it. I would lock your environments to the last known working version of Python Markdown, but it makes no sense to try and retroactively tag the release as 4.0.

[ilCatania]

Thanks @facelessuser, I do agree this is more of an hindsight conversation at this point. I've pinned Markdown==3.3.4 in my environment and things work as expected. Will close this issue for now

[waylan]

For the record, version 4.4 of importlib-metadata is required to match the behavior of Python 3.10 and Python 3.10 was released last month. Additionally, importlib-metadata is now at version 4.8.2, with 11 releases being made after 4.4. Therefore, I assumed that most packages would support Python 3.10 and at a minimum expect version 4.4 of importlib-metadata. Clearly that assumption was wrong.

[Krilivye]

Hello team, thanks for the work!
Regarding this closed issue, I also ran into a dependency problem regarding the latest markdown pushing importlib-metadata>=4.4 and latest flake8 constraining it to importlib-metadata<4.3

I'm a bit confused as I see that markdown support python 3.6->3.10.
According to importlib-metadata Changelog the https://importlib-metadata.readthedocs.io/en/latest/history.html needs only 3.0.0 to support python 3.6.

So I would suggest restricting importlib-metadata to >=3.0.0 to keep backward dependencies compatible code with the ecosystem (who is not up to date with python 3.10 which requires 4.4 ;) ).

I'm downgrading markdown to 3.3.4 due to that but it makes me sad.

Theorie reasoning behind.
I'm a user of both flake8 and markdown (requirements.txt only contains flake8 and markdown).
constraints are free (no pinned version)
on a python3.6->3.9 I won't be able to install the latest markdown -> conflict
on python3.10 flake8 will be the bad guy... but that is expected as flake8 does not support python3.10 yet (nor 3.9).

With the lighter constraint, i'll be able to have markdown installed in every python3.6 -> 3.10 ecosystem

I discovered that pip6 support python version constraint so it could be something like that (didn't test it):

importlib-metadata >=1.8, <4.4; python_version <= '3.10'
importlib-metadata >=4.4 ; python_version >= '3.10'