Support mkdocs 1.6.0

[HonkingGoose]

Note

Edit by @squidfunk: fix available, please check if your documentation builds successfully, see #7076 (comment)

---

Context

When I try to install mkdocs 1.6.0 I get a warning from pip that mkdocs-material only supports a 1.5.x version of mkdocs.

Copy/pasting a relevant comment from @squidfunk:

Happy to collaborate on the 1.6 transition! Former minor version MkDocs releases sometimes broke parts of Material for MkDocs, which is why we decided to be a little more selective about what version we support. I currently have other things on my plate, but if you'd like to collaborate on a PR to bring Material for MkDocs and Insiders to 1.6, I'm happy to help wherever I can. Reading the release notes of 1.6, I think it should be pretty straight forward to support.

Source for comment: renovatebot/renovatebot.github.io#439 (comment)

I don't feel confident to work on the code on mkdocs-material. I can at least make an issue for this work, so you can track it. Maybe someone from the community can help you work on this feature.

Description

Update Material for MkDocs so it works with 1.6.0. of mkdocs.

Related links

Read the mkdocs 1.6.0 changelog to check for important changes.

mkdocs 1.6.0 also has a new enabled setting for all plugins. I don't know if/how this affects Material for MkDocs. Maybe you should default to enabled: false on some parts, or explictly enabled: true on others...

Use Cases

MkDocs 1.6.0 has new validation options:

• opt-in to validate absolute links
• opt-in to validate absolute links within nav
• recommended to validate anchors in links

It's a good idea to validate the links before publishing the docs, that way readers don't suddenly end up with a broken link.

Visuals

No response

Before submitting

• I have read and followed the change request guidelines.
• I have verified that my idea is a change request and not a bug report.
• I have ensured that, to the best of my knowledge, my idea will benefit the entire community.
• I have included relevant links to the documentation, related issues, and discussions to underline the need for my idea.

[sillydan1]

I ran into this issue this morning as well.

This is specifically a problem with pip<=22.0.2 (the version shipped with ubuntu:22.04 is 22.0.2)

Upgrading the pip version to pip>=22.0.3 fixes the issue.

Recreation steps:

Launch a new ubuntu 22.04 container:

docker run --rm -it -v .:/code -w /code ubuntu:22.04

Inside the container

apt update
apt install python3 python3-pip
python3 -m pip install mkdocs mkdocs-material

Observe that pypa/pip#10851 occurs

additional information

[squidfunk]

Thanks for noting. As said, any help is appreciated!

[oprypin]

Here you go. Don't need to credit me

In here you need to perform this change:

mkdocs-material/requirements.txt

Line 24 in 5f40a53

mkdocs~=1.5.3

-mkdocs~=1.5.3
+mkdocs>=1.5.3

[oprypin]

For more background regarding reasoning (though that comment of mine is on an unrelated project) see Guts/mkdocs-rss-plugin#137 (comment)

[squidfunk]

Thanks @oprypin, I'm aware how to update the MkDocs dependency. I'm just concerned it breaks something in the community edition or in Insiders, which is why we need to test everything before considering the upgrade safe ☺️ We just don't want to be forced into doing a major release, as we had to before 2 (or more, not sure) times.

Thus, if somebody has some time, testing MkDocs 1.6 against the community edition and Insiders is appreciated ✌️ If nobody is up for it, I'll try to find some time on the weekend.

[oprypin]

I tested all changes against 40 websites using mkdocs-material (and I don't have access to mkdocs-material-insiders), on a continuous basis. Is that enough?
https://github.com/mkdocs/regressions/actions/runs/8638288330

[squidfunk]

Sounds good ☺️ Have you also verified the resulting sites? Title handling has undergone significant changes, so I'd be curious if we need to adjust templates or plugins to reflect the upstream changes. Additionally, we need to test Insiders, especially the typeset plugin, and all other plugins that use titles from pages.

As said, I'll try to find some time by the end of the week 💪

[oprypin]

1. I don't know what you're referring to when you say you were forced into doing a major release. MkDocs didn't cause any breakage, especially one that would force dropping older versions of MkDocs. That was done voluntarily to pick up new features.

2. My point isn't about that anyway. By pinning versions, you're trading a possible theoretical breakage for a guaranteed breakage. Please run pip install 'mkdocs>=1.6.0' 'mkdocs-material' and observe what happens. That's kinda nightmarish just by itself.

[oprypin]

Have you also verified the resulting sites

There's no diff in HTML in almost all cases so there's no need to even visually inspect it.

There were diffs on this one site but I think the site's build includes randomization or something https://github.com/mkdocs/regressions/actions/runs/8638288330/job/23682412864

[alexvoss]

@oprypin pip install 'mkdocs>=1.6.0' 'mkdocs-material' is the scenario where someone forces a version of MkDocs that Material has not been tested with. Everyone who installs the version that Material has been tested against will be fine. No point telling people they can use 1.6 when the result of that is as yet unknown. @squidfunk is doing the prudent thing here and has already pledged to look at what the issues might be. I will also have a look as I have an interest in this since I used the new support for generated files (which is cool btw.).

[squidfunk]

I've checked our changelog and it was 2.0 and 3.0. 3.0 turns out be updating to MkDocs 1.0, so all good, long ago, so my memories were probably faded 😅 Regardless, the title handling that came with MkDocs 1.5 was broken for almost a year now, which is why I decided to limit versions to be more careful.

Please run pip install 'mkdocs>=1.6.0' 'mkdocs-material'

Jup, that is expected. As I've outlined in mkdocs/mkdocs#3636 (comment), my opinion is that themes and plugins should define with which version of MkDocs they are compatible. In our installation guide, we do not list in any place that you should install mkdocs explicitly – au contraire, you should install mkdocs-material and let us pick the supported latest version. This way, we can ensure that all plugins work, and yes, we will upgrade to 1.6, but with tens of thousands of projects using our software, we need to be careful about that ✌️

[squidfunk]

@alexvoss jup, the generated files API looks very promising! We can probably get rid of a few dozens lines of code in our plugins, specifically the blog plugin!

[oprypin]

Again,
version pinning for libraries simply doesn't work well enough in pip to be able to recommend it to anyone. Each library that does this turns the whole stack more and more into a fragile house of cards.

[squidfunk]

Again, it is only a temporary solution, until we found the time to test it with the entirety of our code base. Please be patient, we will handle this in the coming days. Thank you for your understanding ☺️

[oprypin]

Please run pip install 'mkdocs>=1.6.0' 'mkdocs-material'

Jup, that is expected.

Did you run this? What is expected?

Is it really expected to you that pip does the following:

Download mkdocs_material-9.5.17. Is it compatible with MkDocs 1.6? No
Download mkdocs_material-9.5.16. Is it compatible with MkDocs 1.6? No
Download mkdocs_material-9.5.15. Is it compatible with MkDocs 1.6? No
Download mkdocs_material-9.5.14. Is it compatible with MkDocs 1.6? No
Download mkdocs_material-9.5.13. Is it compatible with MkDocs 1.6? No
Download mkdocs_material-9.5.12. Is it compatible with MkDocs 1.6? No
Download mkdocs_material-9.5.11. Is it compatible with MkDocs 1.6? No
Download mkdocs_material-9.5.10. Is it compatible with MkDocs 1.6? No
Download mkdocs_material-9.5.9. Is it compatible with MkDocs 1.6? No
Download mkdocs_material-9.5.8. Is it compatible with MkDocs 1.6? No
Download mkdocs_material-9.5.7. Is it compatible with MkDocs 1.6? No
Download mkdocs_material-9.5.6. Is it compatible with MkDocs 1.6? No
Download mkdocs_material-9.5.5. Is it compatible with MkDocs 1.6? No
Download mkdocs_material-9.5.4. Is it compatible with MkDocs 1.6? No
Download mkdocs_material-9.5.3. Is it compatible with MkDocs 1.6? No
Download mkdocs_material-9.5.2. Is it compatible with MkDocs 1.6? Yes. Great, let's use it!

Is it expected that, until the end of days, users will occasionally install mkdocs_material-9.5.2 because that was the last version that supported all versions of MkDocs?

[squidfunk]

No, I did not, because it's nothing we recommend doing on our documentation:

pip install mkdocs-material

This will automatically install compatible versions of all dependencies: MkDocs, Markdown, Pygments and Python Markdown Extensions. Material for MkDocs always strives to support the latest versions, so there's no need to install those packages separately.

Is it expected that, until the end of days, users will occasionally install mkdocs_material-9.5.2 because that was the last version that supported all versions of MkDocs?

Nope, as said in #7076 (comment), we're working on it!

[squidfunk]

Additionally:

Is it really expected to you that pip does the following:

I agree that the output is rather ugly. That's something that should be reported upstream to the maintainers of pip.

[squidfunk]

FYI, I pinned the issue so that users coming to our issue tracker see immediately that we're working on it ☺️

[rarkins]

As a sponsor of this library, I would be disappointed if the maintainer had given in to the pressure above to YOLO every release instead of testing/verifying it. Thank you for the good tradecraft exhibited.

[ssbarnea]

There is a workaround that could be used to make everyone happy: add a lock extra that adds the constraint while removing it from the default dependencies.

The risk-afraid ones can install mkdocs-material[lock] and the ones that like the 'freedom' can install just mkdocs-material. Some projects can even have pinned dependencies inside this extra, but in the end is up to you to decide if you want ranges or pins.

One extra benefit is that doing this could allow keeping the CI/CD pipelines green, as no new dependencies will randomly break the. Still, I recommend keeping one 'devel' pipeline that is not using the constraints, one that would report on upcoming breakages.

[oprypin]

Users should simply lock dependencies themselves by running pip-compile or equivalent. That's the only way to protect CI and to avoid any unexpected changes.

[squidfunk]

I'm coming from the rather straight forward dependency management in Node.js (which, yes, also has its problems), and I really have trouble with Python's dependency management. I do not expect users of Material for MkDocs to be experts in how to use pip, since many of them are non-technical, so our goal should be to make it as simple as possible.

IMHO, offering more installation options like mkdocs-material[lock] would add to the confusion 😉 Thus, let's try harder and keep it simple. Additionally, note that we're also recommending to lock dependencies in our installation guide, if you decide to install a specific version of Material for MkDocs:

[sisp]

No, I did not, because it's nothing we recommend doing on our documentation:

pip install mkdocs-material

This will automatically install compatible versions of all dependencies: MkDocs, Markdown, Pygments and Python Markdown Extensions. Material for MkDocs always strives to support the latest versions, so there's no need to install those packages separately.

I don't think that recommendation is good because using mkdocs-material still requires using the mkdocs CLI, and this CLI could change in an unexpected way (although this should happen only in a major release, but there's no guarantee in any case). Thus, I believe mkdocs should also be a direct dependency, so we can explicitly manage its version as well.

I'm coming from the rather straight forward dependency management in Node.js (which, yes, also has its problems), and I really have trouble with Python's dependency management.

In case you haven't come across it yet, there's a (IMO) fantastic article on Python dependency management with regard to upper bounds: https://iscinumpy.dev/post/bound-version-constraints In contrast to package managers in Node.js, which can install multiple versions of the same library at the same time, Python uses global dependency resolution, so upper bounds can render the dependency tree unresolvable pretty quickly. So, in most cases no upper bounds should be used, temporary upper bounds until a proper fix is available are okay, and upper bounds can always be introduced in userland.

[squidfunk]

Thus, I believe mkdocs should also be a direct dependency, so we can explicitly manage its version as well.

So we should remove mkdocs from our requirements.txt? I kindly disagree, because Material for MkDocs ships many plugins that rely on MkDocs APIs. By removing our explicit version requirement, Material for MkDocs might break unexpectedly when MkDocs releases a new version.

Once we upgraded, my suggestion would be to change the requirement to:

mkdocs~=1.6

This would get us all new releases except for the next major release. Setting it to 1.5.3 was just a temporary safety net. Please also understand that we have many, many non-technical users that have no idea how Python version managment works, so essentially, we're trying to make it simpler for them by providing this safety net.

[squidfunk]

In my view, Material for MkDocs is an application and not a library, as it is not consumed by other Python packages. MkDocs on the other hand is an application and a library, as it exports utilities that are used by plugins, etc. Material for MkDocs is essentially a leaf root dependency, which means: something that is consumed by endusers that are not necessarily developers.

There are no Python packages that have Material for MkDocs as a dependency. Thus, we're not at the root, but at the leaf. I'm also not sure I understand what you mean by "user's websites are the root of the dependency tree". User's websites that are built with MkDocs and Material for MkDocs are an artifact and outside of the dependency tree.

Edit: ignore my words, yes, I'm mistaken, it's a root and not a leaf dependency (got it twisted in my head). Regardless the choice of the wrong word, the rest remains: IMHO, not a library, which is why we should be cautious and prudent about our upstream dependencies.

[oprypin]

Fair, maybe it's a rare example of something that doesn't exactly fit into either of the two cases

[rarkins]

I agree, Material for MkDocs is an application and not a library. It should always install on its own successfully, and it should be validated to work with whatever version of MkDocs it "ships with" and not "maybe it will work with whatever is the latest MkDocs, maybe it will have subtle bugs, maybe it will have severe errors". In other words, exactly what's happening now.

[squidfunk]

Let's agree that we disagree here, i.e., have conflicting views, which is perfectly fine and good to talk about ☺️ I'm of course open to removing all upper bounds again if the majority of our users actually wants to manage dependencies by themselves, I just don't believe this is the case given the demographics of this project. That needs to be balanced against.

As said, I recommend we go back to mkdocs~=1.6 once we consider the upgrade safe. That means we don't run into this problem again until MkDocs releases 2.0. There's still a problem in Insiders discovered by @alexvoss that we need to fix, but we should be good to go in the coming days ✌️

[squidfunk]

I just fixed the most apparent error that we discovered in Insiders. Let's short-circuit this discussion by getting to work. Here are two PRs, one for the community edition and one for Insiders! Let's test it for a few days and if we think we caught all problems, release it 🚀 As discussed, I set the version range to ~=1.6.

Community edition

pip install git+https://github.com/squidfunk/mkdocs-material.git@chore/update-to-mkdocs-1.6

• Updated MkDocs to 1.6.0 #7082

Insiders

pip install git+https://${GH_TOKEN}@github.com/squidfunk/mkdocs-material-insiders.git@chore/update-to-mkdocs-1.6

• https://github.com/squidfunk/mkdocs-material-insiders/pull/87

[pawamoy]

Interesting discussion. Can't help but give my 2 cents 😄

I agree that Material for MkDocs is kinda in between applications and libraries. Application because for end-users, not consumed by other Python packages. Library because it still blends with all other dependencies of any given project using it, in the same resolution algorithm used to resolve these other dependencies.

In my experience though, as someone who glues a lot of Python "tools" together, many projects that consider themselves applications (or CLI tools, or any other name that doesn't imply "library") are actually libraries. In my experience again, all Python packages are libraries. I'll go further: all packages are libraries, whatever the ecosystem. If you're in the ecosystem, you (willingly or not) play by its (resolution) rules, and you should be a kind fellow citizen. If you can depend on an "application", whether you consume it or not, then you can use it as a library. We should never underestimate the imaginative ways developers can consume or depend on something 😉

True applications are projects which are not packaged within this ecosystem. They are above it. True applications simply provide an install script, a requirements.txt file, a ZIP file, or whatever suits the devs of the application and their users, but they do not exist within the underlying ecosystem(s) (ecosystems can be layered).

I see Material for MkDocs as a building block to build websites, not a final product: you install it alongside MkDocs core, as well as many other plugins, Markdown extensions, and various Python tooling to lint, test, build, deploy your docs.

To me, Material for MkDocs is indeed a library 😛 Especially now that it provides many plugins that can be used independently of the theme itself.

---

Don't get me wrong: I absolutely don't mind temporary pins or upper bounds: I sometimes use them myself, and they do the job perfectly when I want to prevent a known breakage while waiting for / working on a fix, or more generally waiting for / working on supporting the new version of the pinned/capped dependency.

But yeah, the Python ecosystem suffers from this flat packages layout, and so pins and upper bounds are highly discouraged. I really recommend the article mentioned by @sisp: https://iscinumpy.dev/post/bound-version-constraints, it explains it well.

[pawamoy]

As for the demographics of Material for MkDocs, I don't have anything smart to say. Installing Material for MkDocs is easy, but understanding that there's a compatibility issue, understanding where to search for ways to resolve it, and understanding the suggested ways to resolve it (if there are any) is definitely not as easy. So I understand the need to simplify this for your users 🙂

Would be interesting to know the percentage of non-technical users that manage their Python dependencies themselves vs. those who rely on technical peers for that. But that's for another day 😄

[nicfv]

I'm not sure if this is related, but I'm getting this error in my mkdocs workflow:

Successfully built paginate
ERROR: Exception:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/pip/_internal/cli/base_command.py", line 165, in exc_logging_wrapper
    status = run_func(*args)
  File "/usr/lib/python3/dist-packages/pip/_internal/cli/req_command.py", line 205, in wrapper
    return func(self, options, args)
  File "/usr/lib/python3/dist-packages/pip/_internal/commands/install.py", line 389, in run
    to_install = resolver.get_installation_order(requirement_set)
  File "/usr/lib/python3/dist-packages/pip/_internal/resolution/resolvelib/resolver.py", line 188, in get_installation_order
    weights = get_topological_weights(
  File "/usr/lib/python3/dist-packages/pip/_internal/resolution/resolvelib/resolver.py", line 2[76](https://github.com/nicfv/Me/actions/runs/8807702921/job/24175351696#step:2:77), in get_topological_weights
    assert len(weights) == expected_node_count
AssertionError
Error: Process completed with exit code 2.

But it seems like mkdocs 1.6.0 and mkdocs-material 9.5.18 both installed successfully.

[oprypin]

Yes it is related, see #7076 (comment) mkdocs/mkdocs#3477 (comment)

[squidfunk]

@nicfv you can already test a fix, as noted in #7076 (comment) – we'll release it in the coming days ☺️

pip install git+https://github.com/squidfunk/mkdocs-material.git@chore/update-to-mkdocs-1.6

[squidfunk]

Our testing shows that we consider it quite safe to upgrade to MkDocs 1.6 🎉 If we hit problems, we'll fix them on the way – as always. The new anchor validation functionality is very useful, and allowed me to fix tons of anchor links in 2585b82 that went stale due to several restructures. Some of the warnings cannot be fixed, because anchors that do exist are not picked up by MkDocs, but that's no problem for us, we'll just live with that.

[squidfunk]

Released as part of 9.5.19. 4 days from MkDocs' release of 1.6 to us adopting it – not bad 😎 If you find any issues that are related to the MkDocs version bump, please create a new issue here or upstream, depending on what the cause is. If unsure, please create a discussion first. This issue can be considered resolved now.

[intentionally-left-nil]

For future reference, all of these really tight dependencies are making it really hard to upgrade packages independently. It's not clear to me there's a high value in doing this.

For example, I'm still blocked because mkdocs-techdocs-core requires a very specific version of mkdocs-material:
https://github.com/backstage/mkdocs-techdocs-core/blob/main/requirements.txt

Obviously that's it's own problem (using exact dependencies).

But anyways, by changing your requirements to be mkdocs ~1.6, you're saying that mkdocs-material is no longer compatible with 1.5. This isn't true and is unnecessarily restricting the allowable solve space.

Instead, it would be better to be mkdocs >= 1.5

Since you all are really insistent on not allowing future versions of mkdocs without a manual step, you could do mkdocs >= 1.5 && mkdocs < 1.7 (and then you can keep bumping both the minimum and maximum support level as necessary)

[squidfunk]

@intentionally-left-nil I understand your concerns. However, while the community edition should still be compatible with 1.5, Insiders is definitely not, and we try to keep it synchronized to ease interop and switching back and forth for our sponsors. Our goal is to stay up-to-date with the latest version of MkDocs, as it's currently our most critical dependency.

Additionally ~=1.6 is not really a tight version constraints, as it includes all versions until < 2. All version constraints in our requirements.txt are limited to the current major version, which is absolutely reasonable and standard in the software world. Major versions are meant for breaking changes, so keeping it to the current major version is sensible.

[sisp]

All version constraints in our requirements.txt are limited to the current major version, which is absolutely reasonable and standard in the software world. Major versions are meant for breaking changes, so keeping it to the current major version is sensible.

Again, I really recommend reading https://iscinumpy.dev/post/bound-version-constraints/ and https://hynek.me/articles/semver-will-not-save-you/. For Python libraries, use only lower bounds whenever possible; for apps (root of the dependency tree), use pins (lock file or even manifest pins + lock file). Clearly, mkdocs-material is not an app under this definition. Limiting to current major versions is common in, e.g., Node.js, but Python isn't Node.js (see the first article), and SemVer is a theoretically great concept but can in practice only serve as a hint at best (see the second article).

@pawamoy also summarized it very well: #7076 (comment)

[squidfunk]

@sisp Thanks again for your input! We'll re-evaluate this in the near future. Please understand that we're currently very, very heavily working in the background, doing a large refactor of some of the project's features, documentation, working on an entirely new search etc., and other big things we have not communicated yet.

We're also starting to build a team to keep up with the need and demands of enterprises and organizations, and all of this is a lot, lot, lot of work. We're still only 3-4 (mostly part-time) individuals, depending on how you count, with me working full-time on Material for MkDocs.

Given this context, I hope you understand that changing the way we manage dependencies is currently not at the top of our priority list. However, I've heard the criticism and we'll likely offer both options in the future. Probably something like:

pip install mkdocs-material[stable] # bounded
pip install mkdocs-material         # unbounded

The specifics still need to be worked out, and while I understand that you have deep knowledge of Python package management, most Material for MkDocs users don't. They just want it to work, and for this we try to make it as simple as possible. It's after all a strategic decision. As said, we're going to offer more fine-grained options in the future, but for the time being this issue should be resolved.