How to use mkdocs as a package to read a Markdown page?

[Guts]

Hello Mkdocs community,

I manage a Mkdocs website with 1000+ pages and sometimes we need to update the structure of our pages. Typically, we're moving our YAML frontmatter date field from str or datetime (date: "2024-02-21 10:00:00") to date (date: 2024-02-21).

We wrote some scripts with regex and it works fine but it's not the first nor the last time we need to do this and it would be really convenient if we can open and write back the markdown file with mkdocs to be sure that we use the same parser.

Is it possible?

[squidfunk]

Why don't you just use a hook and run it through mkdocs build? on_page_markdown might be what you're looking for.

[pawamoy]

I think the front-matter is already stripped by this point. And I guess @Guts wants to permanently replace the dates, not fix them on-the-fly for every build.

[Guts]

@squidfunk I'm a big fan and intensive user of Mkdocs hooks but as said by @pawamoy, I want to permanently update files.

[squidfunk]

It is stripped, but available via page.meta, so you can alter it, and write it back by serializing the YAML after transformation to the originating file ☺️ That was at least my (probably too naive) idea.

[kamilkrzyskow]

I also agree with the hook idea. Since @Guts wrote:

I can't figure out how to write the changes back to the file. Keep looking...

I made a very quick example, even some conditionals are missing and up to implementation.
You can diff compare the docs and docs2 directory to see the changed front-matter.

modify_docs.zip

HOOK

from mkdocs.utils import write_file
from pathlib import Path
import yaml

IS_BUILD = False

def on_startup(command, dirty):
    global IS_BUILD
    IS_BUILD = command == "build"


def on_page_markdown(markdown, page, config, files):
    print("---")
    
    if not page.meta.get("date"):
        print(f"no date, skipping {page}")
        return
    
    print(f"{IS_BUILD=}")
    print(f"{config.extra.get('modify_docs')=}")
    
    print(f"{config['docs_dir']=}")
    
    target_dir = Path(config["docs_dir"]).parent / "docs2"
    
    print(f"{page.file.abs_src_path=}")
    
    page.meta["date"] = "2011-11-11"
    
    target_path = page.file.abs_src_path.replace(config["docs_dir"], str(target_dir))
    
    print(f"{target_path=}")
    
    output_bytes = f"---\n{yaml.dump(page.meta).strip()}\n---\n{markdown}".encode()
    
    write_file(output_bytes, target_path)

[Guts]

Interesting, I dit not imagine using a hook as a one-shot script but it does make sense.

[pawamoy]

I believe Myst builds an AST of the source, which would have been great here (I could definitely use such a thing to create auto-migration for mkdocstrings options in Markdown pages). Unfortunately Python-Markdown does not: it converts directly to HTML. But I'll think about it since I'm interested too, and report back if I find something 🙂

[Guts]

Okay, I'm tired or loosing memory because I've found a script that I wrote myself 2 years ago using

mkdocs/mkdocs/utils/meta.py

Line 56 in e755aae

def get_data(doc: str) -> tuple[str, dict[str, Any]]:

In a nutshell:

from pathlib import Path
from mkdocs.utils.meta import get_data

markdown_filepath = Path("sample_mkdocs_markdown.md")

with markdown_filepath.open(encoding="utf-8-sig", errors="strict") as f:
    source = f.read()

page_meta = get_data(source)[1]

[...]

[squidfunk]

That is exactly what I meant in my comment above – this is what MkDocs does when you use on_page_markdown. The upside is that you don't have to enumerate all files, as MkDocs will do it for you.

[Guts]

Sadly it's quite incomplete since I can't figure out how to write the changes back to the file. Keep looking...

[Andre601]

Sadly it's quite incomplete since I can't figure out how to write the changes back to the file. Keep looking...

Pretty sure this is kind of out of scope for MkDocs.
Remember: It's purpose is to turn markdown files into static html pages, not the other way around or similar.

I've never seen MkDocs modify the actual source file. Only the copy it has in code/memory.

[pawamoy]

I don't think @Guts is asking for such a feature to be integrated into MkDocs, my understanding is that they're simply looking for help to achieve this once, with a script that takes advantage of MkDocs internals / public API.

[Guts]

Thanks @pawamoy for understanding my question and @Andre601 for confirming :).

[Guts]

Until now, I've made it through a script based on python-frontmatter. It works fine even it's not perfect since the output file is written with 0 indentation (despite the indent=4)

Script

#! python3  # noqa: E265

"""Script meant to be used as one-shot to convert YAML fonrtmatter 'date' value from
    str into date.

See: https://github.com/mkdocs/mkdocs/discussions/3576

"""

# -- IMPORTS --

# standard
import logging
from datetime import date, datetime
from pathlib import Path

# 3rd party
import frontmatter
import yaml
from yaml import SafeDumper

# -- MAIN --

dt_format = "%Y-%m-%d %H:%M"
logging.basicConfig(level=logging.INFO)

# match = re.search(
#     r"^---\s*\n.*?date: (\d{4}-\d{2}-\d{2}).*?\n---",
#     content,
#     re.DOTALL | re.MULTILINE,
# )


for md_filepath in Path("content").glob("**/*.md"):
    # read and load content
    with md_filepath.open(mode="r", encoding="UTF-8") as in_file:
        content = frontmatter.load(in_file)

    # manipulate date in YAML frontmatter
    if content_date := content.metadata.get("date"):
        if type(content_date) == date:
            logging.debug(
                f"File {md_filepath} has already a proper date defined in YAML "
                "frontmatters."
            )
            continue
        elif isinstance(content_date, str):
            # parse date from str
            try:
                content.metadata["date"] = datetime.strptime(
                    content_date, dt_format
                ).date()
            except ValueError:
                logging.error(
                    f"File {md_filepath} has a date in YAML frontmatters but it does "
                    f"not comply with expected format ({dt_format}): {content_date}"
                )

            # write new version
            with md_filepath.open(mode="w", encoding="UTF-8") as out_file:
                # frontmatter.dump(content, out_file) # not working
                out_file.write(frontmatter.dumps(content, sort_keys=False, indent=4))
        else:
            logging.warning(
                f"File {md_filepath} has a date in YAML frontmatters but it's not a "
                f"str: {content_date}"
            )

        continue

    logging.info(f"File has no date defined in YAML frontmatters: {md_filepath}")

[Guts]

since the output file is written with 0 indentation (despite the indent=4)

FYI it's the same with your script @kamilkrzyskow that I've just tested it.

[kamilkrzyskow]

Here is the version with indent modify_docs-indent.zip
It changes the order of the keys a bit, because it sort them differently, but it's basically the same thing 👌

[Guts]

No worries, I did not give a sample but you can try with some tags:

docs/index.md:

---
date: '2010-10-10'
tags:
    - hop
    - test
    - sample
---

[...]

docs2/index.md:

---
date: '2011-11-11'
tags:
- hop
- test
- sample
---

[...]

It sounds not blocking but it's messing up the git history and markdown readability (subjective).

[Guts]

Faster than I!

[kamilkrzyskow]

So for the tags example @Guts, I looked up https://stackoverflow.com/questions/25108581/python-yaml-dump-bad-indentation
and made appropriate changes via copy and paste ;)

---
date: '2011-11-11'
indent:
    indent:
        inner: 2
    inner: 1
tags:
    - hop
    - test
    - sample
---

modify_docs.zip

[Guts]

@kamilkrzyskow I was following the same with a custom YAML Dumper and it workks pretty fine.

There is still an issue with encoding:

In:

---
date: '2011-11-11'
indent:
    indent:
        inner: 2
    inner: 1
tags:
    - hop
    - test
    - sample
    - encore plus compliqué avec des caractères spéciaux
    - merci le français
---

Out:

---
date: '2011-11-11'
indent:
    indent:
        inner: 2
    inner: 1
tags:
    - hop
    - test
    - sample
    - "encore plus compliqu\xE9 avec des caract\xE8res sp\xE9ciaux"
    - "merci le fran\xE7ais"
---

[Guts]

allow_unicode=True does the trick!

[kamilkrzyskow]

\xE9 is an escaped unicode character, so by default it escapes them
https://github.com/yaml/pyyaml/blob/48838a3c768e3d1bcab44197d800145cfd0719d6/lib/yaml/__init__.py#L215-L220
here I found a allow_unicode option, I also removed the default_flow_style=False option since it didn't do a difference in my testing.

modify_docs.zip

EDIT: Damn GitHub auto-refresh, never works when it should, 10 minutes too late hah.