Making tables from files (and more)

[roaldarbol]

Contribution guidelines

• I've read the contribution guidelines and wholeheartedly agree

I want to suggest an idea and checked that ...

• ... to my best knowledge, my idea wouldn't break something for other users
• ... the documentation does not mention anything about my idea
• ... there are no open or closed issues that are related to my idea

Description

First of all, thanks for all the brilliant work you're putting into this project - it's my go-to for everything documentation now!
This is a multi-feature request, which I'd be happy to discuss further. I'm no JS wizard so I'm not sure if I can help much myself on that side.

I'm proposing enabling the following workflow:

• Query and read data file (e.g. filtering, choosing columns, and more)
• Render table in Markdown
• Render the Markdown table with .js (like now, but maybe with better options for customisation)

---

Tables from files
Being able to parse and query a data file (e.g. csv) to Markdown. There might be inspiration to get from the Obsidian obsidian-csv-table plugin. They take the approach where you can read the file by:

```csvtable
source: my_csv_file.csv
filter: population < 100000000
```

More js options
The above point is the important one! However, although being able to sort the table is good, it could easily be more. I'd think this should be relatively easy to implement (though I haven't tried and might be very wrong) e.g. with DataTables which is open source and can be further customized with extensions.

Use Cases

The data used in tables most often lives in data files (be it .csv, .txt, .xlsx etc.). And currently we have to type out the content of these files to render them as a table. If we'd be able to call a data file and render it directly, when something changes in the data, we wouldn't have to scour the pages to find the entries and replace them, they'd be changed automatically. I think this would be useful to almost everyone who has the need to make tables.

Personally, I'm currently working on an open hardware project where these have proved vital. We have a collection of parts in a file and want to showcase subsets of parts needed to make each build.

Screenshots / Mockups

No response

[facelessuser]

You could do this with a custom fence with SuperFences: https://facelessuser.github.io/pymdown-extensions/extensions/superfences/#custom-fences.

The one thing to note is that when using SuperFence's custom fences, whatever is returned by the custom fence logic is stored away and replaced with a placeholder such that further Markdown parsing is not run on it. Placeholders are replaced with the actual content at the end.

You can do almost anything with them. We use it in the pymdown-extension documentation to create special blocks that spawn a new Markdown process such that we can display Markdown rendered with different options (specified with YAML frontmatter) than what the documents use: https://github.com/facelessuser/pymdown-extensions/blob/main/tools/pymdownx_md_render.py. Then we just provide our blocks:

```md-render
---
extensions:
- pymdownx.magiclink
- pymdownx.saneheaders
- markdown.extensions.attr_list

extension_configs:
  pymdownx.magiclink:
    repo_url_shortener: true
    repo_url_shorthand: true
    social_url_shorthand: true
    social_url_shortener: true
    user: facelessuser
    repo: pymdown-extensions
---
- Just paste links directly in the document like this: https://google.com.
- Or even an email address: fake.email@email.com{.magiclink-ignore}.
```

So, you could simply create your own that simply parses the code block as a CSV and return it as an HTML table.

[squidfunk]

Thanks for suggesting. Material for MkDocs is a theme, so what it does is mainly rendering. All Markdown processing is done by upstream projects, mainly Python Markdown and Python Markdown Extensions by @facelessuser. Preprocessing tabular data into Markdown tables is definitely Markdown processing, which is beyond the scope of this project. Material for MkDocs tries to make the most functional and beautiful website of the Markdown you're throwing at it, but that's basically it. Furthermore, preprocessing is theme-agnostic, since you're taking a format and converting it into Markdown, so it could probably be implemented as a Markdown extension.

I would love to improve data tables, but the Markdown table syntax doesn't allow for much flexibility. The DataTables project depends on jQuery, which we don't have as a dependency for several reasons. Do you know of any other good data tables libraries, ideally without any external dependencies? I could take a look at whether we can add native support, similar to TableSort, which is also a third-party library.

[squidfunk]

Moving this into a discussion

[roaldarbol]

Thanks for the replies! @squidfunk that of course makes sense, I keep forgetting that its a theme. Maybe it would be worth considering adding this case to the docs (once we perhaps end up at a good solution). As for alternatives, I especially like Simple DataTables - it's also lightweight, has no external dependencies and seems to be be well-maintained so that might be worth considering. (examples here)

@facelessuser that seems a good approach. I've not previously worked with PyMdown or its extensions so I'm note quite sure how to go about this: Say I have a .csv file in docs/data. How do I parse that? Do I make a SuperFence with plain python code? And do I also explicitly need to parse that into HTML too?
Ideally I'd hope to get to a solution like the one I mentioned above where, just as with the mermaid example you call the name just after the backticks, but I don't know if that's too intricate?

[roaldarbol]

Sorry, I think there's just something plain and simple I'm missing.

Yes, assuming I understand what you are asking. The example I gave was done with pure Python

I was not sure if the python code would go actually within the fences like in Rmarkdown or Jupyter Notebooks. I'm now beginning to get that I would write a script somewhere else called csvtable.py which is then called upon? As you say, that would achieve what I was asking for initially. And are there any best practices on where to keep the custom scripts?

Not sure what you mean, but you would return the HTML desired as a string.

If I use python (csv.reader()) to import the data it still needs to be converted to a table. You mentioned that it should be converted directly to HTML as Markdown wouldn't be re-rendered if I understood it correctly.

Sorry about the confusion, I think there's some implicit knowledge I'm missing here. Not asking for custom solutions, just need to understand how to achieve it. Thanks for the help, I really appreciate it!

[facelessuser]

I was not sure if the python code would go actually within the fences like in Rmarkdown or Jupyter Notebooks. I'm now beginning to get that I would write a script somewhere else called csvtable.py which is then called upon? As you say, that would achieve what I was asking for initially.

Yes, external code is linked to a custom fence name which you can use just like mermaid is used.

If I use python (csv.reader()) to import the data it still needs to be converted to a table. You mentioned that it should be converted directly to HTML as Markdown wouldn't be re-rendered if I understood it correctly.

Yes, you would convert it to HTML syntax. So it would be a string that of HTML: "<table>...</table>". The caveat with using the SuperFences custom fence is that once you return that string, no further processing can be done on it. For things like what you are wanting, mermaid, etc., this is usually fine. In your case, you are likely just displaying data, not trying to provide more content for the Markdown parser to process.

In my example, which I linked, I get around the limitation of the content not further being processed by the Markdown parser by launching another Markdown process, but it is essentially sandboxed from the content outside of the fence, but this is perfect for what I wanted. Anyways, for most people, such a thing is not needed.

The point is that you can do anything you want to to the content in the fence, you just need to format the result exactly as you want it as that is what will end up in the final output.

[roaldarbol]

Ah, that makes sense, thanks!
Last question: The generated HTML table, will that then be post-processed by JS e.g. the tablesort which mkdocs-material supports?

[roaldarbol]

From the documentation my understanding is that the javascript would find the <table> blocks and render them.

[facelessuser]

JavaScript is run at browser runtime. That is long done after Python Markdown is through with it. The table will be available for tablesort, no problem.

[facelessuser]

Not asking for custom solutions, just need to understand how to achieve it.

Understood. I like to set expectations up front as sometimes I am asked for more than I am willing to give 🙂.

[roaldarbol]

No worries, one has to set boundaries in this space! I found a good answer of yours to an issue on pymdown-extensions which clarifies things here - I think it provides a good minimal example for how to implement your own custom fencing which could be great documentation for us rookies. :-)

[facelessuser]

Yeah, hopefully, it is up to date, but at the very least, it hopefully gives you enough of an understanding of the basics.

[chrimaho]

I've found a really good solution for this.

You can read more on this blog: The best MkDocs plugins and customizations.
Which references the plugin mkdocs-table-reader-plugin, as documented on GitHub and their Docs site.

All you need to do is:

pip install mkdocs-table-reader-plugin

Then extend the mkdocs.yml like:

plugins:
  - search
  - table-reader

And then add a CSV to your page like:

{{ read_csv('path_to_table.csv') }}

Or an Excel like:

{{ read_excel('path_to_table.xlsx') }}

This plugin uses the Pandas library, so this function should comply with that syntax.

No need to roll your own Super Fences or custom Java Script processes.

[roaldarbol]

That's really amazing - I'll revisit this then, since I never got on from from it and kind of stalled. Thanks!

[chrimaho]

Also related:

• Extend the Data Tables page to include info on how to load data from static file #5080
• Extend the Data Tables page to include info on how to load data from static file #5081

[roaldarbol]

So just to check, as it uses Pandas, is filtering possible - something like:

{{ read_csv('path_to_table.csv').query('population < 10000') }}

[timvink]

Hi, author of mkdocs-table-reader-plugin here. Filtering using .query() is definitely possible, but not arbitrary. The plugin will not just run any python inside {{ }} because that is potentially unsafe. Instead, I extract the function and arguments using regex and use ast.literal_eval().

I'm not sure if it's a good design choice though. You could use a hook to preprocess tables also (see preprocessing tables using hooks), which is probably a better design pattern (because then you don't have any data logic inside your documentation code, it's just input/output).

I've created an issue on the repo, let me know if you still find this useful compared to using a hook.

Thanks for pointing me to https://github.com/coddingtonbear/obsidian-csv-table, I like that syntax as well. I've already created another plugin using custom fences (https://github.com/timvink/mkdocs-charts-plugin) and I can confirm that it's really easy to do. I've created another issue for that one, if there are some upvotes I'll implement it: timvink/mkdocs-table-reader-plugin#45