♻️📚 Restructure code base and documentation (#566)

This PR restructures the code base into modules, to be more coherent as to the purpose of each module.

It also restructures the documentation, to make it easier for users to follow, and expose the core concerns at the top-level.

Finally, it introduces document-level configuration *via* the Markdown top-matter, under the `myst` key.
This configuration is generated from the code `MdParserConfig` dataclass, so is inherently consistent with global configuration.
The (YAML) top-matter of a MyST file is always read (within the docutils/sphinx parsers) before the full document is parsed, in order to acquire this file configuration, which overrides the default/global configuration (see `docs/configuration.md`).

## Breaking changes

This should not be breaking, for general users of the sphinx extension,
but will be for anyone directly using the Python API, mainly just requiring changes in import module paths.

The `to_docutils`, `to_html`, `to_tokens` (from `myst_parser/main.py`) and `mock_sphinx_env`/`parse` (from `myst_parser.sphinx_renderer.py`) functions have been removed.
These were really just for testing, and were confusing for users, since they did not run the full sphinx build.
Instead, for single page builds, users should use the recently added docutils parser API/CLI (see `docs/docutils.md`),
and for testing, functionality has been moved to https://github.com/chrisjsewell/sphinx-pytest.
 
The top-level `html_meta` and `substitutions` top-matter keys have also been deprecated (i.e. they will still work but will emit a warning), as they now form part of the `myst` config, e.g.

```yaml
---
html_meta:
  "description lang=en": "metadata description"
substitutions:
  key1: I'm a **substitution**
---
```

is replaced by:

```yaml
---
myst:
  html_meta:
    "description lang=en": "metadata description"
  substitutions:
    key1: I'm a **substitution**
---
```

.github/workflows/docutils_setup.py

@@ -44,12 +44,12 @@ def modify_readme(content: str) -> str:
 if __name__ == "__main__":
     project_path = sys.argv[1]
     readme_path = sys.argv[2]
-    with open(project_path, "r") as f:
+    with open(project_path) as f:
         content = f.read()
     content = modify_toml(content)
     with open(project_path, "w") as f:
         f.write(content)
-    with open(readme_path, "r") as f:
+    with open(readme_path) as f:
         content = f.read()
     content = modify_readme(content)
     with open(readme_path, "w") as f:

.github/workflows/tests.yml

@@ -89,7 +89,7 @@ jobs:
     - name: Install dependencies
       run: |
         pip install .
-        pip install pytest~=6.2 pytest-param-files~=0.3.3 docutils==${{ matrix.docutils-version }}
+        pip install pytest~=6.2 pytest-param-files~=0.3.3 pygments docutils==${{ matrix.docutils-version }}
     - name: ensure sphinx is not installed
       run: |
         python -c "\
@@ -100,7 +100,7 @@ jobs:
         else:
             raise AssertionError()"
     - name: Run pytest for docutils-only tests
-      run: pytest tests/test_docutils.py tests/test_renderers/test_fixtures_docutils.py
+      run: pytest tests/test_docutils.py tests/test_renderers/test_fixtures_docutils.py tests/test_renderers/test_include_directive.py
     - name: Run docutils CLI
       run: echo "test" | myst-docutils-html
 

.pre-commit-config.yaml

@@ -19,6 +19,12 @@ repos:
     - id: end-of-file-fixer
     - id: trailing-whitespace
 
+  - repo: https://github.com/asottile/pyupgrade
+    rev: v2.32.0
+    hooks:
+    - id: pyupgrade
+      args: [--py37-plus]
+
   - repo: https://github.com/PyCQA/isort
     rev: 5.10.1
     hooks:
@@ -44,7 +50,7 @@ repos:
     - id: mypy
       args: [--config-file=pyproject.toml]
       additional_dependencies:
-      - sphinx~=3.3
+      - sphinx~=4.1
       - markdown-it-py>=1.0.0,<3.0.0
       - mdit-py-plugins~=0.3.0
       files: >

.readthedocs.yml

@@ -1,7 +1,7 @@
 version: 2
 
 python:
-  version: 3
+  version: "3"
   install:
       - method: pip
         path: .

docs/_static/custom.css

@@ -1,11 +1,24 @@
-.bg-myst-one {
-    background-color: #52d16f3b;
+/** Add a counter before subsections **/
+h1 {
+    counter-reset: subsection;
+    text-decoration: underline;
 }
-
-.bg-myst-two {
-    background-color: #e7dd7b73;
+h2 {
+    counter-reset: subsubsection;
+}
+h2::before {
+    counter-increment: subsection;
+    content: counter(subsection) ". ";
+}
+h3::before {
+    counter-increment: subsubsection;
+    content: counter(subsection) "." counter(subsubsection) ". ";
 }
 
-.bg-myst-three {
-    background-color: #e7b07b96;
+/** No icon for admonitions with no-icon class */
+.admonition > .admonition-title, div.admonition.no-icon > .admonition-title::before {
+    content: "";
+}
+.admonition > .admonition-title, div.admonition.no-icon > .admonition-title {
+    padding-left: .6rem;
 }