⬆️ UPGRADE: sphinx v4 (& drop sphinx v2, change mathjax override) (#390)

This commit updates the requirements of myst-parser from `sphinx>=2,<4` to `sphinx>=3,<5`, and updates the test matrix to sphinx 3 & 4. The major change to the core code, is that MathJax is now handled differently: Instead of removing all `$` processing for the whole project, during MyST document parsing, the top-level section is given the classes`tex2jax_ignore` and `mathjax_ignore` (turning off default mathjax processing of all HTML elements) and mathjax is configured to process elements with the `math` class.
executablebooks · Jun 12, 2021 · 340e719 · 340e719
1 parent 5f37c0b
commit 340e719
Show file tree

Hide file tree

Showing 50 changed files with 1,735 additions and 137 deletions.
diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
@@ -25,15 +25,15 @@ jobs:
     strategy:
       matrix:
         python-version: [3.6, 3.7, 3.8, 3.9]
-        sphinx: [">=3,<4"]
+        sphinx: [">=4,<5"]
         os: [ubuntu-latest]
         include:
           - os: ubuntu-latest
             python-version: 3.8
-            sphinx: ">=2,<3"
+            sphinx: ">=3,<4"
           - os: windows-latest
             python-version: 3.8
-            sphinx: ">=3,<4"
+            sphinx: ">=4,<5"
 
     runs-on: ${{ matrix.os }}
 
@@ -46,8 +46,8 @@ jobs:
     - name: Install dependencies
       run: |
         python -m pip install --upgrade pip
-        pip install "sphinx${{ matrix.sphinx }}"
         pip install -e .[linkify,testing]
+        pip install --upgrade-strategy "only-if-needed" "sphinx${{ matrix.sphinx }}"
     - name: Run pytest
       run: pytest
 

diff --git a/codecov.yml b/codecov.yml
@@ -6,5 +6,5 @@ coverage:
         threshold: 0.5%
     patch:
       default:
-        target: 79%
+        target: 75%
         threshold: 0.5%
diff --git a/docs/using/intro.md b/docs/using/intro.md
@@ -299,14 +299,12 @@ Math specific, when `"dollarmath"` activated, see the [Math syntax](syntax/math)
   - `True`
   - If False then inline math will only be parsed if there are no initial/final digits,
     e.g. `$a$` but not `1$a$` or `$a$2` (this is useful for using `$` as currency)
-* - `myst_amsmath_enable`
-  - `False`
-  - Enable direct parsing of [amsmath LaTeX environments](https://ctan.org/pkg/amsmath)
 * - `myst_update_mathjax`
   - `True`
-  - If using [sphinx.ext.mathjax](https://www.sphinx-doc.org/en/master/usage/extensions/math.html#module-sphinx.ext.mathjax) (the default) then `mathjax_config` will be updated,
-  to ignore `$` delimiters and LaTeX environments, which should instead be handled by
-  `myst_dmath_enable` and `myst_amsmath_enable` respectively.
+  - If using [sphinx.ext.mathjax](https://www.sphinx-doc.org/en/master/usage/extensions/math.html#module-sphinx.ext.mathjax) (the default) then `mathjax_config` will be updated to only process specific HTML classes.
+* - `myst_mathjax_classes`
+  - `"tex2jax_process|mathjax_process|math"`
+  - A regex for the HTML classes that MathJax will process
 `````
 
 ## Disable markdown syntax for the parser

diff --git a/docs/using/syntax.md b/docs/using/syntax.md
@@ -810,22 +810,24 @@ See [the extended syntax option](syntax/amsmath).
 (syntax/mathjax)=
 ### Mathjax and math parsing
 
-When building HTML using the [sphinx.ext.mathjax](https://www.sphinx-doc.org/en/master/usage/extensions/math.html#module-sphinx.ext.mathjax) extension (enabled by default), its default configuration is to also search for `$` delimiters and LaTeX environments (see [the tex2jax preprocessor](https://docs.mathjax.org/en/v2.7-latest/options/preprocessors/tex2jax.html#configure-tex2jax)).
+When building HTML using the [sphinx.ext.mathjax](https://www.sphinx-doc.org/en/master/usage/extensions/math.html#module-sphinx.ext.mathjax) extension (enabled by default),
+Myst-Parser injects the `tex2jax_ignore` (MathJax v2) and  `mathjax_ignore` (MathJax v3) classes in to the top-level section of each MyST document, and adds the following default MathJax configuration:
 
-Since such parsing is already covered by the plugins above, MyST-Parser disables this behaviour by overriding the `mathjax_config['tex2jax']` option with:
+MathJax version 2 (see [the tex2jax preprocessor](https://docs.mathjax.org/en/v2.7-latest/options/preprocessors/tex2jax.html#configure-tex2jax):
 
-```python
-mathjax_config["tex2jax"] = {
-  "inlineMath": [["\\(", "\\)"]],
-  "displayMath": [["\\[", "\\]"]],
-  "processRefs": False,
-  "processEnvironments": False,
-}
+```javascript
+MathJax.Hub.Config({"tex2jax": {"processClass": "tex2jax_process|mathjax_process|math"}})
+```
+
+MathJax version 3 (see [the document options](https://docs.mathjax.org/en/latest/options/document.html?highlight=ignoreHtmlClass#the-configuration-block)):
+
+```javascript
+window.MathJax = {"options": {"processHtmlClass": "tex2jax_process|mathjax_process|math"}}
 ```
 
-Since these delimiters are how `sphinx.ext.mathjax` wraps the math content in the built HTML documents.
+This ensurea that MathJax processes only math, identified by the `dollarmath` and `amsmath` extensions, or specified in `math` directives.
 
-To inhibit this override, set `myst_update_mathjax=False`.
+To change this behaviour, set a custom regex, for identifying HTML classes to process, like `myst_mathjax_classes="math|myclass"`, or set `update_mathjax=False` to inhibit this override and process all HTML elements.
 
 (syntax/frontmatter)=
 

diff --git a/myst_parser/docutils_renderer.py b/myst_parser/docutils_renderer.py
@@ -104,6 +104,14 @@ def setup_render(
         )
         self._level_to_elem: Dict[int, nodes.Element] = {0: self.document}
 
+    @property
+    def sphinx_env(self) -> Optional[Any]:
+        """Return the sphinx env, if using Sphinx."""
+        try:
+            return self.document.settings.env
+        except AttributeError:
+            return None
+
     def create_warning(
         self,
         message: str,
@@ -232,12 +240,8 @@ def add_document_wordcount(self) -> None:
             return
 
         # save the wordcount to the sphinx BuildEnvironment metadata
-        try:
-            sphinx_env = self.document.settings.env
-        except AttributeError:
-            pass  # if not sphinx renderer
-        else:
-            meta = sphinx_env.metadata.setdefault(sphinx_env.docname, {})
+        if self.sphinx_env is not None:
+            meta = self.sphinx_env.metadata.setdefault(self.sphinx_env.docname, {})
             meta["wordcount"] = wordcount_metadata
 
         # now add the wordcount as substitution definitions,
@@ -455,13 +459,11 @@ def render_fence(self, token: SyntaxTreeNode) -> None:
             return self.render_directive(token)
 
         if not language:
-            try:
-                sphinx_env = self.document.settings.env
-                language = sphinx_env.temp_data.get(
-                    "highlight_language", sphinx_env.config.highlight_language
+            if self.sphinx_env is not None:
+                language = self.sphinx_env.temp_data.get(
+                    "highlight_language", self.sphinx_env.config.highlight_language
                 )
-            except AttributeError:
-                pass
+
         if not language:
             language = self.config.get("highlight_language", "")
         node = nodes.literal_block(text, text, language=language)
@@ -488,6 +490,14 @@ def render_heading(self, token: SyntaxTreeNode) -> None:
         self.add_line_and_source_path(title_node, token)
 
         new_section = nodes.section()
+        if level == 1 and (
+            self.sphinx_env is None
+            or (
+                "myst_update_mathjax" in self.sphinx_env.config
+                and self.sphinx_env.config.myst_update_mathjax
+            )
+        ):
+            new_section["classes"].extend(["tex2jax_ignore", "mathjax_ignore"])
         self.add_line_and_source_path(new_section, token)
         new_section.append(title_node)
 
@@ -1060,10 +1070,8 @@ def render_substitution(self, token: SyntaxTreeNode, inline: bool) -> None:
             **self.config.get("myst_substitutions", {}),
             **getattr(self.document, "fm_substitutions", {}),
         }
-        try:
-            variable_context["env"] = self.document.settings.env
-        except AttributeError:
-            pass  # if not sphinx renderer
+        if self.sphinx_env is not None:
+            variable_context["env"] = self.sphinx_env
 
         # fail on undefined variables
         env = jinja2.Environment(undefined=jinja2.StrictUndefined)

diff --git a/myst_parser/main.py b/myst_parser/main.py
@@ -38,14 +38,19 @@ class MdParserConfig:
         default="sphinx", validator=in_(["sphinx", "html", "docutils"])
     )
     commonmark_only: bool = attr.ib(default=False, validator=instance_of(bool))
+    enable_extensions: Iterable[str] = attr.ib(factory=lambda: ["dollarmath"])
+
     dmath_allow_labels: bool = attr.ib(default=True, validator=instance_of(bool))
     dmath_allow_space: bool = attr.ib(default=True, validator=instance_of(bool))
     dmath_allow_digits: bool = attr.ib(default=True, validator=instance_of(bool))
     dmath_double_inline: bool = attr.ib(default=False, validator=instance_of(bool))
 
     update_mathjax: bool = attr.ib(default=True, validator=instance_of(bool))
 
-    enable_extensions: Iterable[str] = attr.ib(factory=lambda: ["dollarmath"])
+    mathjax_classes: str = attr.ib(
+        default="tex2jax_process|mathjax_process|math",
+        validator=instance_of(str),
+    )
 
     @enable_extensions.validator
     def check_extensions(self, attribute, value):
@@ -118,6 +123,10 @@ def check_sub_delimiters(self, attribute, value):
                     f"myst_sub_delimiters does not contain strings of length 1: {value}"
                 )
 
+    @classmethod
+    def get_fields(cls) -> Tuple[attr.Attribute, ...]:
+        return attr.fields(cls)
+
     def as_dict(self, dict_factory=dict) -> dict:
         return attr.asdict(self, dict_factory=dict_factory)
 

diff --git a/myst_parser/mathjax.py b/myst_parser/mathjax.py
@@ -20,8 +20,12 @@
 
 
 def override_mathjax(app: Sphinx):
-    """Override aspects of the mathjax extension, but only if necessary."""
+    """Override aspects of the mathjax extension.
 
+    MyST-Parser parses dollar and latex math, via markdown-it plugins.
+    Therefore, we tell Mathjax to only render these HTML elements.
+    This is accompanied by setting the `ignoreClass` on the top-level section of each MyST document.
+    """
     if (
         "amsmath" in app.config["myst_enable_extensions"]
         and "mathjax" in app.registry.html_block_math_renderers
@@ -30,32 +34,34 @@ def override_mathjax(app: Sphinx):
             html_visit_displaymath,  # type: ignore[assignment]
             None,
         )
-    # https://docs.mathjax.org/en/v2.7-latest/options/preprocessors/tex2jax.html#configure-tex2jax
-    if (
-        app.config.mathjax_config is None
-        and app.env.myst_config.update_mathjax  # type: ignore[attr-defined]
-    ):
-        app.config.mathjax_config = {  # type: ignore[attr-defined]
-            "tex2jax": {
-                "inlineMath": [["\\(", "\\)"]],
-                "displayMath": [["\\[", "\\]"]],
-                "processRefs": False,
-                "processEnvironments": False,
-            }
-        }
-    elif app.env.myst_config.update_mathjax:  # type: ignore[attr-defined]
-        if "tex2jax" in app.config.mathjax_config:
+
+    if not app.env.myst_config.update_mathjax:  # type: ignore[attr-defined]
+        return
+
+    mjax_classes = app.env.myst_config.mathjax_classes  # type: ignore[attr-defined]
+
+    if "mathjax3_config" in app.config:
+        # sphinx 4 + mathjax 3
+        app.config.mathjax3_config = app.config.mathjax3_config or {}  # type: ignore[attr-defined]
+        app.config.mathjax3_config.setdefault("options", {})
+        if "processHtmlClass" in app.config.mathjax3_config["options"]:
+            logger.warning(
+                "`mathjax3_config['options']['processHtmlClass']` is being overridden "
+                f"by myst-parser to {mjax_classes}. "
+                "Set `myst_mathjax_classes = None` if this is undesirable."
+            )
+        app.config.mathjax3_config["options"]["processHtmlClass"] = mjax_classes
+    elif "mathjax_config" in app.config:
+        # sphinx 3 + mathjax 2
+        app.config.mathjax_config = app.config.mathjax_config or {}  # type: ignore[attr-defined]
+        app.config.mathjax_config.setdefault("tex2jax", {})
+        if "processClass" in app.config.mathjax_config["tex2jax"]:
             logger.warning(
-                "`mathjax_config['tex2jax']` is set, but `myst_update_mathjax = True`, "
-                "and so this will be overridden. "
-                "Set `myst_update_mathjax = False` if you wish to use your own config"
+                "`mathjax_config['tex2jax']['processClass']` is being overridden by "
+                f"myst-parser to {mjax_classes}. "
+                "Set `myst_mathjax_classes = None` if this is undesirable."
             )
-        app.config.mathjax_config["tex2jax"] = {
-            "inlineMath": [["\\(", "\\)"]],
-            "displayMath": [["\\[", "\\]"]],
-            "processRefs": False,
-            "processEnvironments": False,
-        }
+        app.config.mathjax_config["tex2jax"]["processClass"] = mjax_classes
 
 
 def html_visit_displaymath(self: HTMLTranslator, node: nodes.math_block) -> None:

diff --git a/setup.cfg b/setup.cfg
@@ -42,7 +42,7 @@ install_requires =
     markdown-it-py>=1.0.0,<2.0.0
     mdit-py-plugins~=0.2.8
     pyyaml
-    sphinx>=2.1,<4
+    sphinx>=3,<5
 python_requires = >=3.6
 include_package_data = True
 zip_safe = True

diff --git a/tests/test_renderers/fixtures/sphinx_directives.md b/tests/test_renderers/fixtures/sphinx_directives.md
@@ -17,16 +17,16 @@ default-domain (`sphinx.directives.DefaultDomain`):
 .
 
 --------------------------------
-SPHINX3 object (`sphinx.directives.ObjectDescription`):
+SPHINX4 object (`sphinx.directives.ObjectDescription`):
 .
 ```{object} something
 ```
 .
 <document source="notset">
     <index entries="">
-    <desc desctype="object" domain="" noindex="False" objtype="object">
-        <desc_signature>
-            <desc_name xml:space="preserve">
+    <desc classes="object" desctype="object" domain="" noindex="False" objtype="object">
+        <desc_signature classes="sig sig-object">
+            <desc_name classes="sig-name descname" xml:space="preserve">
                 something
         <desc_content>
 .
@@ -178,7 +178,7 @@ acks (`sphinx.directives.other.Acks`):
 .
 
 --------------------------------
-SPHINX3.5 hlist (`sphinx.directives.other.HList`):
+SPHINX4 hlist (`sphinx.directives.other.HList`):
 .
 ```{hlist}
 
@@ -430,47 +430,47 @@ SPHINX3 productionlist (`sphinx.domains.std.ProductionList`):
 .
 
 --------------------------------
-SPHINX3 cmdoption (`sphinx.domains.std.Cmdoption`):
+SPHINX4 cmdoption (`sphinx.domains.std.Cmdoption`):
 .
 ```{cmdoption} a
 ```
 .
 <document source="notset">
     <index entries="('pair',\ 'command\ line\ option;\ a',\ 'cmdoption-arg-a',\ '',\ None)">
-    <desc classes="std" desctype="cmdoption" domain="std" noindex="False" objtype="cmdoption">
-        <desc_signature allnames="a" ids="cmdoption-arg-a">
-            <desc_name xml:space="preserve">
+    <desc classes="std cmdoption" desctype="cmdoption" domain="std" noindex="False" objtype="cmdoption">
+        <desc_signature allnames="a" classes="sig sig-object" ids="cmdoption-arg-a">
+            <desc_name classes="sig-name descname" xml:space="preserve">
                 a
-            <desc_addname xml:space="preserve">
+            <desc_addname classes="sig-prename descclassname" xml:space="preserve">
         <desc_content>
 .
 
 --------------------------------
-SPHINX3 rst:directive (`sphinx.domains.rst.ReSTDirective`):
+SPHINX4 rst:directive (`sphinx.domains.rst.ReSTDirective`):
 .
 ```{rst:directive} a
 ```
 .
 <document source="notset">
     <index entries="('single',\ 'a\ (directive)',\ 'directive-a',\ '',\ None)">
-    <desc classes="rst" desctype="directive" domain="rst" noindex="False" objtype="directive">
-        <desc_signature ids="directive-a">
-            <desc_name xml:space="preserve">
+    <desc classes="rst directive" desctype="directive" domain="rst" noindex="False" objtype="directive">
+        <desc_signature classes="sig sig-object" ids="directive-a">
+            <desc_name classes="sig-name descname" xml:space="preserve">
                 .. a::
         <desc_content>
 .
 
 --------------------------------
-SPHINX3 rst:directive:option (`sphinx.domains.rst.ReSTDirectiveOption`):
+SPHINX4 rst:directive:option (`sphinx.domains.rst.ReSTDirectiveOption`):
 .
 ```{rst:directive:option} a
 ```
 .
 <document source="notset">
     <index entries="('single',\ ':a:\ (directive\ option)',\ 'directive-option-a',\ '',\ 'A')">
-    <desc classes="rst" desctype="directive:option" domain="rst" noindex="False" objtype="directive:option">
-        <desc_signature ids="directive-option-a directive:option--a">
-            <desc_name xml:space="preserve">
+    <desc classes="rst directive:option" desctype="directive:option" domain="rst" noindex="False" objtype="directive:option">
+        <desc_signature classes="sig sig-object" ids="directive-option-a directive:option--a">
+            <desc_name classes="sig-name descname" xml:space="preserve">
                 :a:
         <desc_content>
 .