IMPROVE: Support Sphinx v4

[mmcky]

This PR is to upgrade the jupyter-book stack for compatibility with sphinx>=4,<5

Tasks

• MAINT: update sphinx to include sphinx v4 sphinx-jupyterbook-latex#75
• fix the following warning for pdfhtml to move from mathjax2 to mathjax3 config (as raised by @chrisjsewell MAIN: Enable support for Sphinx>=4 MyST-NB#352 (comment))

WARNING: mathjax_config/mathjax2_config does not work for the current MathJax version, use mathjax3_config instead

Additional items identified by @chrisjsewell

• Adding tasklist to the default myst_enable_extensions will close Task lists in executable book #1133
  [Status: Completed in b0258cd]
• setting myst_dmath_double_inline=True will close Inline display math (double dollar sign) #1181
  • perhaps that could also be set to true by default here (currently set to False, to be overly cautious about being back-compatible)
    [Status: Completed in d246b70]
• the myst_url_schemes can be removed from the default config, since it is now set already
  [Status: Completed in fb180ad]
• sphinx 4 will close MathJax version 3 support #963, but at the same time user who were previously setting mathjax_config should be warned that they now need to set mathjax2_config/mathjax3_config https://www.sphinx-doc.org/en/master/usage/extensions/math.html#confval-mathjax3_config
  [Status: Completed in 273e9ea]

Docs:

• users should be warned about the removed deprecations: https://github.com/executablebooks/MyST-Parser/blob/master/CHANGELOG.md#remove-v013-deprecations-%EF%B8%8F
  [@chrisjsewell is this just a documentation issue for the release to link to these deprecations?]
  [Status: Added d6a8315, but @chrisjsewell let me know if you mean you want more than this listed in the CHANGELOG for the next release]
• update documentation for juputer-book from ✨ NEW: nb_merge_streams configuration MyST-NB#364 and tag Coalesce multiple stdout/stderr Notebook outputs into single blocks #973 for closure
  [Status: Adding some docs on this feature ee38352. This PR shouldn't be merged before https://github.com/Update getting started and reference docs MyST-NB#365]

Fixes:

fixes #1133
fixes #1438
fixes #1181

[mmcky]

• Sphinx 4.1.2 is installed but sphinx<4,>=2 is required by {'sphinx-book-theme'}. UPDATE: Sphinx v4.0.0 sphinx-book-theme#364

[mmcky]

• sphinx 4 support required in sphinx-jupyterbook-latex

Warning, treated as error:
E         [jb-latex]: myst-nb version not compatible with >=0.11,<0.13: 0.13.0

[choldgraf]

hey @mmcky - what's the status on this one? Anything we need to do to unblock? It looks like the only remaining piece is the jupyterbook-latex one. I've added that as a task list to the top comment

[mmcky]

@choldgraf working our way through the stack to update the relevant packages. Currently on sphinx-jupyterbook-latex then I think we are pretty close. I have had to rework some of the tests in sphinx-jupyterbook-latex so they didn't depend on jupyter-book which is now technically upstream.

[mmcky]

Current build warnings

WARNING: cannot override config setting 'html_add_permalinks' with unsupported type, ignoring

this must be new coming in from master. Since sphinx>=3.5 this option is no longer supported and replaced by html_permalinks with a default value of True so I will remove html_add_permalinks from tests/test_config/* and default config.py`.

[codecov]

Codecov Report

Merging #1448 (199d086) into master (a0d719f) will increase coverage by 0.07%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #1448      +/-   ##
==========================================
+ Coverage   91.09%   91.17%   +0.07%     
==========================================
  Files           7        7              
  Lines         674      680       +6     
==========================================
+ Hits          614      620       +6     
  Misses         60       60              

Flag	Coverage Δ
pytests	91.17% <100.00%> (+0.07%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
jupyter_book/config.py	93.75% <100.00%> (+0.22%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a0d719f...199d086. Read the comment docs.

[mmcky]

sphinx 4 will close MathJax version 3 support #963, but at the same time user who were previously setting mathjax_config should be warned that they now need to set mathjax2_config/mathjax3_config https://www.sphinx-doc.org/en/master/usage/extensions/math.html#confval-mathjax3_config

I have added 3f9bf18 to address this issue. It will issue a warning if a user is still using mathjax_config and sphinx>=4.0. If the user has specified v2 of mathjax via mathjax_path it will switch this warning off.

If the warning is issued then it will also migrate mathjax_config to mathjax3_config to enable projects to build.

@chrisjsewell @choldgraf what do you think of this solution? From using cdn addresses do you think:

"mathjax@2" in user_yaml_update["mathjax_path"]

is robust enough? All the mathjax_path address strings I have seen seem to contain mathjax@2 in them.

• add test for 3f9bf18 so coverage doesn't fail see IMPROVE: Support Sphinx v4  #1448 (comment)

[chrisjsewell]

another TODO for you 😉:

• executablebooks/MyST-NB@v0.13.0...v0.13.1 introduces nb_merge_streams, to close Coalesce multiple stdout/stderr Notebook outputs into single blocks #973, and so should be included in the jupyter-book config

[choldgraf]

This looks good to me - I think it's OK if the Sphinx config pattern changes in a subsequent PR, since right now that config is only exposed via the sphinx: key, which is more of a power-user feature anyway.

[mmcky]

thanks @choldgraf

@chrisjsewell I'd like LGTM, so let me know if you plan to review and I will hold off until you can do so.
Otherwise I'll merge on Friday.

[chrisjsewell]

A quick review thanks @mmcky

[mmcky]

@chrisjsewell should

myst_dmath_double_inline: true  # Allow display math ($$) within an inline context

in default_config.yml be added to the schema file? Not exactly sure how the schema works.

[chrisjsewell]

be added to the schema file?

yes, it should assert the type to be boolean

[mmcky]

be added to the schema file?

yes, it should assert the type to be boolean

done in 9a6b5e6

[mmcky]

thanks @chrisjsewell I think I have addressed your comments now 🎉

[chrisjsewell]

cheers

[mmcky]

looks like windows based actions are timing out. I will re-run this in the morning to get the green tick of approval then I'll merge. Thanks everyone for your reviews.

[choldgraf]

I think that this one is good to go once tests are happy. I added an extra label to executablebooks/MyST-NB#365 so that the cross-ref works here too. I think we can merge this one once executablebooks/MyST-NB#365 is merged so that the docs work properly :-)

[mmcky]

• merge after Update getting started and reference docs MyST-NB#365

[mmcky]

thanks @chrisjsewell and @choldgraf for reviews. Greatly appreciated.