Centered equations not rendering properly

[rsokl]

When trying to reproduce the centered equation from your documentation (using the latest version of nbsphinx from pypi):

---

---

I find that the centered equation does not render properly:

---

---

I used the exact content of your notebook cell:

## Equations

Equations can be formatted really nicely, either inline, like $\text{e}^{i\pi} = -1$, or on a separate line, like

\begin{equation}
\int_{-\infty}^\infty f(x) \delta(x - x_0) dx = f(x_0)
\end{equation}

*Note:* Avoid leading and trailing spaces around math expressions, otherwise errors like the following will occur when Sphinx is running:

    ERROR: Unknown interpreted text role "raw-latex".

See also the [pandoc docs](http://pandoc.org/MANUAL.html#math):

> Anything between two `$` characters will be treated as TeX math. The opening `$` must have a non-space character immediately to its right, while the closing `$` must have a non-space character immediately to its left, and must not be followed immediately by a digit.

[mgeier]

What Sphinx version are you using?

This looks like a bug from versions < 1.4.2, see #44 and sphinx-doc/sphinx#2473 and sphinx-doc/sphinx#2476.

[rsokl]

I'm running Sphinx v1.6.4

Edit: Upgraded to v1.6.5 and edited the equation to make sure the the HTML for it was rebuilt. Same issue.

[mgeier]

Hmmm, that's strange.

I don't know what exactly is going wrong here, but probably having a look at the intermediary RST file gives us a clue. Can you please check out the save-rst branch and look in the generated file in the HTML output directory (I guess it'll be called markdown.rst) and have a look how the relevant math directive looks?

BTW, I've given the wrong reference #44 above. For Markdown cells, the "nowrap" thing was introduced in #43 and updated in #110.

[rsokl]

Sorry, I'm not quite following. I checked out save-rst, navigated to the docs directory, and ran python -m sphinx . _build -j4.

There are no intermediate .rst in the _build dir; I even verified that nbsphinx_save_rst = True in conf.py.

The html for the centered equation is:

<span class="math" id="MathJax-Span-12" aria-hidden="true" style="vertical-align: 0.215em;">
    <span class="noError" id="MathJax-Span-13" style="display: inline-block;">
        "\begin{align}\begin{aligned}�:nowrap:�\\\begin{equation}"
      <br>
       "\int_{-\infty}^\infty&nbsp;f(x)&nbsp;\delta(x&nbsp;-&nbsp;x_0)&nbsp;dx&nbsp;=&nbsp;f(x_0)"
      <br>
       "\end{equation}\end{aligned}\end{align}"
    </span>
</span>

[mgeier]

You should follow the instructions in CONTRIBUTING.rst to get a development installation of the nbsphinx module.

[ghost]

Ah! Of course.

[rsokl]

This is what the intermediate rst looks like:

.. math::



   �:nowrap:�



   \begin{equation}

   \int_{-\infty}^\infty f(x) \delta(x - x_0) dx = f(x_0)

   \end{equation}

Because GitHub isn't rendering the symbols to either end of "nowrap", here is what it looks like on my end:

[mgeier]

Hmmm, one strange thing is that you seem to have additional blank lines between each line ...
Is this the case in the whole intermediate RST file?

Can you please check with a hex-editor if the "newlines" are CRLF or LF or alternating or something else?
See https://en.wikipedia.org/wiki/Newline.

Which OS and which Python version are you using?

Or maybe this has something to do with pandoc? Which version are you using and where did you get it?

Finally, can you please try to insert this line right before the last return statement in the function markdown2rst() in the file src/nbsphinx.py:

    print(repr(rststring))

... and run Sphinx again (probably using the -E option to re-build the whole environment)?
How does the output for the math directive look in this case?

[rddaz2013]

mmhh..

Line 886?

 def rawlatex2math(text):
        json_data = json.loads(text, object_hook=rawlatex2math_hook)
        return json.dumps(json_data)

    rststring = pandoc(text, 'markdown', 'rst', filter_func=rawlatex2math)
    return re.sub(r'^\n( *)\x0e:nowrap:\x0f$',
                  r'\1:nowrap:',
                  rststring,
                  flags=re.MULTILINE)

so the 'Special marker characters are removed below' aren't removed? #109
(1e0a4c9) - why we have a ending $ in the re.sub...'\x0e:nowrap:\x0f\n\n is the part we want to replace. (Line 881)

In the intermediate rst the '\x0e:nowrap:\x0f\n\n still remains...

[rsokl]

Sorry my responses are delayed. I wish I could address this stuff while I was at work :/

Yes, every line in the rst has an extra line. The newlines in the intermediate .rst file is CRLF.

I'm running:

• Windows 10
• Python 3.6
• Pandoc 2.1 (obtained via conda-forge)

This is the output of the repr:

'Equations\r\n---------\r\n\r\nEquations can be formatted really nicely, either inline, like\r\n:math:`\\text{e}^{i\\pi} = -1`, or on a separate line, like\r\n\r\n.. math::\r\n\r\n   \x0e:nowrap:\x0f\r\n\r\n   \\begin{equation}\r\n   \\int_{-\\infty}^\\infty f(x) \\delta(x - x_0) dx = f(x_0)\r\n   \\end{equation}\r\n\r\n*Note:* Avoid leading and trailing spaces around math expressions,\r\notherwise errors like the following will occur when Sphinx is running:\r\n\r\n::\r\n\r\n    ERROR: Unknown interpreted text role "raw-latex".\r\n\r\nSee also the `pandoc docs <http://pandoc.org/MANUAL.html#math>`__:\r\n\r\n    Anything between two ``$`` characters will be treated as TeX math.\r\n    The opening ``$`` must have a non-space character immediately to its\r\n    right, while the closing ``$`` must have a non-space character\r\n    immediately to its left, and must not be followed immediately by a\r\n    digit.\r'

[mgeier]

Thanks for the additional information, I think we found the problem!

You are getting:

.. math::\r\n\r\n   \x0e:nowrap:\x0f\r\n

... but I am (or rather the regular expression is) expecting:

.. math::\n\n   \x0e:nowrap:\x0f\n

This is caused by Windows using CRLF, but I'm not sure if the '\r' characters are introduced by Python or by pandoc.

The question now is if we should try to remove the dreaded '\r' characters or if we should try to make the regex work with both '\n' and '\r\n'.

[rddaz2013]

the never ending story .-)

https://docs.python.org/3/glossary.html#term-universal-newlines

This is caused by Windows using CRLF, but I'm not sure if the '\r' characters are introduced by Python or by pandoc.

I think it is pandoc - back home i will test that. I have a running WIN and Linux env.

[ghost]

I suspect pandoc is the culprit. It looks like there is a flag that can be set to force line enders

It looks like everything else is rendering fine, but I have to wonder why this isn't messing with more things.

[rsokl]

I set up a conda environment in Windows Subsystem for Linux and was able to get the centered equations to run properly. It is most definitely the Windows issue.

I do suspect that /n/r characters are occurring via pandoc in the intermediate rst form.

(Unfortunately, I cannot get parallel building to work on WSL; this along with WSL's hindered disk IO makes the sphinx build slow... An eventual fix to this would be awesome :D)

Edit: googling "WSL waiting for workers" brings up a bunch of Marxist info pages haha :)

[mgeier]

I tried using universal_newlines=True in subprocess.Popen() which seems to work with Python 3 on Linux and I guess it should solve the problem for Windows.

You can try #160 to see if it works for you.

However, this doesn't work for Python 2, because it doesn't use UTF-8 by default and there is no encoding argument.

I could drop Python 2 support, but I'm not sure if I should do that already.

Is it possible to get pandoc to emit LF instead of CRLF?

[ghost]

This flag is available in pandoc. Not sure if it is exposed in the python API:

--eol=crlf|lf|native

Manually specify line endings: crlf (Windows), lf (macOS/Linux/UNIX), or native (line endings appropriate to the OS on which pandoc is being run). The default is native

[mgeier]

Thanks @meowklaski, that sounds promising! Looks like this option was added in version 2.0.

I've created #161, but I have no idea if that actually works.

Can somebody please test it with pandoc >= 2.0 on Windows (and also on other OSs)?
With pandoc < 2.0 it will still fail on Windows.

BTW, there isn't really a Python API for pandoc. It's just started as a sub-process and arbitrary command line arguments can be passed.

[rsokl]

This did the trick! Verified on Windows with pandoc 2.1

The \begin statement has to have a blank line before it for the math to be rendered correctly.

[mgeier]

@rsokl Thanks for testing!

[rsokl]

@mgeier I wanted to say: nbsphinx is a phenomenally useful tool. Thank you so much for creating and maintaining this tool.

This is what I've been able create (work in progress!), thanks to nbsphinx.

[mgeier]

Thanks @rsokl, I'm glad that you like nbsphinx!

Thanks for sharing your notebooks about Python, it's always nice to see nbsphinx in action. I also think it is a good resource, thanks for providing it!
In case you are interested, I've also created (it's actually a work in progress) a few introductory notebooks, but not using Sphinx: https://nbviewer.jupyter.org/github/mgeier/python-audio/blob/master/index.ipynb.

I was wondering why you don't seem to be using code cells in your notebooks?

BTW, I've just made a new release, now the pandoc/Windows issue should be gone.

[rsokl]

Stylistically, I like how the markdown code-blocks mesh with the surrounding text more. It has a more integrated feel to it. The code cells and their numbering make the text feel a bit too divided, visually. I do make use of them when specifically referring to IPython console and Jupyter Notebook commands

This is a funny coincidence. In the summers, I teach a course for high school kids to learn about programming and some cutting edge applications - we do a week on audio processing (which is not my specialty), and I have the kids code up their own "shazaam-ish" music recognition program. I'll definitely make use of your audio resources! I'm actually making that Learning Python website in part for these students. It might overlap with some of the basic review material that you're doing.

[mgeier]

@rsokl OK, it's of course fine to use to use Python console-like code listings. However, there are a few things I'd like to point out:

• Your use of comment lines is often inconsistent. If you have an interactive Python prompt, it's not possible to write comments starting at the beginning of the line, you'll always have the >>> prompt in front of them:

  >>> # some comment
  >>> 1
  1

  There will also be no empty lines. If you just press "Enter" you'll still get >>>.

• There is a special Pygments lexer called "pycon" which you should probably use for those Python console listings.

  ```pycon
  >>> # some comment
  >>> 1
  1
  ```
  

• If you have pages with only Python console listings, you might as well use an RST file instead of a Jupyter notebook. RST files support Python console listings without additional markup by default!

  normal reST text
  
  >>> # this is automatically recognized as Python console code
  >>> 2 + 2
  4
  
  normal text again
  

  Apart from easier editing, and more importantly, this way you can automatically test if your code is actually working. You can use the doctest builder provided by the Sphinx extension sphinx.ext.doctest, see http://www.sphinx-doc.org/en/stable/ext/doctest.html#module-sphinx.ext.doctest.
  Alternatively, you can use pytest, which also supports this: https://docs.pytest.org/en/latest/doctest.html

[rsokl]

Thanks for the useful pointers! :)

It actually had occurred to me today that the commenting style that I was using in the code blocks wouldn't accommodate easy copying and pasting for the reader. It's unfortunate that I'd need to clutter those leading comments with the >>> markup, since they are merely meant to serve as captions for the code snippets. My desire is that my code blocks provide enough annotations such that the reader could browse the code blocks alone and glean 80% of a page's contents from them.

I think what I will do is make note of my commenting style to the reader upfront, and indicate that those portions of the code are not to be copied/pasted.

Most of my text have enough formatted elements that getting to see them render within the notebook is a nice way for me to size up what the reader will be seeing in html form, which is why I steer away from pure-rst files.

The pycon-annotated code blocks don't appear to render any differently than do the python-annotated ones, once the html is rendered.

[mgeier]

I just want to emphasize that there are errors (typos) in your code examples, and it would be really simple to spot them using the Sphinx doctest extension.
IMHO avoiding errors should be a high priority in a Python tutorial.

getting to see them render within the notebook is a nice way for me to size up what the reader will be seeing in html form

For that, I highly recommend sphinx_autobuild, see http://nbsphinx.readthedocs.io/en/latest/usage.html#Watching-for-Changes-with-sphinx-autobuild.
Granted, there may be a few seconds lag when editing, but you get to see the actual final HTML output. You can even have the source text and HTML side-by-side in different windows, which isn't even possible in the Jupyter Notebook.

The pycon-annotated code blocks don't appear to render any differently than do the python-annotated ones

Oh, that's interesting.
I remember that in the past there was a difference, but it looks like the normal "python" lexer has learned how to handle the >>> prompts ...
So with a recent Pygments version this seems to make no difference anymore.

But just for the record, on Gihub comments there is still a difference ...

[mgeier]

Regarding copy-pasting:
sphinx-doc/sphinx#267
http://sbrunner.github.io/sphinx-prompt/

[rsokl]

@mgeier I will definitely make it a priority to make use of doctest or pytest in my workflow. I appreciate you emphasizing that.

It may seem silly, but working with notebooks (with a spellcheck plugin) is really much more convenient than is writing .rst files and checking the output with sphinx_build. Given the amount of content that I'm dealing with, having the convenient interface of switching between markdown and code cells ultimately nets me a huge amount of time.

What I will likely end up doing is writing a simple tool that extracts the code blocks from my notebooks, writes them to corresponding .rst files, and then runs pytest on them. I may just make use of your save-rst branch to grab the intermediate rst files.

I'll check out sphinx-prompt too!

Thank you for all this great direction. I'm hoping that I can produce a really high quality resource for people to learn Python; it'd be great if you had any other feedback

[rsokl]

I just realized that in the most recent version of ipython and jupyter notebook, that my format of code blocks do actually copy/paste nicely. E.g. the following code runs fine when pasted directly in both an ipython console and a notebook

# assign the variable `x` to the integer 1
>>> x = 1

# checking the type of `x`
>>> type(x)
int

# verifying that `x` is an integer-type object
>>> isinstance(x, int)
True

Previously, the leading comment caused a syntax error, so that's good news :)

[mgeier]

That's great!

One last question: are the source notebooks available somewhere?
I've looked at your Github repos but I couldn't find it.

[rsokl]

@mgeier sorry for such a late response!

I am hosting my repo privately - some content (graded assignments) I do not want to be available to students. I'd be happy to add you to the repo, if you'd like.

The content is now hosted at https://www.pythonlikeyoumeanit.com/

Unfortunately, I found pytest.doctest to be a massive pain, mainly due to this bug. Also it gets certain trivial things wrong. For example, it flags the following as incorrect

>>> type(1)
int

That being said, I am very glad to have combed through the code blocks; it pointed me to several mistakes.

[mgeier]

some content (graded assignments) I do not want to be available to students

Are there assignments?
I didn't see any ...
Can't you split this into a private and a public repo?

I'd be happy to add you to the repo, if you'd like.

No thanks, I'd much prefer if the whole thing were public.
Then people could play around with the notebooks themselves.
You could even provide binder links, like I did in the nbsphinx docs on the top of every page, e.g. http://nbsphinx.readthedocs.io/en/0.3.1/code-cells.html

it flags the following as incorrect

>>> type(1)
int

Because it is incorrect?

>>> type(1)
<class 'int'>

[rsokl]

Because it is incorrect?

Oh that's so weird. I never knew the IPython hooked into anything different for printing the console-output . It looks like it formats several things more cleanly:

• <class 'int'> -> int
• <built-in function print> -> <function print>

I really hope pytest fixes that nasty bug, that prevents pytest.docstest from printing all of the cases at once. It really is a show-stopper.

I'd much prefer if the whole thing were public.Then people could play around with the notebooks themselves.
You could even provide binder links

I'm definitely open to making the repo public. My online class just went live, and there have been many pressing things that I've had to deal with, so I was just playing things on the safe side.

There really isn't anything to play with. These notebooks are almost all pure-markdown. Their sole purpose is to map to easy-to-follow/read web pages. Down the road, I might add work sheets and/or demos, in which case it would be nice to supply binder links.

[mgeier]

@rsokl You were talking about the >>> prompts sometimes being annoying (w.r.t. copying code).
I've seen some kind of Sphinx extension before that provided a button to switch those on and off.
I couldn't remember where it was, but now I stumbled upon a site that has the same feature:

https://networkx.github.io/documentation/latest/tutorial.html

It seems to be using this: https://github.com/networkx/networkx/blob/master/doc/_static/copybutton.js

And now I know where I saw it originally: in the official Python docs!

[rsokl]

@mgeier Thank you so much for bringing this to my attention! I will definitely be able to make good use of this in my site. Greatly appreciated! 😁