Escapeinside content is misplaced in pygtex file for nodejsrepl type

[isc-aray]

Minimal example:

\documentclass{report}
\usepackage{minted}
\begin{document}
\begin{minted}[escapeinside=@@]{nodejsrepl}
    > test;
    @\textit{Test}@
\end{minted}
\end{document}

This produces the following pygtex file:

\begin{Verbatim}[commandchars=\\\{\},codes={\catcode`\$=3\catcode`\^=7\catcode`\_=8\relax}]
\PYG{+w}{    }\PYG{o}{\PYGZgt{}}\PYG{+w}{ }\PYG{n+nx}{test}\PYG{p}{;}
\PYG{+w}{    }
\PYG{esc}{\textit{Test}}\end{Verbatim}

The \PYG{esc}{\textit{Test}} should end up on the previous line. In its current position, it doesn't render in the output.

[muzimuzhi]

Adding autogobble option solves the lexing issue. It seems the problem is caused by the nodejsrepl lexer in Pygments, or indented lines are not a valid nodejsrepl input.

You can then set left margin with xleftmargin=<dimension>.

\documentclass{article}
\usepackage{minted}

% for debugging
\setminted{frame=single}

\begin{document}
Before
\begin{minted}[escapeinside=@@]{nodejsrepl}
    > testa;
    @\textit{Test}@
\end{minted}

After (with \texttt{autogobble})
\begin{minted}[escapeinside=@@, autogobble]{nodejsrepl}
    > testa;
    Test
\end{minted}

After v2 (with \texttt{autogobble, xleftmargin=2em})
\begin{minted}[escapeinside=@@, autogobble, xleftmargin=2em]{nodejsrepl}
    > testa;
    @\textit{Test}@
\end{minted}
\end{document}

[isc-aray]

Hmm. For some reason, that doesn't fix my actual test, though it does fix the MWE. I'm not sure why. As a workaround, I'm just injecting the correct output into a minted "text" environment:

\begin{minted}[escapeinside=@@]{text}
@\PYG{g+gp}{\PYGZgt{}}\PYG{+w}{ }\PYG{n+nb}{document}\PYG{p}{.}\PYG{n+nx}{querySelector}\PYG{p}{(}\PYG{l+s+s2}{\PYGZdq{}tbody tr:nth\PYGZhy{}child(4)\PYGZdq{}}\PYG{p}{)}\PYG{p}{;}
% etc ...
\PYG{p}{]}@
\end{minted}

This works adequately for my purposes, since I don't expect the example to change.

[muzimuzhi]

It seems the problem is caused by the nodejsrepl lexer in Pygments, or indented lines are not a valid nodejsrepl input.

From highlighting result, indented lines are not valid nodejsrepl input hence autogobble is required. So nodejsrepl input is not tokenized correctly. Just compare the color of prompt symbol >.

Example showing autogobble IS required

\documentclass{article}
\usepackage[cachedir=_minted_cache_debug]{minted}

% for debugging
\setminted{frame=single}

\begin{document}
\begin{minted}{nodejsrepl}
    > "test";
    'test'
    > 1 + 2 > 3;
    false
\end{minted}

\begin{minted}[autogobble]{nodejsrepl}
    > "test";
    'test';
    > 1 + 2 > 3;
    false
\end{minted}

\begin{minted}[escapeinside=@@]{nodejsrepl}
    > test;
    @\textit{Test}@
\end{minted}

\begin{minted}[escapeinside=@@, autogobble]{nodejsrepl}
    > test;
    @\textit{Test}@
\end{minted}
\end{document}

[muzimuzhi]

Hmm. For some reason, that doesn't fix my actual test, though it does fix the MWE.

Can you provide a new reproducible example?

[isc-aray]

\documentclass{report}
\usepackage[outputdir=build]{minted}

\begin{document}
\begin{minted}[autogobble,escapeinside=@@]{nodejsrepl}
    > test2;
    NodeList (6) [
    @\textit{Test2}@
    ]
\end{minted} 
\end{document}

I can see forbidding indentation of the > symbol, but forbidding indentation in the result is certainly a mistake in the lexer, since Node REPL will absolutely give indented output in some circumstances, for example:

> [[1,2,3],[4,5,6],[7,8,9],[10,11,12],[13,14,15]]
[
  [ 1, 2, 3 ],
  [ 4, 5, 6 ],
  [ 7, 8, 9 ],
  [ 10, 11, 12 ],
  [ 13, 14, 15 ]
]

[muzimuzhi]

Then this is indeed a https://github.com/pygments/pygments problem.

$ cat minted-gh388-nodejsrepl.js
> test2;
NodeList (6) [
  @\textit{Test2}@
]

$ pygmentize -l 'nodejsrepl' -f latex -P escapeinside='@@' minted-gh388-nodejsrepl.js
\begin{Verbatim}[commandchars=\\\{\},codes={\catcode`\$=3\catcode`\^=7\catcode`\_=8\relax}]
\PY{g+gp}{\PYZgt{}}\PY{+w}{ }\PY{n+nx}{test2}\PY{p}{;}
\PY{n+nx}{NodeList}\PY{+w}{ }\PY{p}{(}\PY{l+m+mf}{6}\PY{p}{)}\PY{+w}{ }\PY{p}{[}
\PY{+w}{  }
\PY{p}{]}
\PY{esc}{\textit{Test2}}\end{Verbatim}

• pygmentize -l 'nodejsrepl' -f latex -P escapeinside='@@' <input file> is simplified from the one used by minted (which can be found in .log file)

  pygmentize -l 'nodejsrepl' -f latex -P commandprefix=PYG -F tokenmerge \
    -P stripnl='False' -P escapeinside='@##' -o <output file> <input file>
  

• for short input like those in example below, one can use

  echo '> a\n[ @\\textit{1}@, 2 ]' | pygmentize -l 'nodejsrepl' -f latex -P escapeinside='@@'
  

  doc: https://pygments.org/docs/cmdline/#highlighting-stdin-until-eof

I also found nodejsrepl lexer is already broken with escapeinside, even without indents.

LaTeX example and CLI stdins

\documentclass{report}
\usepackage[cachedir=_minted_cache]{minted}

\begin{document}
\begin{minted}[escapeinside=@@]{text}
# expected
> a
[ @\textit{1}@, 2 ]
\end{minted}

\begin{minted}[escapeinside=@@]{nodejsrepl}
> a
[ @\textit{1}@, 2 ]
\end{minted}

\begin{minted}[escapeinside=@@]{nodejsrepl}
> ab
[ @\textit{1}@, 2 ]
\end{minted}

\begin{minted}[escapeinside=@@]{nodejsrepl}
> abc
[ @\textit{1}@, 2 ]
\end{minted}
\end{document}

$ echo '> a\n[ @\\textit{1}@, 2 ]' | pygmentize -l 'nodejsrepl' -f latex -P escapeinside='@@'
\begin{Verbatim}[commandchars=\\\{\},codes={\catcode`\$=3\catcode`\^=7\catcode`\_=8\relax}]
\PY{g+gp}{\PYZgt{}}\PY{+w}{ }\PY{n+nx}{a}
\PY{p}{[}\PY{+w}{ }\PY{p}{,}\PY{+w}{ }\PY{l+m+mf}{2}\PY{+w}{ }\PY{esc}{\textit{1}}\PY{p}{]}
\end{Verbatim}

$ echo '> ab\n[ @\\textit{1}@, 2 ]' | pygmentize -l 'nodejsrepl' -f latex -P escapeinside='@@'
\begin{Verbatim}[commandchars=\\\{\},codes={\catcode`\$=3\catcode`\^=7\catcode`\_=8\relax}]
\PY{g+gp}{\PYZgt{}}\PY{+w}{ }\PY{n+nx}{ab}
\PY{p}{[}\PY{+w}{ }\PY{p}{,}\PY{+w}{ }\PY{l+m+mf}{2}\PY{+w}{ }\PY{p}{]}\PY{esc}{\textit{1}}
\end{Verbatim}

$ echo '> abc\n[ @\\textit{1}@, 2 ]' | pygmentize -l 'nodejsrepl' -f latex -P escapeinside='@@'
\begin{Verbatim}[commandchars=\\\{\},codes={\catcode`\$=3\catcode`\^=7\catcode`\_=8\relax}]
\PY{g+gp}{\PYZgt{}}\PY{+w}{ }\PY{n+nx}{abc}
\PY{p}{[}\PY{+w}{ }\PY{p}{,}\PY{+w}{ }\PY{l+m+mf}{2}\PY{+w}{ }\PY{p}{]}
\PY{esc}{\textit{1}}\end{Verbatim}

[isc-aray]

I have opened pygments/pygments#2666 for this issue.