2.1.0 problem with extending the lexers dictionary

[jmckenna]

We extend the lexers dictionary (this has worked since around 2008) such as:

  def setup(app):
      from sphinx.highlighting import lexers
      lexers['myalias'] = MyLexer()

which since the 2.1.0 release causes these warnings and doesn't trigger the highlighting:

  WARNING: Could not lex literal_block as "myalias".  Highlighting skipped.

[tk0miya]

Thank you for reporting.
At 2.1.0, sphinx makes custom lexers sensitive to syntax errors. In detail, it automatically installs raiseonerror filter to the lexers. That is not intentional change. So I'll fix it in next release.

[jmckenna]

Thank you, I will test the next release. We can close this issue for now.

[tk0miya]

Fixed by #6502.
I'll release 2.1.2 within a few days. Please wait until then.
Thanks,

[jmckenna]

Verified the fix with version 2.1.2 Great, thanks!

[tk0miya]

@jmckenna Good to know.

BTW, since v0.6, we've provided app.add_lexer() API for this purpose. To make extensions less coupled (not to touch module global variable directly), I recommend you to use it to install custom lexers.
Thanks,

[jmckenna]

So would the syntax be something like the following:

app.add_lexer('mapfile', MapFileLexer())

where MapFileLexer() is the instance and 'mapfile' is the alias. Is this syntax correct for >2.1.0 ?

[tk0miya]

Yes, it working now.
for Sphinx-2.1.0 or newer, it takes lexer class as 2nd argument.
https://www.sphinx-doc.org/en/master/extdev/appapi.html#sphinx.application.Sphinx.add_lexer

So it would be nice if possible:

import sphinx

def setup(app):
    if sphinx.version_info >= (2, 1):
        app.add_lexer('mapfile', MapFileLexer)
    else:
        app.add_lexer('mapfile', MapFileLexer())

Since 2.1, Sphinx provides :force: option for some code directives (code-block, literalinclude and so on). The class style add_lexer() API allows you to use it on custom lexers.

[jmckenna]

With 2.1.2, if I try:

app.add_lexer('mapfile', MapFileLexer)

I get the same error:

WARNING: Could not lex literal_block as "mapfile". Highlighting skipped.

But it works if I call it by instance instead:

app.add_lexer('mapfile', MapFileLexer())

[tk0miya]

Oh, sorry. I did not describe the API in detail. A lexer registered as a class raises an error on syntax error. On the other hand, A lexer registered as an instance ignores syntax error. You can choose either way.