[tsx] TypescriptLexer does not correctly support tsx

[sk-]

Currently tsx is supposed to be supported via the TypescriptLexer, however there are several issues when parsing tsx files. I'd suggest removing that support for the TypescriptLexer, just like it is for the JavascriptLexer, which does not support jsx. Additionally prioritizing tsx and jsx support would be highly appreciated, so that we don't need to rely on external packages like https://github.com/richardbann/pygments-lexer-babylon. (See issues #868 and #709.)

Errors

• Text inside nodes is parsed as Token.Name or Token.Keyword
• <, >, and / are parsed as operators
• Closing tags generate errors

Below is a comparison of the same markup parsed with the html and typescript parser.

HTML

    <div>
      this is some
      <p>text</p>
    </div>

pygments tokens

 (Token.Punctuation, '<'),
 (Token.Name.Tag, 'div'),
 (Token.Punctuation, '>'),
 (Token.Text, '\n      this is some\n      '),
 (Token.Punctuation, '<'),
 (Token.Name.Tag, 'p'),
 (Token.Punctuation, '>'),
 (Token.Text, 'text'),
 (Token.Punctuation, '<'),
 (Token.Punctuation, '/'),
 (Token.Name.Tag, 'p'),
 (Token.Punctuation, '>'),
 (Token.Text, '\n    '),
 (Token.Punctuation, '<'),
 (Token.Punctuation, '/'),
 (Token.Name.Tag, 'div'),
 (Token.Punctuation, '>'),

TSX

function f() {
  return (
    <div>
      this is some
      <p>text</p>
    </div>
  )
}

pygments tokens

 (Token.Operator, '<'),
 (Token.Name.Other, 'div'),
 (Token.Operator, '>'),
 (Token.Text, '\n      '),
 (Token.Keyword, 'this'),
 (Token.Text, ' '),
 (Token.Name.Other, 'is'),
 (Token.Text, ' '),
 (Token.Name.Other, 'some'),
 (Token.Text, '\n      '),
 (Token.Operator, '<'),
 (Token.Name.Other, 'p'),
 (Token.Operator, '>'),
 (Token.Name.Other, 'text'),
 (Token.Operator, '<'),
 (Token.Text, ''),
 (Token.Error, '/'),
 (Token.Error, 'p'),
 (Token.Error, '>'),
 (Token.Text, '\n'),
 (Token.Text, ''),
 (Token.Text, '    '),
 (Token.Operator, '<'),
 (Token.Text, ''),
 (Token.Error, '/'),
 (Token.Error, 'd'),
 (Token.Error, 'i'),
 (Token.Error, 'v'),
 (Token.Error, '>'),

[Anteru]

.tsx is gone for now (as part of #1814 )

[mfp22]

I'm confused. Why is there a lexer for JSX listed here? https://pygments.org/docs/lexers/#lexers-for-jsx-react

I'm trying to create a TSX code block with Manim, which uses pygments. It says over there that any lexer listed https://pygments.org/docs/lexers would be supported. But it's not recognizing language="tsx"

So the Manim docs are wrong, or am I missing something?

[jeanas]

The JSX lexer exists in the latest source code but not in a release yet.

[mfp22]

What kind of timeline for a release? Right now I'm trying to get ChatGPT to write and register a lexer for me and I have no idea what I'm doing, lol

[mfp22]

Well thanks for mentioning the source code containing the JSX lexer. That really helped, because ChatGPT was giving me a lot of nonsense.

Getting this released will be nice, but it would be nice to have a TSX lexer too, not just JSX.

I copied the code from here https://github.com/pygments/pygments/blob/master/pygments/lexers/jsx.py

And imported the TypeScriptLexer instead of the JavascriptLexer, and updated names to tsx instead of jsx

And I did something probably hacky, but add this at the bottom of my python script (called custom_lexers.py)

# Adjust the module name for registration
module_name = __name__ if __name__ != "__main__" else "custom_lexers"

# Registering the lexer with pygments
lexers.LEXERS["TSXLexer"] = (module_name, "TSX", ("tsx",), ("*.tsx",), ("text/x-tsx",))

And then I just imported this at the top of my Manim script:
from custom_lexers import TSXLexer

It's working for me.

[jeanas]

And I did something probably hacky,

Very definitely hacky. Why don't you just install the latest master version of Pygments? It should be as easy as pip install https://github.com/pygments/pygments/archive/refs/heads/master.zip, or specifying that URL, https://github.com/pygments/pygments/archive/refs/heads/master.zip, as a requirement in your pyproject.toml or requirements.txt or whatever method you are using to get Pygments.

[mfp22]

I didn't know you could do that. Thanks. Basically Week 2 for me using Python.

But also I need TSX, not just JSX.

[jeanas]

Sorry, we don't have that at the moment.

To do it the proper way instead of that horrible hack, you'd write a plugin. https://pygments.org/docs/plugins/ (You might also find some useful info in this recent issue, I haven't had the time to improve the docs yet: #2550.)