Help in understanding the creation of custom lexers

[Andre601]

For my personal project would I like to create a custom lexer that highlights code like this (Used dev-tools to create this):

Effectively, I want to do the following:

• Color / if it is the first character in the line (So newline resets this)
• Color any text that isn't the very first (subcommand in my example)
• Color any text in-between <>, [] and {} as long as they aren't |
• Color <>, [], {} and | (Last one only if inside {})
• Leave everything else like normal text.

Since the html lexer already does change <tag> did I take a look at it first, but I have serious problems in understanding the RegexLexer here...

I'm completely new to Pygments and even python itself, so I don't quite get how lexers work here. So if someone could give me an easy to understand explanation and even some examples would I apreciate it a lot.

[jeanas]

Well, have you already seen the documentation on creating lexers? https://pygments.org/docs/lexerdevelopment/

[Andre601]

Yes, and I still don't quite get it...

Maybe something similar to this would work (assume this being in a root entry in the tokens list):

(r"^(/)([^\S]+)", bygroups(Name.Tag, Text)),
(r"\w", Punctuation),
(r"([\[\{<])([^\]\}>\S]*)([\]\}\>])", bygroups(Punctuation, Name.Tag, Punctuation))

I've written this on mobile, so not sure if it would be valid regex...
Also, not sure how I could do the | inside {} here...

[Andre601]

Looking at bit further into this maybe this could work?

"""A Pygments lexer for the command examples"""
import re

from pygments.lexer import RegexLexer, bygroups
from pygments.token import *

__all__ = ("CommandLexer",)

class CommandLexer(RegexLexer):
    """Simple lexer to highlight arguments of commands."""
    
    name = 'Command'
    aliases = ['command']
    filenames = ['*.command']
    
    flags = re.IGNORECASE
    tokens = {
        'root': [
            (r'^(/)([^\S]+)', bygroups(Name.Tag, Text)),
            (r'[\[\{<]', Punctuation, 'option'),
            (r'\w', Punctuation)
        ],
        'option': [
            (r'\|', Punctuation),
            (r'[^\[\{<>\}\|]+', Name.Tag),
            (r'[\[\{<]', Punctuation, '#push'),
            (r'[\]\}>]', Punctuation, '#pop')
        ]
    }

Would really apreciate some help. Regex is not my strong suit here...

[jeanas]

That doesn't look bad. You probably want to choose something else for highlighting \w (Punctuation is semantically wrong) and you don't need the line flags = re.IGNORECASE. Of course, if the question is "does it work?" then the main way to find out the answer is to test it and check that it behaves like you expect (which I'm not sure I fully understand from your OP).

[Andre601]

You probably want to choose something else for highlighting \w (Punctuation is semantically wrong)

I feel like it's the right choice as by the example I've shown do I want any subsequent text after /something to be darker...

[Marrin]

Than choose or write a colour theme where the semantically correct marked token type is coloured the way you like. Those token types are not meant to say how it should look but to say what that part of the source is or means and characters are not punctuation.

[Andre601]

I managed to get it to work.... kinda.
A small issue I currently face right now is, that the closing bracket isn't highlighted like I want it to (As grey).

It's most likely a regex issue on my end, so I would appreciate if someone could take a look here and give me a clue on what to change.
My full command_lexer.py file:

"""A Pygments lexer for the command examples"""
import re

from pygments.lexer import RegexLexer, bygroups
from pygments.token import *

__all__ = ("CommandLexer",)

class CommandLexer(RegexLexer):
    """Simple lexer to highlight arguments of commands."""
    
    name = 'Command'
    aliases = ['command']
    filenames = ['*.command']
    
    tokens = {
        'root': [
            (r'^(/)([^\S]+)', bygroups(Name.Tag, Text)),
            (r'[\[\{<]', Punctuation, 'option'),
            (r'\w', Text)
        ],
        'option': [
            (r'\|', Punctuation),
            (r'[^\[\{<>\}\|]+', Name.Tag),
            (r'[\[\{<]', Punctuation, '#push'),
            (r'[\]\}>]', Punctuation, '#pop')
        ]
    }

And while I'm at it, I am wondering what the recommended way is to register this lexer for Pygments to use.
I found this StackOverflow answer, but using it gives me deprecation warnings which I would like to avoid.

I'm looking for a setup that I could use in GitHub Actions.

[Andre601]

Forgot to add: This is how it looks right now (Notice the closing brackets being blue)

[birkenfeld]

It seems that your regex in "option" for Name.Tag is missing the closing square bracket in the excluded characters.

[Andre601]

I think I found the error. I forgot to include a \] in the second option entry for Name.Tag It now renders as it should for me.

The other question about registering it is still open tho, so I would apreciate help there.

Edit: GitHub didn't display the reply from birkenfeld here and the mail was really slow, so this comment was made assuming nobody has replied so far.

[Andre601]

I'm still looking for a proper setup to register my lexer in Pygments to be used in other projects... My current setup uses deprecated stuff, which I would like to avoid.
Not sure what the best aproach would be here, especially for a setup using GitHub Actions, so if someone could help me out here, I would really apreciate it.

The current setup I use is to run python setup.py develop in GitHub Actions...
https://github.com/Andre601/DecentHolograms-Wiki/blob/feature/move-to-mkdocs/.github/workflows/publish_site.yml#L44

Would the equivalent for this be similar to what the Pygments docs mention (python -m pygments -x -l commandlexer/command_lexer.py:CommandLexer <file>?
Tho, that one would have issues for me as I'm using Pygments through MkDocs with a markdown extension that includes it, and how I understand that command - especially with the <input_fil> arg - tells me, that I would need to provide a file to parse content in, which I don't want here...

Any suggestions?

[jeanas]

What you are looking for is entirely unrelated to python -m pygments -x commandlexer/command_lexer.py:CommandLexer a.k.a. pygmentize -x -l commandlexer/command_lexer.py:CommandLexer.

There are basically two ways to use Pygments: using the pygmentize command line interface, or using the Python API. The above is a convenient way to use a custom lexer with the command line interface, but mkdocs uses the API.

The way to make plugins discoverable for the API is to install them as Python packages. That's what python setup.py develop does, except it's long been deprecated. The replacement is pip install --editable .. In fact, for you CI, you don't need --editable, so just do pip install ..

It is also deprecated not to have a pyproject.toml file, which should at a minimum have a build-system table declaring a build backend. It is not deprecated to use setup.py for project metadata like the name, version, ..., when using setuptools as the build backend, but it is recommended to declare that in the project table of the pyproject.toml instead.

In conclusion: remove your setup.py, use the pyproject.toml template provided, and replace python setup.py develop with pip install ..

[Andre601]

Alright, thanks. Managed to get it working (Especially after looking at the boilerplate plugin repository).

[Andre601]

To give a complete answer to this for everyone, I managed to create the following lexer:

"""A Pygments lexer for the command examples"""
import re

from pygments.lexer import RegexLexer, bygroups
from pygments.token import *

__all__ = ("CommandLexer",)

class CommandLexer(RegexLexer):
    """Simple lexer to highlight arguments of commands."""
    
    name = 'Command'
    aliases = ['command']
    filenames = ['*.command']
    
    tokens = {
        'root': [
            (r'^(\/)([^\S]+)', bygroups(Name.Tag, Text)),
            (r'[\[\{<]', Punctuation, 'option'),
            (r'\w', Text)
        ],
        'option': [
            (r'\|', Punctuation),
            (r'[^\[\{<>\}\|\]]+', Name.Tag),
            (r'[\[\{<]', Punctuation, '#push'),
            (r'[\]\}>]', Punctuation, '#pop')
        ]
    }

Also made an empty __init__.py in the same folder as the lexer file.

And I setup this pyproject.toml:

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "commandlexer"
description = "A custom RegexLexer for DecentHolograms commands highlighting"
version = "1.0.0"
authors = [
    { name = "Andre601" }
]
dependencies = ["pygments"]

[project.entry-points."pygments.lexers"]
commandlexer = "commandlexer.command_lexer:CommandLexer"

Finally did I use pip install . in the folder with the pyproject.toml to install and register the lexer.

Thanks everyone who helped me getting this to work!