Pygments issue with Objective-J Cappuccino

[ichbinsnicht]

We are trying to run pygount which uses pygments and we are having an issue running a small script in which we loop over GitHub repos to extract the lines of codes with pygount.

Looping through repos has worked well for all our repos but for ‘cappuccino.’ Cappuccino has its own language, Objective-J, and we suspected that may be an issue for pygount. The Pygount Repo owner who tested pygount for cappuccino confirmed our suspicion and points towards issues with pygments: "When executing pygount for cappuccino it seems that pygments' Objective-J readings are very slow even with small files and eventually pygments ends up in an infinite loop on the file cappuccino/AppKit/Platform/DOM/CPPlatformPasteboard.j. The same happens when you pass this to the "pygmentize" command line tool from pygments."

We are hoping that you can help us with this issue.

[jeanas]

Is it this file?

https://github.com/cappuccino/cappuccino/blob/master/AppKit/Platform/DOM/CPPlatform.j

I can't reproduce this. After downloading it, pygmentize CPPlatform.j takes 0.4s for me. Are you using the latest version of Pygments?

[roskakori]

The culprit is CPPlatformPasteboard.j. Here's an attached copy: CPPlatformPasteboard.j.zip

When I run

pygmentize cappuccino/AppKit/Platform/DOM/CPPlatformPasteboard.j

with pygments 2.13.0 on a MacBook Pro 2015 with macOS 12.5.1 and Python 3.9.13, it keeps running for minutes with 1 CPU maxed out. When I eventually press Control-C, the output is:

^C/*
 * CPPlatformPasteboard.j
 * AppKit
 *
 * Created by Alexander Ljungberg.
 * Copyright 2013, SlevenBits Ltd.
 *
 * This library is free software; you can redistribute it and/or
 * modify it under the terms of the GNU Lesser General Public
 * License as published by the Free Software Foundation; either
 * version 2.1 of the License, or (at your option) any later version.
 *
 * This library is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
 * Lesser General Public License for more details.
 *
 * You should have received a copy of the GNU Lesser General Public
 * License along with this library; if not, write to the Free Software
 * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
 */

@import <Foundation/CPObject.j>
@import <Foundation/CPRunLoop.j>

@import "CPCompatibility.j"
@import "CPEvent.j"
@import "CPPasteboard.j"
@import "CPPlatform.j"
@import "CPPlatformWindow+DOMKeys.j"

@global CPApp
@global Traceback (most recent call last):
  File "/Users/someone/Library/Caches/pypoetry/virtualenvs/pygount-WBU9EwCb-py3.9/bin/pygmentize", line 8, in <module>
    sys.exit(main())
  File "/Users/someone/Library/Caches/pypoetry/virtualenvs/pygount-WBU9EwCb-py3.9/lib/python3.9/site-packages/pygments/cmdline.py", line 640, in main
    return main_inner(parser, argns)
  File "/Users/someone/Library/Caches/pypoetry/virtualenvs/pygount-WBU9EwCb-py3.9/lib/python3.9/site-packages/pygments/cmdline.py", line 492, in main_inner
    highlight(code, lexer, fmter, outfile)
  File "/Users/someone/Library/Caches/pypoetry/virtualenvs/pygount-WBU9EwCb-py3.9/lib/python3.9/site-packages/pygments/__init__.py", line 82, in highlight
    return format(lex(code, lexer), formatter, outfile)
  File "/Users/someone/Library/Caches/pypoetry/virtualenvs/pygount-WBU9EwCb-py3.9/lib/python3.9/site-packages/pygments/__init__.py", line 64, in format
    formatter.format(tokens, outfile)
  File "/Users/someone/Library/Caches/pypoetry/virtualenvs/pygount-WBU9EwCb-py3.9/lib/python3.9/site-packages/pygments/formatters/terminal256.py", line 250, in format
    return Formatter.format(self, tokensource, outfile)
  File "/Users/someone/Library/Caches/pypoetry/virtualenvs/pygount-WBU9EwCb-py3.9/lib/python3.9/site-packages/pygments/formatter.py", line 94, in format
    return self.format_unencoded(tokensource, outfile)
  File "/Users/someone/Library/Caches/pypoetry/virtualenvs/pygount-WBU9EwCb-py3.9/lib/python3.9/site-packages/pygments/formatters/terminal256.py", line 256, in format_unencoded
    for ttype, value in tokensource:
  File "/Users/someone/Library/Caches/pypoetry/virtualenvs/pygount-WBU9EwCb-py3.9/lib/python3.9/site-packages/pygments/lexer.py", line 190, in streamer
    for _, t, v in self.get_tokens_unprocessed(text):
  File "/Users/someone/Library/Caches/pypoetry/virtualenvs/pygount-WBU9EwCb-py3.9/lib/python3.9/site-packages/pygments/lexer.py", line 632, in get_tokens_unprocessed
    m = rexmatch(text, pos)
KeyboardInterrupt

[izahn]

Confirmed, I can replicate that

pygmentize CPPlatformPasteboard.j

maxes out a CPU until interrupted. Anybody have an idea about what goes wrong here?

[roskakori]

For the record: I checked the hexdump of the line where it seems to get stuck (according to the stack trace above), but could not see anything unusual or kinky.

00000420  0a 0a 40 67 6c 6f 62 61  6c 20 43 50 41 70 70 0a  |..@global CPApp.|
00000430  40 67 6c 6f 62 61 6c 20  43 50 50 6c 61 74 66 6f  |@global CPPlatfo|
00000440  72 6d 57 69 6e 64 6f 77  0a 0a 2f 2f 20 46 72 6f  |rmWindow..// Fro|

[jeanas]

I have opened #2225 to address this issue. It does not fully fixes the problem, as the lexer seems not to recognize @global syntax, but it does fix the hang.