Fix CFamilyLexer preprocessor tokenization errors (#1830)

CFamilyLexer failed to tokenize preprocessor macros when they were preceded by line break surrounded by spaces. This was the case because prerpocessor regex rule expected to start at the beginning of the line, but the space regex rule matched also the whitespace after the line break. Now the space rule has been refined not to match the line break. Because of this, the preprocessor regex rule correctly matches prerpocessor tokens even when they are preceded by white spaces, at the cost of adding some more tokens in the token stream in some cases. This change preserves the behavior of invalid preprocessor usage failing to tokenize.
pygments · Jun 20, 2021 · fea1fbc · fea1fbc
1 parent a65613e
commit fea1fbc
Show file tree

Hide file tree

Showing 19 changed files with 913 additions and 226 deletions.
diff --git a/pygments/lexers/c_cpp.py b/pygments/lexers/c_cpp.py
@@ -56,7 +56,7 @@ class CFamilyLexer(RegexLexer):
             ('^(' + _ws1 + ')(#)',
              bygroups(using(this), Comment.Preproc), 'macro'),
             (r'\n', Text),
-            (r'\s+', Text),
+            (r'[^\S\n]+', Text),
             (r'\\\n', Text),  # line continuation
             (r'//(\n|[\w\W]*?[^\\]\n)', Comment.Single),
             (r'/(\\\n)?[*][\w\W]*?[*](\\\n)?/', Comment.Multiline),

diff --git a/tests/examplefiles/arduino/Blink.ino.output b/tests/examplefiles/arduino/Blink.ino.output
diff --git a/tests/examplefiles/c/example.c.output b/tests/examplefiles/c/example.c.output
diff --git a/tests/examplefiles/charmci/Charmci.ci.output b/tests/examplefiles/charmci/Charmci.ci.output
diff --git a/tests/examplefiles/cpp/example.cpp.output b/tests/examplefiles/cpp/example.cpp.output
diff --git a/tests/examplefiles/cpp/namespace.cpp.output b/tests/examplefiles/cpp/namespace.cpp.output
diff --git a/tests/examplefiles/cuda/test.cu.output b/tests/examplefiles/cuda/test.cu.output