Fix a catastrophic backtracking bug in JavaLexer

[kurtmckee]

This closes #1586, which triggered the bug.

1. Pygments guessed from an input string that it should use SspLexer to lex the file.
2. The SspLexer is a delegating lexer that relies on XmlLexer and JspRootLexer.
3. JspRootLexer relies on regular expressions from the JavaLexer.
4. The JavaLexer has a catastrophic backtracking bug in its string literal regular expression.

This patch addresses the problem by replacing the string literal regex with a sub-state for string literal parsing.

[Anteru]

Thanks! High time we add pytest-time, I guess :)

[kurtmckee]

Thanks! High time we add pytest-time, I guess :)

If it's okay with you and Georg, I'll work on a patch to add pytest-timeout (which will attempt to hard-stop tests that run too long) and to update existing backtrack-checking tests to use the timeout.

Let me know if that's something you'd like me to tackle. =)

[Anteru]

I have no objection, and given Georg didn't red flag it, I'd say please go ahead. It's fairly obvious to me by now that backtracking needs robust testing, and I don't see how we can do it without timeouts.

[kurtmckee]

Okay, then I'll create a ticket for tracking and discussion and submit a patch for review.

[birkenfeld]

Sure, I'd just like a graceful behavior if the timeout plugin isn't installed.

[kurtmckee]

Understood. I anticipate that I'll add it as a dependency in tox so that it's guaranteed to be available when running the tests.

[kurtmckee]

By the way, I just tested the pytest timeout plugin. It supports a thread-based and a signal-based method.

On Windows, which only supports thread-based timeouts, the plugin is unable to interrupt catastrophic backtracking like this:

import re

def test_catastrophic_backtracking():
    re.search(r'\d+\d+a', '0' * 10_000)

The thread-based timeouts can successfully interrupt operations like time.sleep(60) but cannot successfully interrupt catastrophic backtracking. For this reason I'm not pursuing using the pytest-timeout plugin.