Block comments highlighting issue

[yegorpetrov]

Well, not just block comments, but it's reproducible with antlr/jetbrains-plugin-sample: http://recordit.co/ODFaSxunOl

Same issue in my project using this adapter: http://recordit.co/dsL5dlHS9t

[parrt]

thanks

[yegorpetrov]

I've played with this a little. It looks like LexerEditorHighlighter doesn't expect lexers to recover and split incomplete input into more than two (three?) tokens, which ANTLR does when fed with /* aa * at the first pass. This can be dealt with by editing the grammar to accept unclosed comments or strings (e.g. ANTLRv4Lexer.g4). But it doesn't seem to be very easy to do for other complex lexing rules like the one in my second recording (http://pastebin.com/GEXk1HJr).

[parrt]

wow! Thanks very much for the help! I to get back to the plug-in related stuff soon.

[Kisioj]

@parrt How's the progress on this?

[parrt]

Sorry. Pinging @bjansen to see if he has time to look.

[bjansen]

There's an interesting comment in the thread that @Kisioj opened on JetBrains' forums. From what I understand, because the lexer used by IntelliJ is incremental, there might be cases where the whole comment will not be re-lexed. For example, when a space is inserted in the closing */, then removed, the lexer will only re-lex from the beginning of the token that was located at that offset. I.e. it will only lex starting from * or / I guess.

Because IDEs will very likely have to parse incorrect or partial source files, the correct (and recommended) approach to this problem seems to make lexers recognize unclosed comments, strings etc.