What are Antlr4 channels used for?

[matthew-dean]

The documentation doesn't really say. It shows channels marked with HIDDEN but not what the utility of that would be.

[ericvergnaud]

May I suggest that you read the book ?

[matthew-dean]

Is this the answer for all of Antlr documentation? Buy a book? 🤔

[kaby76]

During lexical analysis, input is broken into tokens and placed on the token stream, which is just a list of tokens. A channel is an attribute of a token that can be used to hide the token from parsing. Without a channel, the parser grammar would need to be changed to explicitly handle the token wherever it could occur. E.g. adjusting the grammar for everywhere whitespace could occur would make a grammar extremely messy. When you have a lexer rule like WS: [ \n\r]+ -> channel(HIDDEN); the token channel is "hidden", which is implemented as just an integer. A rule WS: [ \n\r]+ -> skip; means to not even create a token for the string, and not add a token for that string on the token stream.

[matthew-dean]

Thanks for answering. In the case of the two WS examples, doesn't skip also hide the token (or lack of a token) from parsing? I don't get the distinction?

[kaby76]

"skip" means there is no token created for the string. Create a simple grammar with both "skip" and "channel(HIDDEN)". Then print out the token list. For example, sometimes one wants to collect all comments during tokenization. If the rule is "skip", you won't be able to recover comments from the token list because it's not on the list. But, skip shortens the token list.

[matthew-dean]

@kaby76 Ah okay, so you could do some kind of post-processing on hidden tokens, but skipped tokens would be lost forever.

[kaby76]

Yes. You may want to do something with these tokens later on, e.g., reconstruct the text after refactoring the token stream. The skipped tokens aren't really "lost", because you almost always have the original text, but it would be harder to reconstruct the text with the skipped text.