Why this happens?

[vega0]

https://www.youtube.com/watch?v=rfr-CSKrues&feature=youtu.be

In single file it works fine except single character numbers, but after combining into one file it enexpective behave into errors that i can't understand. Why its happens.

Repo : https://gitflic.ru/project/nightmare0/rossijskij-yap

[Marti2203]

I suggest debugging by seeing the emitted tokens to get the better idea as the issue is not on the parser level but the lexer level - the lexer tries to give the longest possible string to a rule, no matter the order so if you have xstringa it will return an identifier 'xstringa' and not 'x' 'string' 'a' tokens. The other issue is that you define the digits as speical tokens which will cause some problems. I suggest removing the W_НОМЕР_НОЛЬ and inlining the values.

[kaby76]

The Eclipse IDE does not show the token type, which is again why you need to stop using it.

Please read this carefully: An Antlr4 lexer works independently from the parser. The lexer matches the LONGEST string first regardless of the order of the lexer rules. But, if two or more lexer rules match a string of the same length, the first one listed is selected.

"E+3456" is matched by FRAGMENTPUTI. aЕпсилонЧастьЧисла never matches because the parser does not guide how to lex. The lexer reads all the input and tokenizes the input before the parser does anything. As you type in the input string in Eclipse, the first thing that is done is to tokenize the entire string from the first character to the last. The parser is not even involved. ZNAK_EPSILON cannot possibly ever match. The lexer will always choose the longest string. "E+3456" (6 characters long) is longer than "E" (one character long).

[vega0]

the problem means separately recogninition except full recognition ?

[vega0]

how to separate import?

[vega0]

best rule of import.

[kaby76]

how to test tokens?

Stop using Eclipse, and use Intellij.

• Open YapComplete.g4 using Intellij https://www.jetbrains.com/idea/.
• Add in Antlr4 Plugin.
  • Settings.
  • Plugins.
  • Search marketplace for "antlr4".
  • Install and restart.
• Configure Antlr4.
• YapComplete.g4 is in UTF8. Set UTF8.
• Generate ANTLR Recognizer.
• Verify generate.
• Right-click on parser rule, test rule.
• Type input string, see output with token type and token value.

[vega0]

https://gitflic.ru/project/nightmare0/rossijskij-yap did all that you told me and got wrong parser behavior. Separately it works fine but in assembled file its throws recognition error. No errors when building antlr4 data.

image

[vega0]

uhm, explain https://i.imgur.com/vndkQ0v.png this token recognition mistake that
makes no sense

 // ФАЙЛ ОБЪХЕКТА StrokaFailovyPut
fragmentputi: FRAGMENTPUTI rashireniyeFaila?;
rashireniyeFaila: TOCHKA fragmentputi;
failovyPut: '@' '/'? fragmentputi ('/' fragmentputi)*;

there is no zero defined chars that can be triggered by token...

[kaby76]

Yes, I can reproduce that.

That is exactly how Antlr is supposed to work. An Antlr4 lexer works independently from the parser. The lexer matches the LONGEST string first regardless of the order of the lexer rules. But, if two or more lexer rules match a string of the same length, the first one listed is selected.

In YapComplete.g4, you have these two rules:

A_ШЕСТНАДЦАТИРИЧНЫЕ_СИМВОЛЫ: (A_ДЕСЯТЕРИЧНОЕ_ЧИСЛО | ZNAKI_A_F)+;
...
FRAGMENTPUTI: ('\\ ' | ~(' ' | '/' | '.'))+;

"abc" matches both A_ШЕСТНАДЦАТИРИЧНЫЕ_СИМВОЛЫ and FRAGMENTPUTI. Both matches are three characters in length. But, A_ШЕСТНАДЦАТИРИЧНЫЕ_СИМВОЛЫ (line 95) occurs before FRAGMENTPUTI (line 99) in YapComplete.g4. Therefore A_ШЕСТНАДЦАТИРИЧНЫЕ_СИМВОЛЫ "wins". It does not matter that it is trying to match parser rule fragmentputi. The parser never "guides" how the lexer to do its job. The lexer works independently from the parser. The lexer gathers all lexer rules from YapComplete.g4, and tries to match each rule from top to bottom in order.

If you don't believe me, rewrite A_ШЕСТНАДЦАТИРИЧНЫЕ_СИМВОЛЫ to something like A_ШЕСТНАДЦАТИРИЧНЫЕ_СИМВОЛЫ : 'aaaaaaaa';. Then, A_ШЕСТНАДЦАТИРИЧНЫЕ_СИМВОЛЫ does not compete with FRAGMENTPUTI.

[vega0]

is it possible to split lexer?