Why mismatch? #4263

vega0 · 2023-05-10T12:38:51Z

vega0
May 10, 2023

Why tipBulevo: input 'boolean' doesn't match token value?? Periodically stucks on it when complicating .

jimidle · 2023-05-10T14:29:34Z

jimidle
May 10, 2023

I think you need to take a step back and u Der stand how the rules work. Read through sone of the tutorials. Your mistakes will make sense when you understand how it works.

…

On Wed, May 10, 2023 at 20:39 Nightmare0 ***@***.***> wrote: Why tipBulevo: input 'boolean' doesn't match token value?? Periodically stucks on it when complicating . https://pastebin.com/g6TDXyGv — Reply to this email directly, view it on GitHub <#4263>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJ7TMGHSS7UH3WXN4NCANLXFOD6TANCNFSM6AAAAAAX4V2HYI> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

2 replies

vega0 May 10, 2023
Author

ehm, what exactly i'am doing wrong?

kaby76 May 10, 2023

ehm, what exactly i'am doing wrong?

All you provide is a link to a grammar, YapComplete. We don't know the start symbol for the grammar. You should have in the YapComplete.g4 file a parser rule of the form start_symbol : .............. EOF;or tell us "The start symbol is 'probelniy'." or whatever the parser rule name is. For Bison grammars, the start symbol is declared. Antlr has no such constraint, but we need it to know why the input isn't parsing. We also do not know the input string. Is it just boolean?

jimidle · 2023-05-10T15:26:45Z

jimidle
May 10, 2023

You are trying to write what you think it should do, and not what it does. Take a step back and learn a little more. Start with the very simplest of grammars to see how the lexer tokenizes everything, then the parser runs. Only one lexer rule can win, and it isn’t influenced by the parser. Also, put no ‘literals’ in the parser grammar.

…

On Wed, May 10, 2023 at 23:21 Nightmare0 ***@***.***> wrote: ehm, what exactly i'am doing wrong? — Reply to this email directly, view it on GitHub <#4263 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJ7TMDEN2LW5JWR2XIX2HDXFOXAVANCNFSM6AAAAAAX4V2HYI> . You are receiving this because you commented.Message ID: ***@***.***>

10 replies

jimidle May 11, 2023

What are you trying to do here?

fragmentputi: FRAGMENTPUTI rashireniyeFaila?;
rashireniyeFaila: Tochka fragmentputi;

Your FRAGMENTPUTI makes no sense.

Take a step back here as I think this may be an XY question:

What is it that you are trying to do?
Is this for your company or for a private project?

vega0 May 13, 2023
Author

form of filepath. @/hello/world/path\ object/file.exe

vega0 May 13, 2023
Author

not work correctly as token rule , needto do something like

fragmentputi1: FRAGMENTPUTI;
fragmentputi: fragmentputi1 rashireniyeFaila?;

?? If i put all lexems into parser rule its work fine but i can't use tokens and i cause of token symbols duplicates.

vega0 May 13, 2023
Author

https://i.imgur.com/oz41FEN.png it's work, but not after join into one file .

vega0 May 13, 2023
Author

https://i.imgur.com/xUZw9J7.png because TOKEN MISMATCH, why .... and what.

jimidle · 2023-05-11T07:09:35Z

jimidle
May 11, 2023

Because you are not making lexer tokens out of them, so ',' in the parser is not the same ',' as it is in the lexer. I did say not use `literals` in the parser. You want: COMMA: ',' ; DOT: '.' ; In your lexer, and then use those tokens in your parser. Do not have any literals in your parser: parserRule: expression (COMMA expression)* ; And so on.

…

On Thu, May 11, 2023 at 3:06 PM Nightmare0 ***@***.***> wrote: grammar StrokaFailovyPut; import Oglavleniye; fragmentputi: FRAGMENTPUTI rashireniyeFaila?; rashireniyeFaila: Tochka fragmentputi; failovyPut: SOBAKA DELITEL_PUTI? fragmentputi (DELITEL_PUTI fragmentputi)*; SOBAKA: '@'; DELITEL_PUTI: '/'; EKRANIROVANNIY_DELITEL_PUTI: '\\ '; FRAGMENTPUTI: (EKRANIROVANNIY_DELITEL_PUTI | ~(' ' | '/' | '.'))+; why i can't use token defined in a set *~(' ' | '/' | '.')*? It seems duplicates. — Reply to this email directly, view it on GitHub <#4263 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJ7TMFWIWS7KQX6RSFVYCLXFSFXLANCNFSM6AAAAAAX4V2HYI> . You are receiving this because you commented.Message ID: ***@***.***>

2 replies

vega0 May 11, 2023
Author

How to understand it?

warning(125): YapComplete.g4:80:33: implicit definition of token OKONCHANIYA in parser
YapComplete::modificatorKonstanty:1:0: mismatched input 'konstant' expecting {'константн', 'konstant', 'constant'}
warning(125): YapComplete.g4:33:14: implicit definition of token FRAGMENTPUTI in parser
warning(125): YapComplete.g4:80:33: implicit definition of token OKONCHANIYA in parser
YapComplete::modificatorKonstanty:1:0: mismatched input 'constant' expecting {'константн', 'konstant', 'constant'}

https://i.imgur.com/ksWKHjs.png

obvious must match but its mismatch ?

vega0 May 11, 2023
Author

YapComplete::failovyPut:1:0: mismatched input '@' expecting '@'

jimidle · 2023-05-11T08:25:24Z

jimidle
May 11, 2023

I have already told you. The lexer returns a token for '@' but you have '@' in your parser. So they are two different tokens, despite the text being the same. You need to define EVERYTHING in the lexer if you want a separate lexer and parser: ```antlr myrule: rule1 AT rule2 ; AT: '@' ; ``` The implicit definition warnings are also telling you that there is no lexer rule OKONCHANIYA in the lexer, so you have referred to it in the parser, but there is no such token. Get your lexer working first, then you can build the parser. I implore you to read the tutorials. This is ANTLR 101 stuff that you need to know before trying to write a grammar of any complexity. You might find it easier not to separate the lexer and parser grammars, but it is essentially the same thing. As it stands, your grammars just won't work. Maybe attach them to this conversation. But we don't really have time to give you free individual training over GitHub I am afraid.

…

On Thu, May 11, 2023 at 3:58 PM Nightmare0 ***@***.***> wrote: YapComplete::failovyPut:1:0: mismatched input '@' expecting '@' — Reply to this email directly, view it on GitHub <#4263 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJ7TMAGONWUSQ7FNE35VITXFSL3RANCNFSM6AAAAAAX4V2HYI> . You are receiving this because you commented.Message ID: ***@***.***>

16 replies

vega0 May 13, 2023
Author

when i automatic to join all into one file this doesn't work then too.

kaby76 May 13, 2023

when i automatic to join all into one file this doesn't work then too.

What doesn't work? Please show your work.

YapComplete.g4, which still does not have a start rule, has a duplicate rule for '/'.

05/13-05:48:32 ~/l/x/z/rossijskij-yap
$ gitr
origin  https://gitflic.ru/project/nightmare0/rossijskij-yap.git (fetch)
origin  https://gitflic.ru/project/nightmare0/rossijskij-yap.git (push)
05/13-05:48:41 ~/l/x/z/rossijskij-yap
$ git pull origin
Already up to date.
05/13-05:48:51 ~/l/x/z/rossijskij-yap
$ git status .
On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean
05/13-05:49:06 ~/l/x/z/rossijskij-yap
$ trparse -l YapComplete.g4 | trxgrep " //lexerRuleBlock/lexerAltList/lexerAlt/lexerElements/lexerElement/lexerAtom/terminal/STRING_LITERAL[text() = '''/''']/ancestor::lexerRuleSpec" | trtext
CSharp 0 YapComplete.g4 success 0.0791502

DELITELPUTI: '/';


// ???? ???????? Virajeniya
DELENIYE: '/';
05/13-05:49:16 ~/l/x/z/rossijskij-yap
$ git show
commit 79d00bb4eef825083943b5a3f7e383cab8ae9fdc (HEAD -> master, origin/master, origin/HEAD)
...

YapComplete.g4 cannot work because you define two lexer rules with the same right-hand side: DELITELPUTI and DELENIYE. Antlr will allow you to do that but at least outputs a warning. But, you should treat it as an error because only the first, DELITELPUTI will ever be returned by the lexer. Any rule that derives DELENIYE will not work.

Antlr lexers are not parser aware at all. Antlr lexers operate by two rules: (1) match and return the token with the longest string--regardless of the order of the rules; (2) if two rules match the same string, the first one "wins".

You have to remove the rule DELENIYE and replace all occurrences of it in any rules, e.g., in YapComplete.g4:

virajeniye: virajeniye operaciya=(UMNOJENIYE | DELITELPUTI | STEPENN) virajeniye // умножение / деление
		  | virajeniye operaciya=(A_ПЛЮС | A_МИНУС) 	   		   virajeniye // сложение / вычитание
		  | LEVAYASKOBKA virajeniye PRAVAYASKOBKA
		  | ssilkaNaObiekt
		  | aПолныйВидЧисла;

Again, what is the start rule for which grammar? If you are referring to YapComplete.g4, add an EOF-terminated rule.

vega0 May 13, 2023
Author

Rules can be as much as possible complicated to make so complicated parsers. Why its not should be complicated? And yes, the question, buildscript.

vega0 May 13, 2023
Author

sometimes lost utf8 recognition. Cause ?
https://i.imgur.com/rsGqjRT.png

kaby76 May 13, 2023

https://i.imgur.com/rsGqjRT.png

"Cause?" The "ANTLR Console" pane says that you are using antlr-4.4-complete.jar. That jar is over 9 years old. We are now on version 4.12. Until you change that, expect problems.

jimidle · 2023-05-12T11:11:58Z

jimidle
May 12, 2023

I want to back Ken up here. Just put e Erin one file , get it working, then worry about splitting it up if there is sone overriding reason. I temper dealing with this in v3 and C and it was complicated, then nobody used it.

…

On Fri, May 12, 2023 at 19:09 Ken Domino ***@***.***> wrote: I've updated https://github.com/kaby76/antlr4-4263 to include your original set of grammars in t1 <https://github.com/kaby76/antlr4-4263/tree/0e93645e618a81341e4f219a90e76f5c769f7f0b/t1> and the modified set of grammars (from single-line imports "import a, b, c, d;" to sequential-line imports "import a; import b; import c; import d;") in t2 <https://github.com/kaby76/antlr4-4263/tree/0e93645e618a81341e4f219a90e76f5c769f7f0b/t2>. t1 <https://github.com/kaby76/antlr4-4263/tree/0e93645e618a81341e4f219a90e76f5c769f7f0b/t1> builds (via dotnet build), t2 <https://github.com/kaby76/antlr4-4263/tree/0e93645e618a81341e4f219a90e76f5c769f7f0b/t2> does not. This means that the scoping of the symbols is not flat. The description of import is in the Documentation here <https://github.com/antlr/antlr4/blob/master/doc/grammars.md#grammar-imports>. It says: Rules in the “main grammar” override rules from imported grammars to implement inheritance. There seems to be an issue with an import graph that is a partial order <https://en.wikipedia.org/wiki/Partially_ordered_set#Partial_order>. As I shown in the diagram here <#4262 (comment)>, grammar StrokaFailovyPut is imported on two paths, one directly from grammar InstrukciyaImportirovat -> StrokaFailovyPut, and the other via path InstrukciyaImportirovat -> Peremenniye -> Stroki -> StrokaFailovyPut. It's likely you could refactor the grammar so that it's a total order <https://en.wikipedia.org/wiki/Total_order>, and that will work. However, you really should not be partitioning your grammar over 16 .g4 files. It is nearly impossible to read, and for sure requires an IDE to keep track of which symbol is in which file. I have never seen a grammar like this in my 40+ years of programming. And, there is nothing in the 300+ grammars of grammars-v4 <https://github.com/antlr/grammars-v4> remotely similar. To build, you will need to download Microsoft's NET SDK version 7. Afterward, from a Bash shell prompt, "dotnet build". The Eclipse, Intellij, and lab.antlr.org IDEs all uses an "interpretive" Antlr. It is not identical to that from the Antlr tool. There are subtle differences between "interpretive" and "compiled" Antlrs. — Reply to this email directly, view it on GitHub <#4263 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJ7TMFZIR6FN4PHWUUDQO3XFYK6RANCNFSM6AAAAAAX4V2HYI> . You are receiving this because you commented.Message ID: ***@***.***>

2 replies

kaby76 May 12, 2023

That's a strong thumbs up. Definitely, combine all .g4's into one. This is ridiculous.

vega0 May 13, 2023
Author

combined and it wont work as in splitted state.

not work correctly as token rule , needto do something like

fragmentputi1: FRAGMENTPUTI;
fragmentputi: fragmentputi1 rashireniyeFaila?;
?? If i put all lexems into parser rule its work fine but i can't use tokens and i cause of token symbols duplicates.

@vega0
Comment options
[vega0](https://github.com/vega0)
[4 hours ago](https://github.com/antlr/antlr4/discussions/4263#discussioncomment-5890551)
Author
https://i.imgur.com/oz41FEN.png it's work, but not after join into one file .

@vega0
Comment options
[vega0](https://github.com/vega0)
[4 hours ago](https://github.com/antlr/antlr4/discussions/4263#discussioncomment-5890553)
Author
https://i.imgur.com/xUZw9J7.png because TOKEN MISMATCH, why .... and what.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Why mismatch? #4263

{{title}}

Replies: 5 comments 32 replies

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Why mismatch? #4263

Replies: 5 comments · 32 replies

vega0 May 10, 2023 Author

vega0 May 13, 2023 Author

vega0 May 13, 2023 Author

vega0 May 13, 2023 Author

vega0 May 13, 2023 Author

vega0 May 11, 2023 Author

vega0 May 11, 2023 Author

vega0 May 13, 2023 Author

vega0 May 13, 2023 Author

vega0 May 13, 2023 Author

vega0 May 13, 2023 Author

Replies: 5 comments 32 replies

vega0 May 10, 2023
Author

vega0 May 13, 2023
Author

vega0 May 13, 2023
Author

vega0 May 13, 2023
Author

vega0 May 13, 2023
Author

vega0 May 11, 2023
Author

vega0 May 11, 2023
Author

vega0 May 13, 2023
Author

vega0 May 13, 2023
Author

vega0 May 13, 2023
Author

vega0 May 13, 2023
Author