Need help to create python parser for typescript/javascript target #4317
-
|
hi i have tried to create parser for typescript target using the python3ts grammar that exist in the grammar repo i have changed the grammar file to the python2 but for javascript target this time, |
Beta Was this translation helpful? Give feedback.
Replies: 15 comments 60 replies
-
|
@ericvergnaud hi, please i need help with this, its driving me crazy |
Beta Was this translation helpful? Give feedback.
-
|
You need help from the author of that grammar, it relies on antlr4ts which is a fork we don’t support.Alternately you need to rewrite Lexer::header and Lexer::members sections in the grammarEnvoyé de mon iPhoneLe 17 juin 2023 à 10:09, Elmahdi ***@***.***> a écrit :
@ericvergnaud hi, please i need help with this, its driving me crazy
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
i used just antlr4 tool to generate it |
Beta Was this translation helpful? Give feedback.
-
|
@Elmahdi1962 This is an issue with the grammar, not Antlr per se. The grammar that you will want to use is python/python3, but it only works for CSharp, Java, and Python3. I'll make a port for TypeScript today in a PR to round out all the grammar for all 8 targets (not including Swift because I can't get it to install--never could). In fact, I created an issue on the Python grammars yesterday. before you wrote the question here. antlr/grammars-v4#3539. The state of the Python grammars in grammars-v4 is a mess. |
Beta Was this translation helpful? Give feedback.
-
|
The JavaScript port "works" but it is extremely slow. The grammar for expressions is completely wrong. Someone introduced a But, the TypeScript port cannot work. Turns out that the constructor for CommonToken is wrong. #4318. It cannot be done with 4.13.0--at all--because the constructor declares "source" as a number, yet, it's supposed to be an array (common token factory). |
Beta Was this translation helpful? Give feedback.
-
|
what can i do to generate a python parser for a javascript target but it should not have the import from syntax imports |
Beta Was this translation helpful? Give feedback.
-
|
I think you need to use Babel. Not familiar with that. Sorry. |
Beta Was this translation helpful? Give feedback.
-
|
there is only the grammaer for python2 for target js where can i find one for python3 |
Beta Was this translation helpful? Give feedback.
-
|
Thanks, it was helpful for me. |
Beta Was this translation helpful? Give feedback.
-
|
The python3 grammar works for JavaScript. But it is slow. The only other viable Python3 grammar would be python/python, but there is no JavaScript port. Both grammars are old. |
Beta Was this translation helpful? Give feedback.
-
|
Generated-JavaScript-0.zip. In the zip, look at the build.sh. There's a .ps1 version which will work anywhere, but it doesn't have the |
Beta Was this translation helpful? Give feedback.
-
|
how can i exactly use that |
Beta Was this translation helpful? Give feedback.
-
|
i am in widows10 |
Beta Was this translation helpful? Give feedback.
-
|
Unzip is native to Windows. Install Python. Install antlr4. |
Beta Was this translation helpful? Give feedback.
-
|
i did npm install then run the build.sh and it run with no problem but it did nothing |
Beta Was this translation helpful? Give feedback.
-
|
You're going to have to add output to the build script to see what it does. Here's the code for build.sh. There is no "antlr" command-line call. It's called "antlr4", and it is a Python script to run the tool. This is right from the front page of antlr.org. It's installed by: After installing antlr4-tools, type "antlr4" in a shell of your choice: Explanation of the build.sh codeThis line of code executes a python script in the file "transformGrammar.py" if it exists. It modifies the grammar to work with a particular target. NB: Antlr is NOT target independent code! It has target-specific code called "actions" in the grammar, between open curly brace This code gets the version string from the package.json file. It is literally just "4.13.0". This code executes the "antlr4" python script, which calls the Antlr4 tool to generate a recognizer. The target is JavaScript. For the "Parser.g4", it generates listener and visitor JavaScript classes. Why use the This code reads the package.json file and downloads the required packages. How to run the programYou need to install Node. After installing Node, type |
Beta Was this translation helpful? Give feedback.
-
|
do i need to keep all those files and folders for the parser to work? or that neede just for the generator and the generated files can be used in my actuall app? |
Beta Was this translation helpful? Give feedback.
-
|
Once you have built the code, you don't need the |
Beta Was this translation helpful? Give feedback.
-
|
i have generated the files and took them to my app with the lexer base and parser base files and used babel to transpile them to es5, then i run the app and i get this error when the python lexer class call |
Beta Was this translation helpful? Give feedback.
-
|
For answers to most questions like these, it's a good idea to go to https://chat.openai.com/. The subscription is well worth the price. |
Beta Was this translation helpful? Give feedback.
-
|
maybe for the first question but nor the second |
Beta Was this translation helpful? Give feedback.
-
|
@Elmahdi1962 May I suggest that you read the docs and the book ? This place is for discussion, it seems you are looking for handholding ? |
Beta Was this translation helpful? Give feedback.
-
|
A lexer rule goes in the lexer grammar. A parser rule goes in the parser grammar. In Antlr, parser and lexer symbols are segregated by the name of the symbol: an uppercase name is a lexer symbol, |
Beta Was this translation helpful? Give feedback.
-
|
the docs are not good and straight forward there is no to know how to genearte the parser the way kaby told me, what in the doc is different |
Beta Was this translation helpful? Give feedback.
-
|
@Elmahdi1962 Have you read the book ? |
Beta Was this translation helpful? Give feedback.
-
|
i didnt read the book just the readmes |
Beta Was this translation helpful? Give feedback.
-
|
I strongly suggest you do. Parser generation is neither trivial or intuitive, you can't expect results without learning. |
Beta Was this translation helpful? Give feedback.
-
|
Sorry, but the book is not free and it's outdated (for instance, it doesn't have info about |
Beta Was this translation helpful? Give feedback.
-
|
Well even if it's missing latest updates, it does provide the fundamental architecture that people need to follow. |
Beta Was this translation helpful? Give feedback.
-
|
its not free !! i have to pay to learn the package. nice one |
Beta Was this translation helpful? Give feedback.
-
|
free package but paid docs amazing, thats why there isnt much resources about this tool |
Beta Was this translation helpful? Give feedback.
-
|
I can suggest you free articles, for instance: https://tomassetti.me/antlr-mega-tutorial/ and official doc. The last one was enough for me. |
Beta Was this translation helpful? Give feedback.
-
|
the article is better that the docs the docs are very bad |
Beta Was this translation helpful? Give feedback.
-
|
i need help on creating a visitor and the docs didnt help its just a visitor with the visitchildren method which get called by it self if a i used tree.accept(the visitor) |
Beta Was this translation helpful? Give feedback.
-
|
looks like i had to use the visit method visitor only has this method overridden but how can i get just the return value of the method i override which is the visitFuncdef? |
Beta Was this translation helpful? Give feedback.
-
|
@kaby76 @KvanTTT and this in lexer grammar and i regenerated all the parser file and i overridden the visitAssignment_stmt method |
Beta Was this translation helpful? Give feedback.
-
That's a parse error, not an error from a visitor. When developing a grammar, you must first verify that the parser and lexer are working before using the parse tree in a visitor or listener. But, you don't give the input string, and you don't give the grammar. The .g4's were removed from your .zip file. In fact, none of what you want makes any sense because the original Python3 grammar has assignment already. I suggest that you develop your grammar with Node and VSCode before moving to a browser. Using the original grammar and working on Linux, I generated a driver using Trash, created input with an assignment statement, built the parser, and ran the parser on the input. The output shows the parse tree and the list of tokens for the input. Below is a .tar.gz file with the grammar, driver, and input. (Note, the output below is done was in an Msys shell on Windows11, but I also did it for Linux in WSL2, which is where the the archive was created.) You cannot just add lexer rules to a grammar. That will screw up what tokens are discovered. |
Beta Was this translation helpful? Give feedback.
-
|
i removed the modification of the grammars and i used the Expr-stmt rule but it matches function calls and yields and for annassaign it dosent recognise the tokens properlly as you see in the loged results it take :int=2000000 as a whole token rather than splitting it code input console log |
Beta Was this translation helpful? Give feedback.
-
|
i want to visit just variable assignments not functions calls or anything else |
Beta Was this translation helpful? Give feedback.
-
That statement is just wrong. Using the program I gave you, and creating the input you gave, the parser works on your input, and tokenizes the input correctly. Here you see exactly the list of tokens in the token stream. Notice that the tokens for getText() prints the concatenation of all tokens corresponding to the parse tree node. In order to disallow declarations that you don't want, you should just change the start rule, or add a visitor to disallow parse nodes that you don't want to see in the parse. Do not muck around with the grammar. |
Beta Was this translation helpful? Give feedback.
-
|
how can i disallow parse nodes i dont want and how can i access the node tokens/ |
Beta Was this translation helpful? Give feedback.
-
An Antlr visitor visits everything in a parse tree, whether you implemented a method that acts on the node type or not. I really recommend that you don't change the grammar. You can either start from another rule (you will need to augment the rule with EOF), or write a visitor that does static semantic analysis, which would just return "false" for a parse tree that contains parse tree nodes that you don't want. To get the tokens associated with a node, I suggest you write a method that returns the frontier of a given parse tree. It would look just like the getText() function, which is linked above. It would create and return a list of tokens that you concatenate together from the children of the node. |
Beta Was this translation helpful? Give feedback.
-
|
i dont know how to do any of that |
Beta Was this translation helpful? Give feedback.
-
|
and i didnt understand anything of this |
Beta Was this translation helpful? Give feedback.
-
|
Sorry, but this is rather frustrating. Antlr visitors and listeners are tree traversal implementations you use for program analysis. Please read on how to apply them. See https://tomassetti.me/listeners-and-visitors/. You should understand how to write a recursive function in JavaScript that takes a parse tree node, and performs a DFS traversal that computes a list of tokens associated with the leaves of the tree. It does so by calling the function recursively on the node's children and calculating a list from the value returned for each child. Use ChatGPT to give you a basic implementation of this code. You will need to modify it using the Antlr tree data structures. Using Antlr requires knowledge of tree data structures and algorithms, formal languages, principles in programming languages, and compiler construction. |
Beta Was this translation helpful? Give feedback.
-
|
i managed to do a function that get all tokens of the nodes and it works perfect, but what i couldn't do yet is to visit or rather exclude the nodes that dont represent variable assignment and i am not sure which exact rule that matches just variable assignments |
Beta Was this translation helpful? Give feedback.
-
|
also how can i generate parser in typescript target , i want typescript just for the autocomplete instead of diging in the source code to know the properties available to me |
Beta Was this translation helpful? Give feedback.
-
You will need the changes I have been making to the python grammars. See antlr/grammars-v4#3540. They code is at https://github.com/kaby76/grammars-v4/tree/733c2a79cad925e6eb2ad3ef70c9dfe673d13389/python/python3 |
Beta Was this translation helpful? Give feedback.
-
|
since i am using just the default grammar i am having a hard time extracting just variable assignments from the Expr_stmt rule, even chatgpt cant do it |
Beta Was this translation helpful? Give feedback.
-
|
i need help on making the xpath expression that will find only var assignments |
Beta Was this translation helpful? Give feedback.
-
|
also i dont know how to use xpath the docs are overwhelming especially with the java code |
Beta Was this translation helpful? Give feedback.
-
|
As far as I can tell, the assignment statements are identified by How is this implemented as an Antlr listener? Here's the code:
You'll need to call the listener code after your parser:
|
Beta Was this translation helpful? Give feedback.
-
|
same thing for visitor? |
Beta Was this translation helpful? Give feedback.
-
|
Yes, roughly the same. import antlr4 from 'antlr4';
import Python3ParserVisitor from './Python3ParserVisitor.js';
export default class MyVisitor extends Python3ParserVisitor
{
visitExpr_stmt(ctx) {
if (ctx.ASSIGN() == '=')
{
console.log(ctx.getText());
}
return this.visitChildren(ctx);
}
}...
import MyVisitor from './MyVisitor.js';
...
var visitor = new MyVisitor();
visitor.visitFile_input(tree); |
Beta Was this translation helpful? Give feedback.
@Elmahdi1962 This is an issue with the grammar, not Antlr per se. The grammar that you will want to use is python/python3, but it only works for CSharp, Java, and Python3. I'll make a port for TypeScript today in a PR to round out all the grammar for all 8 targets (not including Swift because I can't get it to install--never could).
In fact, I created an issue on the Python grammars yesterday. before you wrote the question here. antlr/grammars-v4#3539. The state of the Python grammars in grammars-v4 is a mess.