Are older Antlr capabilities implemented in Antlr4

[SteveHespelt]

Just ordered the 2nd edition of the definitive guide but just in case the book doesn't answer this question.
I found an antlr.org wiki page via the wayback machine: http://web.archive.org/web/20130120150251/http://www.antlr.org/wiki/pages/viewpage.action?pageId=557057 in which Terrance, Oliver Zeigermann write an example showing a grammar rule inserting a new CommonTree into the 'current' tree. Nice concise example, hoping to use it to implement import processing in SysML2 textual notation models. I've been digging through the doc folder for stuff giving details on just what constructs are available in Antlr4's parser rules. For example, while -> skip is well-known, does the construct here work in Antlr4?

[....] -> ^('include'  include_filename ^({includetree}))

I'm assuming the grammar has a rule named include_filename as the example does, as well as a token includetree. This syntax to add a CommonTree (or supported types) to the current parse tree seems like a potentially quite powerful feature. Will be nice if Antrl4 has it.

Can someone suggest other materials where the supported RHS of this construct (what's the -> even formally called?) is more thoroughly documented? MTIA.

[SteveHespelt]

Since I'm using the Python3 language target, I'll try to use the parser action to manipulate the RuleContext references (the parentCtx, the self._ctx.parentCtx.children[] & other fields) to insert the 'import ...;' parse tree.

would be ideal if the CommonToken had the name field (& if FileStream ctor set it (since the InputStream has the unused name field... Maybe.

[ericvergnaud]

Can you clarify what you are trying to achieve?

[SteveHespelt]

Hi @ericvergnaud Eric,
Sure, a Python3 package does a partial (limited by design as a POC project) parsing, ingesting of the OMG's SysML v2 textual notation. My task is the expand this functionality. To add the functionality of the v2 spec's import construct. For the purposes of modularity, re-use, the import statement's semantics allow for implementations to pull in from other sources (files, URLs, etc.) the content referred to by the import statement. Functionally, I see this as quite analogous to what a c/c++ preprocessor does, prepping the input stream for the actual compiler to use. As none of the semantical processing of the parse tree happens until after the API exposes it, injecting the import statement's parse tree seems like a natural way of providing the resulting unified parse tree. But I still hold out the strong likelihood I'm very wrong.

And Antl4's grammar was chosen to express the SysML v2 notation - I'm glad of that call as it's not been too difficult to start using Antlr4 (last Monday was the first time I've ever touched Antlr4), quite pleased with the value of using Antlr4.

I hope this has clarified a bit better what I'm trying to achieve. any & all feedback is not only welcome, it's encouraged. Thanks for taking the time to read my posts & for providing feedback.

[kaby76]

The -> ... is called a tree rewrite. Antlr3 had tree rewriting, but Antlr4 does not, at least not in the Antlr4 grammar syntax. Yes, you could implement a tree rewrite with a parser listener. Or, you could use a tree listener with a token stream rewrite. Note, the Python3 target is very slow, so you may want to implement this with a compiled language target. For a command-line solution, use my Trash toolkit.

[SteveHespelt]

Hi @kaby76 Ken, Thanks for the quick & helpful reply. Am I understanding correctly that in my post the content after the -> syntax in the rule is the "tree rewrite"? (ie. the ^('include' include_filename ^({includetree}) is the tree-rewrite as the leading ^ probably implies "insert as the current node produced by this rule via the following fields - node name == 'include', some text/info field contains the $include_filename.text and the last ^ implies 'insert this tree as a node field as well). That sentence was way too long...
I've not seen what the '->' operator is actually called formally.

In my use case, I have to use Python3. My task is to extend the functionality of a Python3 package that consumes SysML v2 textual notation & exposes the parsed content's parse tree. The performance overhead is what it is. The 3 Rs come into the equation ;-)

Yesterday, I used subclasses from the generated parser, lexer classes to deal with injecting new tokens into the stream consumed by the parser, but I think the parse tree injection might be a bit cleaner, less intrusive into the current code.

Thanks for the pointer re the Trash toolkit, I will definitely take a look. If not for this task, some future task.

[kaby76]

@SteveHespelt Check out the tree construction documentation in Antlr3. https://theantlrguy.atlassian.net/wiki/spaces/ANTLR3/pages/2687090/Tree+construction. The doc names the -> symbol as the "rewrite operator". The ^ symbol is the "new tree node operator". There's also a ! operator.

[SteveHespelt]

Turns out my use case has a slightly different wrinkle. SysML v2 imports don't actually map to a particular file(s) but as namespaces, they need to match 'known' namespaces - either seen inline (same notational blob(text stream)) or known to whatever is processing the SysML notational text (eg. the Eclipse modeling framework built with SysML v2 plugins, or the OMG's Jupyter Sysml kernel - these 2 load all recognized files so the content can be matched later). So the Python3 package will need to do the same, adding load, reload, etc. semantics to its API. But I might still use the parse tree constructs - fetch the 'right' files, parse them, extract out the namespace identifiers, keep the mapping of namespace to parse tree for each in a lookup table, retrieve the matching parse tree when the import is seen while applying any import semantics to the sub-tree clone being inserted. So the stuff I mentioned at 10am today mostly still applies. Just a matter of when & more manipulation.