Proposal: Add action templates functionality #4345

DavidGregory084 · 2023-07-03T16:53:13Z

This is an idea about how to resolve #4067 without developing an entirely new actions language.

This PR enables users to write cross-target grammars which make use of actions, by enabling users to provide action templates as StringTemplate .stg group files on the command line.

By providing different action templates for each target language, users can provide a different implementation of the action logic for each target.

Java.stg:

normalizerImports() ::= <<
import java.text.Normalizer;
import java.text.Normalizer.Form;
>>
normalize(s) ::= <<Normalizer.normalize(<s>, Form.NFKC)>>
getText() ::= <<getText()>>
setText(s) ::= <<setText(<s>);>>

Javascript.stg:

normalizerImports() ::= ""
normalize(s) ::= <<<s>.normalize("NFKC")>>
getText() ::= "this.text"
setText(s) ::= "this.text = <s>"

The example below is the motivating example for me - I have a grammar that I'd like to use in both a JVM-based compiler and a VS Code extension:

Example.g4:

lexer grammar Example;

@lexer::header {
<normalizerImports()>
}

ID : (ID_START ID_CONTINUE* | '_' ID_CONTINUE+) { <setText(normalize(getText()))> } ;
ID_START : [\p{XID_Start}] ;
ID_CONTINUE: [\p{XID_Continue}] ;
WS : (' '|'\n') -> skip ;

Thanks to my employer @opencastsoftware for sponsoring this work!

tool/src/org/antlr/v4/tool/GrammarTransformPipeline.java

tool/src/org/antlr/v4/Tool.java

This enables users to write cross-target grammars which make use of actions, by enabling users to provide action templates on the command line. By providing different action templates for each target language, users can provide a different implementation of the action logic for each target. Signed-off-by: David Gregory <2992938+DavidGregory084@users.noreply.github.com>

Signed-off-by: David Gregory <2992938+DavidGregory084@users.noreply.github.com>

ericvergnaud · 2023-07-03T17:31:58Z

Mmmm... this IS an "entirely new actions language". The fact that it leverages templates makes it look like not (and is elegant), but we'd still have to opine on the syntax and more importantly: the supported set of 'keywords', both characteristics of a language...
The latter could be a blocker because we precisely don't want to find ourselves in a situation where we'd have to expand that set for every language, because the existing set does not support the specific use case a user needs to support...
Using templates for testing works because the set is limited to what the engine requires, and the task of supporting additional templates lies with the developer needing it, but with your proposal that task would lie with target authors who don't actually need it, so not sure it would work well.
As you know, there are other options being discussed.

DavidGregory084 · 2023-07-03T17:38:13Z

Mmmm... this IS an "entirely new actions language". The fact that it leverages templates makes it look like not (and is elegant), but we'd still have to opine on the syntax and more importantly: the supported set of 'keywords', both characteristics of a language...

Hmm can you explain what you mean by that? The syntax is that of StringTemplate, and all of the template identifiers used are user-defined - they’re simply supplied as StringTemplate groups.

DavidGregory084 · 2023-07-03T17:58:44Z

Ah I think I understand - I’m not proposing that there is a centrally managed collection of StringTemplate groups shipped with ANTLR - this PR adds a command line option via which users can supply their own templates which they have written themselves.

Writing cross-target grammars is a rather advanced use case and I think it’s reasonable to expect users to learn StringTemplate syntax if they want to do that.

DavidGregory084 · 2023-07-04T12:18:40Z

Why StringTemplate?

It's used in the implementation of ANTLR itself and in the ANTLR code generation process, so it's familiar to ANTLR contributors
It's already on the classpath of the ANTLR tool, so there are no new dependencies needed
It's a stable and well-tested template language that will allow users to provide whatever kind of action logic they want

kaby76 · 2023-07-04T13:02:37Z

I have a few observations and questions.

I've been writing target-agnostic Antlr grammars probably more than most, porting grammars-v4 to different targets, pretty much every day of the week for several years, and just got through yet another, python/python3. The main difficulty is that the runtime APIs across targets are inconsistent. Three bugs were discovered ([typescript] Runtime declares CommonToken(source, .....) wrong. #4318 [Dart] Fix for #4320--export additional types #4340 [Go] Fix for #4342--EmitToken(token) in Go target needs to be exported #4343). While annoying, dealing with actions per target is manageable using target agnostic format.
Importantly, why is this part of a parser generator? And, as with any parser generator, where is the boundary between parser generator and some other tool that uses it? For example, I am firmly against embedding a sub-par XPath engine (pre-version 1) in the Antlr runtime. It exists only for some targets. That is why I ported a XPath version 2 engine to C#, and added a DOM interface over the Antlr parse tree in a separate tool. Antlr trees change very rarely, which means that it can be released on its own schedule, not tied at all to a minor version number change in Antlr.
Note, I've been writing, already, a string processor for Antlr grammars, trgen, so I am familiar with ST--and all it's problems--and the idea of separating ST processing from the parser generator. (I've been thinking of a more general processor as well, and have always been struck by the fact that ST does not contain a general-purpose, command-line processor.)
I am not at all happy with the way the model-view is separated in ST. For example, you cannot write a simple string equality test: <if(target=="CSharp")>....<endif>.

DavidGregory084 · 2023-07-04T14:12:23Z

While annoying, dealing with actions per target is manageable using target agnostic format.

I think the Fortran90 grammar is actually a good example in itself of the need for some kind of solution to target-agnostic actions.

The target-agnostic format described there relies upon superClass, language keywords like this and property dereference via . in order to work, which means that it falls short for non-OO targets like Go and needs adjustments in order to produce valid code. It’s great that such a pattern exists and is well documented, but I think a more integrated solution would be preferable.

In many cases it would be really difficult for users to integrate such a script into their workflow (e.g. when the ANTLR generator is called by a build tool plugin or as part of some other complex workflow).

Importantly, why is this part of a parser generator? And, as with any parser generator, where is the boundary between parser generator and some other tool that uses it?

ST is already a dependency of the ANTLR tool, so I guess to some extent the question of whether to embed it into ANTLR has already been decided. The expansion of ST templates in this PR is completely optional and is only active if the user provides the -DactionTemplates command line flag.

I suppose the boundary here is the point of template expansion, which happens early in the ANTLR generation process just after parsing the grammar file itself. It's also where the most troubling interactions occur, in terms of showing users error messages relating to the template expansion process.

I am not at all happy with the way the model-view is separated in ST.

I think that is a moot point in this implementation as no model binding is performed - the user has to write all of the groups that they want to use in their .stg template, and they can use only those groups in their grammar. 😄

Signed-off-by: David Gregory <2992938+DavidGregory084@users.noreply.github.com>

kaby76 · 2023-07-04T18:31:45Z

Usually, I would say that ST should be supported outside the tool, but after thinking about this a bit, I think you are right. The main reason why it should be supported in the tool is because people would say a grammar that requires further processing shouldn't be called an Antlr grammar. In addition, the grammar could be tagged as requiring a certain version of Antlr. I will try out some examples shortly.

kaby76 · 2023-07-04T19:05:56Z

Semantic predicates are actions, too. It's not expanding template references there, but it is working elsewhere.

header() ::= <<
int count;
>>

action1() ::= <<
this.count = 0;
>>

pred() ::= <<
this.count \<= 0
>>

action2() ::= <<
this.count++;
>>

grammar Expr;

options { actionTemplates='Temp.stg'; }

@members {
<header()>
}

file : x (SEMI x)* SEMI? EOF;
x : { <action1()> } expression;
expression : expression POW expression
 | expression (TIMES | DIV) expression
 | expression (PLUS | MINUS) expression
 | LPAREN expression RPAREN
 | (PLUS | MINUS)* atom ;
atom : scientific | { <pred1()> }? variable { <action2()> } ;
scientific : SCIENTIFIC_NUMBER ;
variable : VARIABLE ;

VARIABLE : VALID_ID_START VALID_ID_CHAR* ;
SCIENTIFIC_NUMBER : NUMBER (E SIGN? UNSIGNED_INTEGER)? ;
LPAREN : '(' ;
RPAREN : ')' ;
PLUS : '+' ;
MINUS : '-' ;
TIMES : '*' ;
DIV : '/' ;
GT : '>' ;
LT : '<' ;
EQ : '=' ;
POINT : '.' ;
POW : '^' ;
SEMI : ';' ;
WS : [ \r\n\t] + -> channel(HIDDEN) ;

fragment VALID_ID_START : ('a' .. 'z') | ('A' .. 'Z') | '_' ;
fragment VALID_ID_CHAR : VALID_ID_START | ('0' .. '9') ;
fragment NUMBER : ('0' .. '9') + ('.' ('0' .. '9') +)? ;
fragment UNSIGNED_INTEGER : ('0' .. '9')+ ;
fragment E : 'E' | 'e' ;
fragment SIGN : ('+' | '-') ;

DavidGregory084 · 2023-07-04T19:11:15Z

Semantic predicates are actions, too. It's not expanding template references there, but it is working elsewhere.

Ah thanks for trying it out - I will take a look at semantic predicates tomorrow

kaby76 · 2023-07-04T22:34:48Z

Looks like you need to test for SEMPRED in addition to ACTION in expandActionTemplates(GrammarRootAST root). Then, trim the extra '?' from the text of the predicate. Honestly, I don't understand why the text isn't just trimmed of '{', '}', and '?' when the parse tree is constructed. You might need to check the equivalent action blocks for a lexer grammar. I don't know if the parse tree is very different.

Otherwise, I think solution is better than the others we've thought of up to now. It fixes the "p" vs "l" problem, as well as the "this." vs "$this->" vs "self." vs "this->" problem. And it doesn't result in a proliferation of if-then-else code sprinkled through the grammar itself.

I don't think we have any more non-OO targets. The Go runtime is slammed into something that resembles an OO framework. It just wan't implemented well to begin with. But, ST actions could come in handy with a new target like C.

This doesn't take care of the target-specific code in local and return clauses. But, we don't use those features in grammars-v4.

You got my vote. @parrt Please consider this PR. It goes a long way to fixing the problems in writing grammars over different targets.

ericvergnaud · 2023-07-05T07:22:12Z

A similar but more independent approach would be to write the grammar as a template and apply the transforms in order to produce a target specific g4. Less convenient but does not require any change to antlr4 tool.

kaby76 · 2023-07-05T09:36:32Z

The problem with writing the grammar as STG-format file is that it is then not an Antlr grammar. The format could not be parsed using the grammars-v4 antlr4 grammar. Existing VSCode or VSIDE extensions would not work. People would not call it an Antlr grammar. The use of the template calls would not be restricted to just the actions. Not a good solution.

You could implement this as a separate command-line tool called "antlr4-plus", processing the .g4 as a ST formatted file where ST attributes are referenced only in the actions, the proposed format for the grammar. This tool could parse, then render a normal Antlr4 grammar. VSCode and VSIDE extensions would still work. But, people would not call the grammar an "Antlr4 grammar", but an "Antlr4-plus" grammar. It would be labeled a third-party tool, and any grammars in the format would probably get a new extension, like ".g4plus".

The template group file reference is directly listed as an option in the grammar. It's an explicitly listed dependency, which is a much better thing than hiding it. If the grammar option contains options { actionTemplates='Temp.stg'; }, and folks accidentally run the Antlr4 tool on it, the Antlr tool would give a warning because the tool warns on all options it does not know. If the build uses -Werror, the build breaks which is correct, but, people would say "What is this? Is this an Antlr4 grammar?". They're then going to have to redo all their build tooling, new rules in the NPM .json, pom.xml, .csproj, etc. teverett and KvanTTT won't like adding .g4plus files to grammars-v4 because they're not exactly Antlr4 grammars.

Folks are very hesitant to use third-party tools. They won't know how to install, screw up on install. Antlr4-tools is official, and directly referenced in antlr.org, nothing further to do. Nobody trusts my Trash toolkit--years in development, used in grammars-v4, extremely useful. But, it's not listed in antlr.org, won't ever get there and only modified by me. Several other tools that are Antlr based, and they're also not listed there, like XText (although that is Antlr3-based). I'm sure there are others.

ericvergnaud · 2023-07-05T10:38:20Z

@kaby76 re the template not being a valid grammar, that's a fair point.
That said I think actionTemplates should be a command line option only. Its purpose is to provide a different implementation per target. Making it a grammar option ( using options { actionTemplates='Temp.stg'; } ) pushes the responsibility of locating that template to antlr, which is wrong.
@DavidGregory084 there is a big downside to this which relates to automation. Many developers use the maven plugin to generate code at build time (rather than at dev time - which I prefer). The plugin would now need to know where to find the action templates, on a per target basis. Does your PR cater for this ?

ericvergnaud · 2023-07-05T10:40:15Z

tool/src/org/antlr/v4/tool/Grammar.java

@@ -80,6 +80,7 @@ public class Grammar implements AttributeResolver {
 		parserOptions.add("TokenLabelType");
 		parserOptions.add("tokenVocab");
 		parserOptions.add("language");
+		parserOptions.add("actionTemplates");


Not sure we want to support this as a grammar option, it should be a cmd line option only

ericvergnaud · 2023-07-05T10:41:04Z

tool/src/org/antlr/v4/tool/Grammar.java

@@ -1178,6 +1179,10 @@ public String getLanguage() {
 		return getOptionString("language");
 	}

+	public String getActionTemplates() {


kaby76 · 2023-07-05T11:46:01Z

Making it a grammar option ( using options { actionTemplates='Temp.stg'; } ) pushes the responsibility of locating that template to antlr, which is wrong.

Well, I could solve the explicit dependency by applying a naming standard to the .g4's and .stg's, e.g., Python3Lexer.g4/Python3Lexer.stg and Python3Parser.g4/Python3Parser.stg. I already apply a coding standard for the location of target-specific files over in grammars-v4. It took years to straighten it all out, but all target-specific files are in sub-directories Cpp/, CSharp/, Dart/, Go/, Java/, JavaScript/, PHP/, Python3/, and TypeScript/. So, these .stg's would go into each of the sub-directories. That's fine because that's where the superClass files already exist. The base classes can be easily rewritten as .stg's. I'd know which grammar requires a command-line option -actionTemplates=foobar for trgen, and could then generate a build script in whatever build system (dotnet, maven, npm, etc) to apply the command-line option.

But, why is an equivalent actionTemplates "wrong"? The Antlr tool already has a superClass option in the grammar. Why even use options { superClass=Python3LexerBase; } when we can just do antlr4 -DsuperClass=Python3LexerBase Python3Lexer.g4 and assume so because we have a naming scheme?

kaby76 · 2023-07-05T12:07:47Z

there is a big downside to this which relates to automation. Many developers use the maven plugin to generate code at build time (rather than at dev time - which I prefer). The plugin would now need to know where to find the action templates, on a per target basis. Does your PR cater for this ?

All the more that it should be explicitly listed in the grammar file, and not pushed to the build tool. You won't have to change the build tool, except that the grammar now requires a new version of the Antlr tool which people do anyways.

ericvergnaud · 2023-07-05T12:11:55Z

@kaby76 Sorry to disagree, but that would have to be a new entry in the pom file. I am not supportive of an approach that requires the tool to resolve the stg file path in the first place, and differently for each target.

kaby76 · 2023-07-05T12:19:31Z

@kaby76 Sorry to disagree, but that would have to be a new entry in the pom file. I am not supportive of an approach that requires the tool to resolve the stg file path in the first place, and differently for each target.

First, actionTemplates is not a directory, it's a top-level file name, including path. The path can encode the target, but it's best not to do that. And, I would not have the -Dlanguage option (or hardwired option in the grammar file) affect where to find the templates. That would not be referentially transparent.

At least in grammars-v4, nothing in the .g4 would need to be changed per target. Just place the .stg's in the same directory as the .g4's. The command line to generate the target-specific recognizers would still be the same: antlr4 -actionTemplates=Python3Lexer.stg -Dlanguage=CSharp Python3Lexer.g4; antlr4 -actionTemplates=Python3Lexer.stg -Dlanguage=CSharp Python3Lexer.g4, or antlr4 -actionTemplates=Python3Lexer.stg -Dlanguage=Java Python3Lexer.g4; antlr4 -actionTemplates=Python3Lexer.stg -Dlanguage=Java Python3Lexer.g4.

The target-specific nature of the templates is laid out in the coding standard--the directory structure--after years of arguing about it in grammars-v4. But, as far as I know, the Maven plugin does not work for other targets besides Java. But, for trgen, the target specific files are overlayed over the files where the .g4's reside.

If you really want to place the .g4's and .stg's in separate directories, the Antlr tool already has a way to do that with -lib on the command-line. I think the PR should be updated to reference that path. The target-specific nature would then tied with an explicit target mentioned in the -Dlanguage and the -lib option together.

Nothing in the pom.xml would need to change whatsoever other than now adding the -actionTemplates option--which I wouldn't do, because I would always write the grammar with the option in the grammar itself, just like the superClass option. You could then group the .stg files with the .g4's all in one directory. Trgen would still work because the target-specific files are copied to where the .g4 files are, overwriting the Java-specific files.

Signed-off-by: David Gregory <2992938+DavidGregory084@users.noreply.github.com>

DavidGregory084 · 2023-07-05T13:18:09Z

That said I think actionTemplates should be a command line option only. Its purpose is to provide a different implementation per target. Making it a grammar option ( using options { actionTemplates='Temp.stg'; } ) pushes the responsibility of locating that template to antlr, which is wrong.

I agree that a target-specific option does not make a lot of sense in the grammar file - the only reason that I have done it this way is that the "language" option is already a grammar-level option and "actionTemplates" seems to belong wherever "language" is defined - happy to change it.

I expect most folks would provide "actionTemplates" as a command line option.

EDIT: I see that @kaby76 has a slightly different pattern in mind but both usage patterns would work with a grammar-level option I think?

@DavidGregory084 there is a big downside to this which relates to automation. Many developers use the maven plugin to generate code at build time (rather than at dev time - which I prefer). The plugin would now need to know where to find the action templates, on a per target basis. Does your PR cater for this ?

Not at the moment - I think that would require changes to the Maven plugin and I haven't looked at that part of the project yet.

The Gradle plugin already requires manually manipulating the command line options for any advanced use-cases, e.g. here's the invocation from my current project:

tasks.generateGrammarSource {
    // See: https://github.com/antlr/antlr4/issues/2335
    val outputDir = file("build/generated-src/antlr/main/org/mina_lang/parser")
    // the directory must exist or ANTLR bails
    doFirst { outputDir.mkdirs() }
    arguments = arguments + listOf(
        "-visitor",
        "-no-listener",
        // the lexer tokens file can't be found by the parser without this
        "-lib", outputDir.absolutePath,
        "-package", "org.mina_lang.parser")
}

Having said that, I don't really know which use cases would require providing different arguments to the same build tool?

e.g. in my case, I will probably move my grammar to a top-level directory, and symlink it into my Gradle source directories and my npm source directories. Then each build tool will invoke the ANTLR tool with different command line arguments to generate the target that's relevant for that build tool - Java for my Gradle build, TypeScript for my npm build.

DavidGregory084 · 2023-07-05T13:22:24Z

tool-testsuite/test/org/antlr/v4/test/tool/TestActionTemplates.java

+		assertEquals(
+			"State: Generate; \n" +
+			"error(211): " + grammarFile + ":2:14: error compiling action template: 3:3: invalid character '¢'\n" +
+			"error(211): " + grammarFile + ":2:14: error compiling action template: 3:0: mismatched input '  ' expecting EOF\n",


This message is a little weird - I'm not sure why changing the action template to be multiline triggers this error message from StringTemplate

parrt · 2023-07-29T17:04:19Z

Very interesting and well thought out proposal. Away from my computer, but I am positively inclined to approve something in this nature. So it sounds like the grammar itself would mention the name of a template file, and then the -lib would indicate a directory with a template file. But what is the convention for changing the name of the template according to the language? we definitely don’t want to name the target language inside the grammar file as that defeats the purpose. Seems to me that you might simply want to have a command line option that indicates the specific template file to use, but then it means we don’t need to specify the template file name inside the grammar. But that has the negative that users will be mystified when the code generator does not compile as you point out

ericvergnaud · 2023-07-29T17:22:50Z

Thanks for this. I’m all for making this a cmd line only option. We don’t went to embed any target directory logic. Re non compiling parser when the option is missing, it doesn’t sound difficult to detect a string template and throw an error if no template is provided.Envoyé de mon iPhoneLe 29 juil. 2023 à 19:04, Terence Parr ***@***.***> a écrit : Very interesting and well thought out proposal. Away from my computer, but I am positively inclined to approve something in this nature. So it sounds like the grammar itself would mention the name of a template file, and then the -lib would indicate a directory with a template file. But what is the convention for changing the name of the template according to the language? we definitely don’t want to name the target language inside the grammar file as that defeats the purpose. Seems to me that you might simply want to have a command line option that indicates the specific template file to use, but then it means we don’t need to specify the template file name inside the grammar. But that has the negative that users will be mystified when the code generator does not compile as you point out —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: ***@***.***>

kaby76 · 2023-07-30T10:32:56Z

I’m all for making this a cmd line only option. We don’t went to embed any target directory logic.

If you want to declare the .stg/.g4 pairing outside the grammar file--in a bash script, makefile, Maven file, etc--that's your choice. But, please don't force the coding style on me. I would prefer to declare the actionTemplates option in the grammar because it has to be specified somewhere.

In my example, there is no mention of target in either of the .g4's: options { actionTemplate='Python3Parser.stg';} is the same for all targets. There are no forked copies of the .g4's for each target (a maintenance nightmare). There is no target-specific logic in the .g4. The templated actions are the same for all targets in the .g4, e.g., @header {<header()>}. Really, there is zero target logic in the .g4 files in my example. The target-specific choice is in the generated build script. trgen overlays the grammar with target-specific files. Alternatively, it would be fine if the target-specific .stg's could be found using the -lib option.

If the pairing of top-level .stg with the .g4 is not specified in the grammar, I'll have to devise a way to know when to add the -DactionTemplates='...' option to the generated build script. Some grammars in grammars-v4 have multiple .g4's, e.g., csharp. If the .stg/.g4 pairing is not declared in the .g4, it would need to be declared in the desc.xml or computed by parsing the .g4's, .stg's. Driver code is not part of the grammar we share in grammars-v4.

DavidGregory084 · 2023-07-30T14:17:02Z

So it sounds like the grammar itself would mention the name of a template file, and then the -lib would indicate a directory with a template file.

Yes, that's right, actionTemplates is currently a grammar-level option, but note that grammar-level options can always be provided to the ANTLR tool via the command line, i.e. antlr4 -DactionTemplates=Actions.stg.

It seems that @ericvergnaud is strongly against keeping this as a grammar-level option and @kaby76 is strongly for it, so I hope you can be a tie breaker. 😅

I will note though that I don't see any harm in leaving this at the grammar level if it suits use cases whereby the relative folder structure or -lib argument is used to determine where a fixed template file name mentioned in the grammar file is picked up from.

ericvergnaud · 2023-07-31T11:48:41Z

Since -lib describes an input directory, I’m fine with the tool trying to locate the template file in that directory.But I will strongly oppose any logic where the template file name needs to be inferred from the target.Envoyé de mon iPhoneLe 30 juil. 2023 à 16:17, David Gregory ***@***.***> a écrit : So it sounds like the grammar itself would mention the name of a template file, and then the -lib would indicate a directory with a template file. Yes, that's right, actionTemplates is currently a grammar-level option, but note that grammar-level options can always be provided to the ANTLR tool via the command line, i.e. antlr4 -DactionTemplates=Actions.stg. It seems that @ericvergnaud is strongly against keeping this as a grammar-level option and @kaby76 is strongly for it, so I hope you can be a tie breaker. 😅 I will note though that I don't see any harm in leaving this at the grammar level if it suits use cases whereby the relative folder structure or -lib argument is used to determine where a fixed template file name mentioned in the grammar file is picked up from. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>

kaby76 · 2023-07-31T16:07:24Z

Since -lib describes an input directory, I’m fine with the tool trying to locate the template file in that directory.But I will strongly oppose any logic where the template file name needs to be inferred from the target.

I agree!

We need a change for finding the .stg along the -lib path.

07/31-12:01:45 ~/spython3-copy
$ grep actionTemplates Python3Lexer.g4
    actionTemplates='Python3Lexer.stg';
07/31-12:01:58 ~/spython3-copy
$ antlr4 -Dlanguage=CSharp -lib CSharp Python3Lexer.g4
error(206):  cannot find action templates file Python3Lexer.stg given for Python3Lexer
07/31-12:02:05 ~/spython3-copy
$ pushd
~/spython3-cmd-line ~/spython3-copy
07/31-12:02:07 ~/spython3-cmd-line
$ grep actionTemplates Python3Lexer.g4
07/31-12:02:21 ~/spython3-cmd-line
$ antlr4 -Dlanguage=CSharp -lib CSharp -DactionTemplates=Python3Lexer.stg Python3Lexer.g4
error(206):  cannot find action templates file Python3Lexer.stg given for Python3Lexer

DavidGregory084 · 2023-07-31T16:13:15Z

@kaby76 that change should already have been implemented as of 84866cc - are you using an earlier commit or is this something that I need to look into?

ericvergnaud · 2023-08-30T09:01:13Z

@parrt this is not ready to merge, but I guess we have your green light to complete the work ?

kaby76 · 2023-08-30T09:11:37Z

@parrt this is not ready to merge, but I guess we have your green light to complete the work ?

I am still waiting for the -lib option fix. That part does not work. I never received a reply from @DavidGregory084 addressing the bug that I found. If he doesn't fix the problem, I would still recommend the merge because it would still be good enough to redo the grammars in grammars-v4. I can fix the bug in a follow-up PR.

ericvergnaud · 2023-08-30T09:15:02Z

@parrt this is not ready to merge, but I guess we have your green light to complete the work ?

I am still waiting for the -lib option fix. That part does not work. I never received a reply from @DavidGregory084 addressing the bug that I found. If he doesn't fix the problem, I would still recommend the merge because it would still be good enough to redo the grammars in grammars-v4. I can fix the bug in a follow-up PR.

mmm... I think we would need to add support for this in the maven and the IntelliJ plugins at minimal, and antlr-lab

DavidGregory084 · 2023-08-30T14:13:18Z

I am still waiting for the -lib option fix. That part does not work. I never received a reply from @DavidGregory084 addressing the bug that I found. If he doesn't fix the problem, I would still recommend the merge because it would still be good enough to redo the grammars in grammars-v4. I can fix the bug in a follow-up PR.

@kaby76 I asked you a question about this in my last comment:

@kaby76 that change should already have been implemented as of 84866cc - are you using an earlier commit or is this something that I need to look into?

Which revision are you using?

kaby76 · 2023-08-30T17:41:30Z

Which revision are you using?

I was using the latest. I always use the latest. But of course, mistakes sometimes happen. But not this time.

I created a repo (https://github.com/kaby76/do-over-and-over.git) with a script (https://github.com/kaby76/do-over-and-over/blob/main/do-over-and-over.sh) that clones your repo, does a build and run for three different versions.

For commit 84866cc, which you assert fixes the -lib problem--it does not as shown here, "error(206):....".
For tip of branch action-templates. The problem still occurs, as shown here, "error(206):....".
For tip of branch action-templates, patching the code using fix.txt at this point in the output. My patch works, as shown here with no errors.

DavidGregory084 · 2023-08-30T21:37:50Z

Cool thanks for confirming - I'll take a look as soon as I can!

ericvergnaud · 2023-10-17T11:29:57Z

@DavidGregory084 hi there. It's been a while. Any chance you will progress this ?

ericvergnaud · 2024-03-01T11:20:22Z

@DavidGregory084 we'd like to include this feature in ANTLR5.
Would you be able to migrate this PR accordingly ?

DavidGregory084 · 2024-03-01T12:01:57Z

Apologies @ericvergnaud I have not had a great deal of time for open source over the past few months. What are the timelines like for ANTLR5? I will try to make some time for this next week!

ericvergnaud · 2024-03-01T12:50:48Z

The overall timeline is flexible, but this feature will rapidly become critical due to the new architecture.

… unit test Signed-off-by: David Gregory <2992938+DavidGregory084@users.noreply.github.com>

DavidGregory084 · 2024-04-15T11:26:19Z

@ericvergnaud apologies for the delay - @kaby76 was absolutely right about the cause of the issue, but I wanted to write a unit test to exercise the -lib directory code and never seemed to find the time.

I have incorporated @kaby76's fix now and added a test.

Next I will migrate the PR to the antlr5 repo.

Will you still accept this PR on antlr4 as well, or should we close it here?

ericvergnaud · 2024-04-15T12:28:22Z

@DavidGregory084 I wouldn't close it. Once it is successfully implemented in antlr5, its usefulness might be proven well enough for Ter to accept it.

DavidGregory084 · 2024-04-15T18:02:45Z

Opened a PR for the port at antlr/antlr5#51

kaby76 · 2024-04-16T20:26:42Z

Note, templates could be applied elsewhere in the grammar e.g., UP_TO_THREE_DOTS: <repeat(3, '.')> ;. Not a great example, but just to illustrate a point. #2577

DavidGregory084 commented Jul 3, 2023

View reviewed changes

tool/src/org/antlr/v4/tool/GrammarTransformPipeline.java Show resolved Hide resolved

DavidGregory084 commented Jul 3, 2023

View reviewed changes

tool/src/org/antlr/v4/tool/GrammarTransformPipeline.java Outdated Show resolved Hide resolved

DavidGregory084 commented Jul 3, 2023

View reviewed changes

tool/src/org/antlr/v4/Tool.java Outdated Show resolved Hide resolved

DavidGregory084 added 2 commits July 3, 2023 18:03

Improve variable naming in action templates test suite

573908d

Signed-off-by: David Gregory <2992938+DavidGregory084@users.noreply.github.com>

DavidGregory084 force-pushed the action-templates branch from 4553aa8 to 573908d Compare July 3, 2023 17:04

More consistent naming in action templates error messages

616d764

Signed-off-by: David Gregory <2992938+DavidGregory084@users.noreply.github.com>

DavidGregory084 force-pushed the action-templates branch from c89adcc to 616d764 Compare July 4, 2023 15:06

ericvergnaud reviewed Jul 5, 2023

View reviewed changes

Integrate ST error reporting with ANTLR's grammar error reporting

f89c520

Signed-off-by: David Gregory <2992938+DavidGregory084@users.noreply.github.com>

DavidGregory084 commented Jul 5, 2023

View reviewed changes

kaby76 mentioned this pull request Aug 7, 2023

[PL/SQL] basic example as described by the documentation reports a TypeScript type error antlr/grammars-v4#3647

Closed

DavidGregory084 closed this Aug 30, 2023

DavidGregory084 reopened this Aug 30, 2023

ericvergnaud mentioned this pull request Oct 17, 2023

Wrong generated code for arguments with default value in Python target #4443

Open

mike-lischke mentioned this pull request Apr 1, 2024

> StringTemplate for target actions in grammars mike-lischke/antlr4-runtime-benchmarks#2

Closed

DavidGregory084 force-pushed the action-templates branch from ec89478 to 5912bef Compare April 15, 2024 11:23

Fix a bug in passing the path to the StringTemplate group file, add a…

0ae16da

… unit test Signed-off-by: David Gregory <2992938+DavidGregory084@users.noreply.github.com>

DavidGregory084 force-pushed the action-templates branch from 5912bef to 0ae16da Compare April 15, 2024 11:24

DavidGregory084 mentioned this pull request Apr 15, 2024

Port action templates functionality to ANTLR5 antlr/antlr5#51

Open

Proposal: Add action templates functionality #4345

Are you sure you want to change the base?

Proposal: Add action templates functionality #4345

Conversation

DavidGregory084 commented Jul 3, 2023 • edited

ericvergnaud commented Jul 3, 2023

DavidGregory084 commented Jul 3, 2023

DavidGregory084 commented Jul 3, 2023

DavidGregory084 commented Jul 4, 2023

kaby76 commented Jul 4, 2023 • edited

DavidGregory084 commented Jul 4, 2023 • edited

kaby76 commented Jul 4, 2023

kaby76 commented Jul 4, 2023 • edited

DavidGregory084 commented Jul 4, 2023

kaby76 commented Jul 4, 2023

ericvergnaud commented Jul 5, 2023 via email

kaby76 commented Jul 5, 2023 • edited

ericvergnaud commented Jul 5, 2023

ericvergnaud Jul 5, 2023 • edited

Choose a reason for hiding this comment

ericvergnaud Jul 5, 2023

Choose a reason for hiding this comment

kaby76 commented Jul 5, 2023 • edited

kaby76 commented Jul 5, 2023

ericvergnaud commented Jul 5, 2023

kaby76 commented Jul 5, 2023 • edited

DavidGregory084 commented Jul 5, 2023 • edited

DavidGregory084 Jul 5, 2023 • edited

Choose a reason for hiding this comment

parrt commented Jul 29, 2023

ericvergnaud commented Jul 29, 2023 via email

kaby76 commented Jul 30, 2023 • edited

DavidGregory084 commented Jul 30, 2023

ericvergnaud commented Jul 31, 2023 via email

kaby76 commented Jul 31, 2023

DavidGregory084 commented Jul 31, 2023

ericvergnaud commented Aug 30, 2023

kaby76 commented Aug 30, 2023

ericvergnaud commented Aug 30, 2023 • edited

DavidGregory084 commented Aug 30, 2023

kaby76 commented Aug 30, 2023

DavidGregory084 commented Aug 30, 2023

ericvergnaud commented Oct 17, 2023

ericvergnaud commented Mar 1, 2024

DavidGregory084 commented Mar 1, 2024

ericvergnaud commented Mar 1, 2024

DavidGregory084 commented Apr 15, 2024 • edited

ericvergnaud commented Apr 15, 2024

DavidGregory084 commented Apr 15, 2024

kaby76 commented Apr 16, 2024

DavidGregory084 commented Jul 3, 2023 •

edited

kaby76 commented Jul 4, 2023 •

edited

DavidGregory084 commented Jul 4, 2023 •

edited

kaby76 commented Jul 4, 2023 •

edited

kaby76 commented Jul 5, 2023 •

edited

ericvergnaud Jul 5, 2023 •

edited

kaby76 commented Jul 5, 2023 •

edited

kaby76 commented Jul 5, 2023 •

edited

DavidGregory084 commented Jul 5, 2023 •

edited

DavidGregory084 Jul 5, 2023 •

edited

kaby76 commented Jul 30, 2023 •

edited

ericvergnaud commented Aug 30, 2023 •

edited

DavidGregory084 commented Apr 15, 2024 •

edited