improving antlr release process to increase frequency of release

[parrt]

Howdy. if you take a look at the antlr release doc, you will see that it is an extremely involved process. The primary culprit is the fact that we support nine different target languages, and each one has special requirements.

One big issue as a result of that is that changing the release number from 4.9.3 to 4.9.4 across all targets is a huge endeavor because we have to find all relevant strings in project files, source code, etc. (Recall that we check the version number of the runtime against the generated parser to make sure things line up.) This is complicated by the fact that part of the tool has generated parsers that were generated with the previous version.

Anyway, what I do now defined these versions is something like

find tool runtime -type f -exec grep -l '4\.9' {} \;

and then I manually edit all those files. There are even readme files to update, not to mention documentation.

Anyway, let's start a conversation about how to simplify the release process so that we can do it more often.

A starting suggestion might be something like let's add a string in a comment, according to the kind of file it's in, such as !!ANTLR VERSION!! so that all lines that need to be updated can be found trivially. Once that happens, we can even write a simple shell script to alter those files.

Ping @ericvergnaud @mike-lischke @KvanTTT @jcking @marcospassos et al

[parrt]

BTW, should we do 4.10 not 4.9.4? Lots of changes including java 7->java 8. Ugh. means finding/changing 4.9.4 tag in all github issues.

[KvanTTT]

I thought such an option exists. Have you tried milestone setting or something similar?

[parrt]

ah. yes. found it. sorry for the noise.

[parrt]

Ok, note that I just changed 4.9.4 to 4.10

[parrt]

I will also draft a release note.

[parrt]

changing 4.10 in poms, pushing to master.

[KvanTTT]

Firstly, I have to investigate the presence of duplicated version string and have to think about how to remove them.

[parrt]

issue is they are in a huge number of files and with different formats like string templates, yml, C# project files, 9 different target languages etc.... that's why I'm thinking that maybe a keyword like !!ANTLR VERSION!! in a comment might work.

[parrt]

BTW, I can easily buy a decent Windows box for less than a year's worth of Azure... but then I need to host it somewhere.

[parrt]

in fact I'm using one right next to me now to run Dragon NaturallySpeaking (through VNC to my mac). that is how I am typing everything

[parrt]

Apparently we've missed some file updates. Here is runtime/Python2/src/antlr4_python2_runtime.egg-info/PKG-INFO:

Version: 4.9.2
Summary: ANTLR 4.9.2 runtime for Python 2.7.12

Are comments allowed in there? I can't find out on web.

Looks like we have files like runtime/Cpp/VERSION with just a number; probably can't add a comment there ;)

Maybe I just need to reduce this down to a command that gets just files with version strings to replace then we use sed? So far, I have:

find tool runtime -type f -exec grep -l '4\.9' {} \; | grep -v -E '\.o|\.a|.zip|.gz|.dylib|node_modules/|.class|tests/|CHANGELOG'

Still won't be perfect...we have files like:

class RuntimeMetaDataTests: XCTestCase {
    func testGetMajorMinorVersion() {
        doGetMajorMinorVersionTest("", "")
        doGetMajorMinorVersionTest("4", "4")
        doGetMajorMinorVersionTest("4.", "4.")
        doGetMajorMinorVersionTest("4.7", "4.7") 
        doGetMajorMinorVersionTest("4.7.1", "4.7")
...

[kaby76]

I do this all the time for my stuff, so I have scripts to just do the editing for me, as I can't remember what files need to be edited.

grep . -R -e 4[.]9  2> /dev/null | grep -e .java | grep -v .jar >  ~/oooo

Then, edit your ~/oooo file to include only the lines you want to change and put it somewhere that you check-in, and use that in the following script.

version=4.10
cat ~/oooo | while read pair
do
	file=`echo $pair | sed 's/:.*$//'`
	line="`echo $pair | sed 's/^[^:]*://'`"
	fixed=`echo "$line" | sed 's/[4][.0-9]*/'$version'/'`
	echo "FILE " $file
	echo "LINE " $line
	echo "FIXED " $fixed
	searchEscaped=$(sed 's/[^^]/[&]/g; s/\^/\\^/g' <<<"$line") # escape it.
	sed s%"$searchEscaped"%"$fixed"% -i "$file"
done

You can do a wc for the number of lines to adjust and compare to the size of "oooo", fail if there's something new.

You may have to adjust files you want and files you want to remove, and escaping the replacement string.

[parrt]

problem can be many references that shouldn't be changed...kinda has to be interactive.

[kaby76]

Right, that's whole point of editing "oooo" and checking that in--it's a spec of all the lines in all files in the source that need to be edited with a new release number. The script only changes those specific files listed in that spec, at those specific lines in those files. Nothing outside that list is changed. But you only go through that spec once now rather than grep 4.x in the source, look at those files every time you make a release. To handle any lines in the source that are new or deleted that match 4.9, just keep a grep of all 4.x everywhere, comparing the list of all lines from a previous release with the new release.

[parrt]

yeah, I was thinking about that too. We would set up a well-defined list of files that need to be modified. The problem is some files need to be modified but they also have multiple strings where a simple mechanism will screw up; e.g.,

class RuntimeMetaDataTests: XCTestCase {
    func testGetMajorMinorVersion() {
        doGetMajorMinorVersionTest("", "")
        doGetMajorMinorVersionTest("4", "4")
        doGetMajorMinorVersionTest("4.", "4.")
        doGetMajorMinorVersionTest("4.7", "4.7") 
        doGetMajorMinorVersionTest("4.7.1", "4.7")
...

[ericvergnaud]

I’m sure it’s possible to move these version strings into dedicated files included/loaded at compile time or runtime That would be better practice and make the versioning activity much safer

…

Le 18 janv. 2022 à 20:13, Ken Domino ***@***.***> a écrit : Right, that's whole point of editing "oooo" and checking that in--it's a spec of all the lines in all files in the source that need to be edited with a new release number. The script only changes those specific files listed in that spec, at those specific lines in those files. Nothing outside that list is changed. But you only go through that spec once now rather than grep 4.x in the source, look at those files every time you make a release. To handle any lines in the source that are new or deleted that match 4.9, just keep a grep of all 4.x everywhere, comparing the list of all lines from a previous release with the new release. — Reply to this email directly, view it on GitHub <#3449 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZNQJGXUR3CXW2EB4JNGVLUWW3W3ANCNFSM5LAHCPQQ>. You are receiving this because you were mentioned.

[kaby76]

I’m sure it’s possible to move these version strings into dedicated files included/loaded at compile time or runtime
That would be better practice and make the versioning activity much safer

That doesn't solve the whole problem. Products are more than compiled code. It includes everything in the repo.

For example, the instructions for building a Cpp target parser says to to copy the cmake/ files to your where your grammar lives. But that contains a ref to the version.

antlr4/runtime/Cpp/cmake/FindANTLR.cmake

Line 5 in 828a688

NAMES antlr.jar antlr4.jar antlr-4.jar antlr-4.9.3-complete.jar)

Someone should be updating that version number with the release.

What do you do with a readme.md file that contains a version? https://github.com/antlr/antlr4/blob/master/doc/building-antlr.md#installing-libs-to-mvn-cache-locally https://github.com/antlr/antlr4/blob/master/doc/building-antlr.md#installing-libs-to-mvn-cache-locally Continue to reference 4.5 and 4.6, years later, in the instructions in github?

[parrt]

yep, I would love to move these into a separate version file, but there are project files and so on that must have the fixed version string stuck inside, dang it.

[parrt]

Ok, I have a few more things to get in but I think I will start the process of releasing 4.10.

[parrt]

Howdy. Making some tentative release candidate 1 release notes:

https://github.com/antlr/antlr4/releases/edit/untagged-101fd5d8c58de5f8a9b3

not sure you can view those so here is a PDF 4.10-release-notes.pdf

[KvanTTT]

I'm against of changing the MAJOR version because all grammars are portable unlike grammars of ANTLR3. I'm not sure any versioning is ideal but sometimes it's just subjective feeling when it makes sense to change the version.

[mike-lischke]

Grammars might be portable, but parsers generated before the ATN serialization change and the new runtimes are not.

[kaby76]

Antlr already has a check to disallow a minor version difference between the tool generated code and the Cpp runtime, and it's been there for years. But, how the check is implemented and what it does allow is target dependent.

• Cpp flags a diff for major and minor version numbers. @mike-lischke wrote that code 6 years ago. So, even though ATN serialization has changed, the runtime/tool version check would already be flagged because the release is declared as a minor version change. In other words, it doesn't have to be a major version number change.
• Java flags a diff for major and minor version numbers.
• Dart flags a diff for major and minor version numbers. Note, it has a notion of compiled code versions. I'm not sure I understand how this is different from the prior check.
• Python3 flags a diff for major and minor version numbers.
• Go flags a diff for all version numbers including patch. But, it flags even patch differences!
• JavaScript flags a diff for all version numbers including patch. But, it flags even patch differences!
• CSharp doesn't have the check (just checked that I can use 4.8 generated code with 4.9.3 runtime).

[mike-lischke]

If a check for incompatible changes would be a valid argument against a major version increase, we could just skip using versions altogether and just use a running number increasing every time when a new release comes out. But anyway, I have no strong feelings for either approach. At the end Ter decides.

[parrt]

@kaby76 yeah, i noticed that just this weekend about diff checking :(

For post 4.10 we should clean up.

[parrt]

I've gotta stick with MAJOR 4 I'm afraid. So it'll be 4.10 for upcoming "major" release.

[mike-lischke]

ok

[parrt]

I released ST4.3.2 and updated pom to use that. working on more automation tools for release. getting close.

[mike-lischke]

ST4 is also a lib I want/have to convert to Typescript, since it is used by ANTLR4, but it might be of general value in the JS/TS community.

[parrt]

Sounds good. Check this out https://github.com/jsnyders/StringTemplate-js

[parrt]

antlr3.5.3 is out. almost ready with 4.10

[mike-lischke]

Didn't know you still maintain ANTLR3. There must still be a significant user base to justify that.

[parrt]

Well the biggest issue is the current version depends on it and so I wanted to make sure everything is still built. Also google internal needs a bill from bazel and I now incorporated those files. There are definitely still projects that use the old version but I don’t know if they are actively develop upon.

[parrt]

Cool. I have two new scripts, one that updates the version numbers in antlr repo and another one that copies javadoc and jars to the website repository. we can automate it further later. See https://github.com/antlr/antlr4/tree/dev/scripts

[parrt]

Now I have to figure out how to get dev cleanly into master. Also, I wonder if I try to do the maven release process from dev or master. hmm... seems like it will be simpler if I get everything over to master as that is where we will do our releases then I can merge that back into dev to start the next cycle.

[parrt]

merged! #3642 let's see if works. haha

[parrt]

4.10 is out the door, folks! Thanks for of your efforts!!!
https://github.com/antlr/antlr4/releases/tag/4.10