#4078 [runtime/java] fixed npe due to CommonToken.getText() returning null when inputStream is null #4251

vityaman · 2023-04-30T11:35:24Z

Solution of Issue 4078: #4078

The problem was that CommonToken::getText() returns null if and only if it has InputStream field null, what happens when we call CommonToken(int type) as there source is equal to

protected static final Pair<TokenSource, CharStream> EMPTY_SOURCE =
    new Pair<TokenSource, CharStream>(null, null);

// CommonToken.java

@Override
public String getText() {
	if ( text!=null ) {
		return text;
	}
	CharStream input = getInputStream();
	if ( input==null ) return null; // <-- oops
	int n = input.size();
	if ( start<n && stop<n) {
		return input.getText(Interval.of(start,stop));
	}
	else {
		return "<EOF>";
	}
}
// ...
public String toString(Recognizer<?, ?> r) {
	String channelStr = "";
	if ( channel>0 ) {
		channelStr=",channel="+channel;
	}
	String txt = getText();
	if ( txt!=null ) { // <-- this is what saved us in toString
		txt = txt.replace("\n","\\n");
		txt = txt.replace("\r","\\r");
		txt = txt.replace("\t","\\t");
	}
	else {
		txt = "<no text>";
	}
	String typeString = String.valueOf(type);
	if ( r!=null ) {
		typeString = r.getVocabulary().getDisplayName(type);
	}
	return "[@"+getTokenIndex()+","+start+":"+stop+"='"+txt+"'<"+typeString+">"+channelStr+","+line+":"+getCharPositionInLine()+"]";
}

The badge in the README suggests that ANTLR requires Java 7 or higher, whereas since ANTLR v4.10 Java 11 or higher is required. Signed-off-by: Robert Adam <dev@robert-adam.de> Signed-off-by: Victor Smirnov <vityaman.dev@yandex.ru>

Allow user to subclass and consume differently. Useful for CR handling as suggested in antlr#2519 Signed-off-by: Alberto Simões <Alberto.Simoes@checkmarx.com> Signed-off-by: Victor Smirnov <vityaman.dev@yandex.ru>

…g null when inputStream is null Signed-off-by: Victor Smirnov <vityaman.dev@yandex.ru>

KvanTTT · 2023-04-30T18:57:27Z

README.md

@@ -1,6 +1,6 @@
 # ANTLR v4

-[![Java 7+](https://img.shields.io/badge/java-7+-4c7e9f.svg)](http://java.oracle.com)
+[![Java 11+](https://img.shields.io/badge/java-11+-4c7e9f.svg)](http://java.oracle.com)


it doesn't look a change related to the title of the current pull request.

KvanTTT · 2023-04-30T19:00:30Z

runtime/CSharp/src/Atn/LexerATNSimulator.cs

@@ -779,7 +779,7 @@ public int Column
 		}


-		public void Consume(ICharStream input)
+		public virtual void Consume(ICharStream input)


It relates neither to PR nor to the Java target.

KvanTTT · 2023-04-30T19:14:07Z

runtime/Java/src/org/antlr/v4/runtime/misc/Utils.java

-			if ( c==' ' && escapeSpaces ) buf.append('\u00B7');
-			else if ( c=='\t' ) buf.append("\\t");
-			else if ( c=='\n' ) buf.append("\\n");
-			else if ( c=='\r' ) buf.append("\\r");
+			if (c == ' ' && escapeSpaces) buf.append('\u00B7');
+			else if (c == '\t') buf.append("\\t");
+			else if (c == '\n') buf.append("\\n");
+			else if (c == '\r') buf.append("\\r");


Please don't mix whitespace and semantics changes (ideally, don't change whitespaces without need).

KvanTTT · 2023-04-30T19:14:29Z

runtime/Java/src/org/antlr/v4/runtime/tree/TerminalNodeImpl.java

-			if ( symbol.getType() == Token.EOF ) return "<EOF>";
-			return symbol.getText();
+		if (symbol.getType() == Token.EOF) {
+			return "<EOF>";
+		}
+		return symbol.getText();


Also, please revert this change.

vityaman · 2023-04-30T22:19:24Z

Will close this and resolve issues in PR #4252 as here I have problems with commit history and it is easier to solve them by creating a fresh one.

Krzmbrzl and others added 3 commits April 30, 2023 14:36

README: Update required Java version (antlr#4129)

952c426

The badge in the README suggests that ANTLR requires Java 7 or higher, whereas since ANTLR v4.10 Java 11 or higher is required. Signed-off-by: Robert Adam <dev@robert-adam.de> Signed-off-by: Victor Smirnov <vityaman.dev@yandex.ru>

Update LexerATNSimulator.cs with virutal Consume

dbc0aa7

Allow user to subclass and consume differently. Useful for CR handling as suggested in antlr#2519 Signed-off-by: Alberto Simões <Alberto.Simoes@checkmarx.com> Signed-off-by: Victor Smirnov <vityaman.dev@yandex.ru>

# 4078 [runtime/java] fixed npe due to CommonToken.getText() returnin…

e25f89d

…g null when inputStream is null Signed-off-by: Victor Smirnov <vityaman.dev@yandex.ru>

vityaman force-pushed the 4078-runtime-java-to-string-tree-npe branch from 86e2795 to e25f89d Compare April 30, 2023 11:37

KvanTTT suggested changes Apr 30, 2023

View reviewed changes

vityaman mentioned this pull request Apr 30, 2023

#4078 [runtime/java] made CommonToken::getText return nonnull #4252

Closed

vityaman closed this Apr 30, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

#4078 [runtime/java] fixed npe due to CommonToken.getText() returning null when inputStream is null #4251

#4078 [runtime/java] fixed npe due to CommonToken.getText() returning null when inputStream is null #4251

vityaman commented Apr 30, 2023 •

edited

KvanTTT Apr 30, 2023

KvanTTT Apr 30, 2023

KvanTTT Apr 30, 2023

KvanTTT Apr 30, 2023

vityaman commented Apr 30, 2023

#4078 [runtime/java] fixed npe due to CommonToken.getText() returning null when inputStream is null #4251

#4078 [runtime/java] fixed npe due to CommonToken.getText() returning null when inputStream is null #4251

Conversation

vityaman commented Apr 30, 2023 • edited

KvanTTT Apr 30, 2023

Choose a reason for hiding this comment

KvanTTT Apr 30, 2023

Choose a reason for hiding this comment

KvanTTT Apr 30, 2023

Choose a reason for hiding this comment

KvanTTT Apr 30, 2023

Choose a reason for hiding this comment

vityaman commented Apr 30, 2023

vityaman commented Apr 30, 2023 •

edited