parse tree to json?

[parrt]

I need to gen json from parse tree for my antlr server. Should we include in antlr runtime? It adds dependency unless we generate json text manually (easy enough). Test currently depend on:

		<dependency>
			<groupId>org.glassfish</groupId>
			<artifactId>javax.json</artifactId>
			<version>1.1.4</version>
			<scope>test</scope>
		</dependency>

No matter what we need field name, structure etc...

Hmm..i'll add to Trees class so I don't break backward compatibility by adding method to ParseTree.

Ok, simple enough; just a tweak to toStringTree(). See:

#3773

JSON output from 1+2 for grammar:

s
	:	expr EOF
	;

expr
	:	INT						# number
	|	expr (MUL | DIV) expr	# multiply
	|	expr (ADD | SUB) expr	# add
	;

looks like:

$ echo '{"s":[{"expr":[{"expr":[{"idx":"0","text":"1"}]},{"idx":"2","text":"+"},{"expr":[{"idx":"4","text":"2"}]}]},{"idx":"5","text":"<EOF>"}]}' | jq
{
  "s": [
    {
      "expr": [
        {
          "expr": [
            {
              "idx": "0",
              "text": "1"
            }
          ]
        },
        {
          "idx": "2",
          "text": "+"
        },
        {
          "expr": [
            {
              "idx": "4",
              "text": "2"
            }
          ]
        }
      ]
    },
    {
      "idx": "5",
      "text": "<EOF>"
    }
  ]
}

[parrt]

Ugh. Somebody should then mimic for other runtimes :(

[KvanTTT]

Similar issue: #233 But I think dependency is not needed for such a straitforward task. Parse tree contains small number of node types. Moreover, JSON should be equal for all runtimes, tests are required.

[parrt]

see my start: #3773

[parrt]

open to suggestions for java 11 string literals to avoid:

		assertEquals("{\"s\":[{\"expr\":[{\"expr\":[{\"idx\":\"0\",\"text\":\"1\"}]},{\"idx\":\"2\",\"text\":\"+\"},{\"expr\":[{\"idx\":\"4\",\"text\":\"2\"}]}]},{\"idx\":\"5\",\"text\":\"<EOF>\"}]}", result);

hahah. resource file? ugh.

[parrt]

Actually, can do:

String expected = "{'s':[{'expr':[{'idx':'0','text':'99'}]},{'idx':'1','text':'<EOF>'}]}";
expected = expected.replace('\'', '"');

[KvanTTT]

Actually, I suggest converting all runtime tests to the descriptor format because they all are related to all runtimes, not to Java one. By the way, I'm working on it, I'll suggest PR with prototype later.

[parrt]

ok. yeah, issue is i only have java impl for this and I need impl now for my server. I'll just leave this PR up/open for use.

[parrt]

@KvanTTT brought up using indexes for token nodes only, no text. makes sense since we need to serialize tokens too. Hmm...how about rulenames? Should they be just rule indexes in json tree? trees look like {"1":[2,3,4]} in this concept instead of {"a":["b","c","d"]}.

[KvanTTT]

If tree is used only for serialization why not. Maybe it makes sense to use an option for compact/full mode. On the other hand, if compactness is very important, a binary format can be used instead of textural json.

[parrt]

Yeah, I guess so. Requires work on receiving end to build viewable tree.

[parrt]

Ok, @KvanTTT now it looks like:

{
  "rules": [
    "s",
    "expr"
  ],
  "input": "99",
  "tokens": [
    {
      "type": 3,
      "line": 1,
      "pos": 0,
      "channel": 0,
      "text": "99"
    },
    {
      "type": -1,
      "line": 1,
      "pos": 2,
      "channel": 0,
      "text": "<EOF>"
    }
  ],
  "tree": {
    "0": [
      {
        "1": [
          0
        ]
      },
      1
    ]
  }
}

[KvanTTT]

Default channel 0 can be omitted (Usually most part of tokens are in 0 channel). Also, offset is more compact compare to line and column.

[parrt]

was thinking position based like [1,2,3,...] w/o field names... yeah, has to be start/stop for text as well.

[parrt]

I just moved stuff to descriptor files for testing. only java works at moment.

[parrt]

This is exploding beyond my initial thoughts haha

[parrt]

Python to load and dump:

import json

with open("/tmp/t.json") as f:
        data = f.read()

data = json.loads(data)
print(data['rules'])
print(data['input'])
for t in data['tokens']:
        print(t)
print(data['tree'])

From parse tree:

(program (def f ( x , y , z ) { (stat x = (expr (expr (expr x) and (expr y)) or (expr z)) ;) (stat (expr (func g ( (expr 30) ))) ;) }) <EOF>)

we get:

['program', 'stat', 'def', 'expr', 'func']
f(x, y, z) { x = x and y or z; g(30); }
{'type': 12, 'line': 1, 'pos': 0, 'channel': 0, 'start': 0, 'stop': 0}
{'type': 3, 'line': 1, 'pos': 1, 'channel': 0, 'start': 1, 'stop': 1}
{'type': 12, 'line': 1, 'pos': 2, 'channel': 0, 'start': 2, 'stop': 2}
{'type': 4, 'line': 1, 'pos': 3, 'channel': 0, 'start': 3, 'stop': 3}
{'type': 12, 'line': 1, 'pos': 5, 'channel': 0, 'start': 5, 'stop': 5}
{'type': 4, 'line': 1, 'pos': 6, 'channel': 0, 'start': 6, 'stop': 6}
{'type': 12, 'line': 1, 'pos': 8, 'channel': 0, 'start': 8, 'stop': 8}
{'type': 5, 'line': 1, 'pos': 9, 'channel': 0, 'start': 9, 'stop': 9}
{'type': 6, 'line': 1, 'pos': 11, 'channel': 0, 'start': 11, 'stop': 11}
{'type': 12, 'line': 1, 'pos': 13, 'channel': 0, 'start': 13, 'stop': 13}
{'type': 1, 'line': 1, 'pos': 15, 'channel': 0, 'start': 15, 'stop': 15}
{'type': 12, 'line': 1, 'pos': 17, 'channel': 0, 'start': 17, 'stop': 17}
{'type': 9, 'line': 1, 'pos': 19, 'channel': 0, 'start': 19, 'stop': 21}
{'type': 12, 'line': 1, 'pos': 23, 'channel': 0, 'start': 23, 'stop': 23}
{'type': 10, 'line': 1, 'pos': 25, 'channel': 0, 'start': 25, 'stop': 26}
{'type': 12, 'line': 1, 'pos': 28, 'channel': 0, 'start': 28, 'stop': 28}
{'type': 2, 'line': 1, 'pos': 29, 'channel': 0, 'start': 29, 'stop': 29}
{'type': 12, 'line': 1, 'pos': 31, 'channel': 0, 'start': 31, 'stop': 31}
{'type': 3, 'line': 1, 'pos': 32, 'channel': 0, 'start': 32, 'stop': 32}
{'type': 11, 'line': 1, 'pos': 33, 'channel': 0, 'start': 33, 'stop': 34}
{'type': 5, 'line': 1, 'pos': 35, 'channel': 0, 'start': 35, 'stop': 35}
{'type': 2, 'line': 1, 'pos': 36, 'channel': 0, 'start': 36, 'stop': 36}
{'type': 7, 'line': 1, 'pos': 38, 'channel': 0, 'start': 38, 'stop': 38}
{'type': -1, 'line': 1, 'pos': 39, 'channel': 0, 'start': 39, 'stop': 38}
{'0': [{'2': [0, 1, 2, 3, 4, 5, 6, 7, 8, {'1': [9, 10, {'3': [{'3': [{'3': [11]}, 12, {'3': [13]}]}, 14, {'3': [15]}]}, 16]}, {'1': [{'3': [{'4': [17, 18, {'3': [19]}, 20]}]}, 21]}, 22]}, 23]}

From input: f(x, y, z) { x = x and y or z; g(30); }

and grammar:

grammar Expr;

program
@after {
    System.out.println(JsonSerializer.toJSON("$ctx"));
}
    : stat EOF
    | def EOF
    ;

stat: ID '=' expr ';'
    | expr ';'
    ;

def : ID '(' ID (',' ID)* ')' '{' stat* '}' ;

expr: ID
    | INT
    | func
    | 'not' expr
    | expr 'and' expr
    | expr 'or' expr
    ;

func : ID '(' expr (',' expr)* ')' ;

INT : [0-9]+ ;
ID: [a-zA-Z_][a-zA-Z_0-9]* ;
WS: [ \t\n\r\f]+ -> skip ;

[KvanTTT]

Is it necessary to include such functionality to ANTLR core now? I don't like the situation when one runtime implements functionality, but other not.

[parrt]

yeah, not necessary; can copy to my antlr4-lab for now.