TypeScript parsing bug and places to find spec

[ppppqp]

Hi, I was using ANTLR (Cpp target) to tokenize and parse typescript scripts. I found that the parser can report "seg fault" when parsing certain syntax, for example, let a = 1 + 2;.

My question is, is this bug expected? i.e. is typescript syntax not fully supported by ANTLR Cpp target?

I looked into the source code further and I found the error is caused by getInterpreter<atn::ParserATNSimulator>()->adaptivePredict() with a specific decision parameter value. However, the official runtime spec does not include the detailed meaning of this function and the parameter. Just wondering if there's a better place to get the information.

I am new to ANTLR and still making sense of how it works. Any general suggestions/resources are also welcomed.

[kaby76]

There's actually a couple of problems.

First, the port of the typescript grammar for Cpp has several errors due to faulty translation from one of the other ports to C++.

• https://github.com/antlr/grammars-v4/blob/0042cb5f6f08fed39954f5dd971d5ec1ed7686d6/javascript/typescript/Cpp/TypeScriptLexerBase.cpp#L6 should have an initialization of lastToken = false;. Call me old school (almost >40 years with C, working with C++ since it was invented), but I don't trust the C++ runtime to initialize this correctly (and I don't want to wade through to 1000-page Language Spec to see if it's supposed to init bools to false).
• The test of lastToken at https://github.com/antlr/grammars-v4/blob/0042cb5f6f08fed39954f5dd971d5ec1ed7686d6/javascript/typescript/Cpp/TypeScriptLexerBase.cpp#L60 is logically reversed. lastToken is a struct, not a pointer as in all the other ports. In C#, the test is _lastToken == null. If it is not null, it is okay to then test _lastToken.Type == OpenBrace. If you look at the assignment of lastToken/lastTokenType, the value of lastTokenType is valid if and only if lastToken is true; in C#, lastToken.lastTokeType can only be dereferenced if lastToken is not null. So the test is wrong. It should be:

    if (!lastToken || lastTokenType == TypeScriptLexer::OpenBrace)

• The code at line 75 is similarly wrong. https://github.com/antlr/grammars-v4/blob/0042cb5f6f08fed39954f5dd971d5ec1ed7686d6/javascript/typescript/Cpp/TypeScriptLexerBase.cpp#L75. lastTokenType can only be tested if lastToken is true.

If you don't believe me on these errors, output the token types to verify all tests are working across all targets.

BTW, this seems to be the same problem I described with another grammar which was forked from this grammar or the other way around. antlr/grammars-v4#2909 (comment)

Any time you see problems with a port, you should first suspect the port is in error, not the Antlr code--although that can happen!

And, in this case, yes, there is a problem there too. (Fix the port errors first, then run.) The program will crash in prediction context comparisons, which I fixed a few months ago. antlr/grammars-v4#2909 (comment)

 	Test.exe!antlr4::atn::operator==(const antlr4::atn::PredictionContext & lhs, const antlr4::atn::PredictionContext & rhs) Line 206	C++
 	Test.exe!`anonymous namespace'::predictionContextEqual(const std::shared_ptr<antlr4::atn::PredictionContext const> & lhs, const std::shared_ptr<antlr4::atn::PredictionContext const> & rhs) Line 26	C++
 	Test.exe!std::equal<std::_Vector_const_iterator<std::_Vector_val<std::_Simple_types<std::shared_ptr<antlr4::atn::PredictionContext const>>>>,std::_Vector_const_iterator<std::_Vector_val<std::_Simple_types<std::shared_ptr<antlr4::atn::PredictionContext const>>>>,bool (__cdecl*)(std::shared_ptr<antlr4::atn::PredictionContext const> const &,std::shared_ptr<antlr4::atn::PredictionContext const> const &)>(const std::_Vector_const_iterator<std::_Vector_val<std::_Simple_types<std::shared_ptr<antlr4::atn::PredictionContext const>>>> _First1, const std::_Vector_const_iterator<std::_Vector_val<std::_Simple_types<std::shared_ptr<antlr4::atn::PredictionContext const>>>> _Last1, const std::_Vector_const_iterator<std::_Vector_val<std::_Simple_types<std::shared_ptr<antlr4::atn::PredictionContext const>>>> _First2, bool(*)(const std::shared_ptr<antlr4::atn::PredictionContext const> &, const std::shared_ptr<antlr4::atn::PredictionContext const> &) _Pred) Line 4588	C++
>	Test.exe!antlr4::atn::ArrayPredictionContext::equals(const antlr4::atn::PredictionContext & other) Line 78	C++
 	Test.exe!antlr4::atn::operator==(const antlr4::atn::PredictionContext & lhs, const antlr4::atn::PredictionContext & rhs) Line 207	C++
 	Test.exe!antlr4::atn::PredictionContext::mergeArrays(std::shared_ptr<antlr4::atn::ArrayPredictionContext const> a, std::shared_ptr<antlr4::atn::ArrayPredictionContext const> b, bool rootIsWildcard, antlr4::atn::PredictionContextMergeCache * mergeCache) Line 411	C++
 	Test.exe!antlr4::atn::PredictionContext::merge(std::shared_ptr<antlr4::atn::PredictionContext const> a, std::shared_ptr<antlr4::atn::PredictionContext const> b, bool rootIsWildcard, antlr4::atn::PredictionContextMergeCache * mergeCache) Line 210	C++
 	Test.exe!antlr4::atn::ATNConfigSet::add(const std::shared_ptr<antlr4::atn::ATNConfig> & config, antlr4::atn::PredictionContextMergeCache * mergeCache) Line 64	C++
 	Test.exe!antlr4::atn::ATNConfigSet::add(const std::shared_ptr<antlr4::atn::ATNConfig> & config) Line 38	C++
 	Test.exe!antlr4::atn::PredictionModeClass::hasSLLConflictTerminatingPrediction(antlr4::atn::PredictionMode mode, antlr4::atn::ATNConfigSet * configs) Line 66	C++
 	Test.exe!antlr4::atn::ParserATNSimulator::computeTargetState(antlr4::dfa::DFA & dfa, antlr4::dfa::DFAState * previousD, unsigned __int64 t) Line 298	C++
 	Test.exe!antlr4::atn::ParserATNSimulator::execATN(antlr4::dfa::DFA & dfa, antlr4::dfa::DFAState * s0, antlr4::TokenStream * input, unsigned __int64 startIndex, antlr4::ParserRuleContext * outerContext) Line 179	C++
 	Test.exe!antlr4::atn::ParserATNSimulator::adaptivePredict(antlr4::TokenStream * input, unsigned __int64 decision, antlr4::ParserRuleContext * outerContext) Line 155	C++
 	Test.exe!TypeScriptParser::singleExpression(int precedence) Line 12903	C++
 	Test.exe!TypeScriptParser::expressionSequence() Line 11772	C++
 	Test.exe!TypeScriptParser::expressionStatement() Line 7118	C++
 	Test.exe!TypeScriptParser::statement() Line 6006	C++
 	Test.exe!TypeScriptParser::sourceElement() Line 5654	C++
 	Test.exe!TypeScriptParser::sourceElements() Line 10620	C++
 	Test.exe!TypeScriptParser::functionBody() Line 10560	C++
 	Test.exe!TypeScriptParser::constructorDeclaration() Line 4502	C++
 	Test.exe!TypeScriptParser::classElement() Line 9254	C++
 	Test.exe!TypeScriptParser::classTail() Line 9089	C++
 	Test.exe!TypeScriptParser::classDeclaration() Line 8964	C++
 	Test.exe!TypeScriptParser::statement() Line 5859	C++
 	Test.exe!TypeScriptParser::sourceElement() Line 5654	C++
 	Test.exe!TypeScriptParser::sourceElements() Line 10620	C++
 	Test.exe!TypeScriptParser::program() Line 5594	C++
 	Test.exe!DoParse(antlr4::CharStream * str, std::string input_name, int row_number) Line 87	C++
 	Test.exe!ParseFilename(std::string input, int row_number) Line 148	C++
 	Test.exe!TryParse(std::vector<std::string,std::allocator<std::string>> & args) Line 217	C++
 	Test.exe!main(int argc, const char * * argv) Line 235	C++
 	Test.exe!invoke_main() Line 79	C++
 	Test.exe!__scrt_common_main_seh() Line 288	C++
 	Test.exe!__scrt_common_main() Line 331	C++
 	Test.exe!mainCRTStartup(void * __formal) Line 17	C++
 	kernel32.dll!BaseThreadInitThunk()	Unknown
 	ntdll.dll!RtlUserThreadStart�()	Unknown

This won't be in 4.11.1. You'll have to get a copy of the latest "dev" branch of the Cpp runtime. I just tested with the new code, and it parses fine.