Typed AST Package

This package is a fork of the CPython 2.7 and 3.5 ast modules with the ability to parse PEP 484 type comments. The primary goals of this package are correctness and speed. This package is compatible with Python 3.3 - 3.5.

Current Caveats for Use

• Type comments in invalid locations produce syntax errors.
• When using per-argument function comment annotations, the type comments must come after the argument-separating comma.

Development Notes

General Notes

• token names in Python.asdl need to match token names both in tokenizer.c AND in pgen's copy of tokenizer.c.

Rebuilding Autogenerated Files

• After changing Parser/Python.asdl, you must run:
• python3 Parser/asdl_c.py -h Include/ Parser/Python.asdl
• python3 Parser/asdl_c.py -c Python/ Parser/Python.asdl
• After changing Grammar/Grammar, you must run:
  • pgen Grammar/Grammar Include/graminit.h Python/graminit.c
• To get a working pgen binary: (this is hacky and will be changed)
• get a clean copy of Python 3.5.1
• overwrite Include/token.h, Include/compile.h, and Parser/tokenizer.c
• (or instead of copying Parser/tokenizer.h just modify _PyParser_TokenNames at the top and tokenization to RARROW) with the versions from typed_ast
• In the Python directory, run: ./configure && make
• Parser/pgen (in the Python directory) can now be used to regenerate typed_ast's graminit files.

TODO

• stop aliasing the official Python headers
• ensure we're not using duplicate versions of important standard library functions (like object creation)
• hide most global symbols from being exported to prevent conflicts with other libraries -- changed to a unique prefix
• allow type ignores to be followed by a comment
• prevent type comments in incorrect locations from causing syntax errors
• find a better way to compile pgen
• parse Python 2.7 ASTs
• ast35: ensure compatibility with older Python versions
• Python 3.4 (works on 3.4.4)
• Python 3.3 (works on 3.3.6)
• ast27: ensure compatibility with older Python versions
• Python 3.4
• Python 3.3
• refactor out shared code
  • common functions in typed_ast.c
  • type_ignore array resizing functions in parsetok.c
  • type comment parsing code in tokenizer.c
  • func_type_input parsing in ast.c