Build failure installing on Python 3.10

[jaraco]

This looks like a possible repeat of #146.

In pypa/setuptools#2597, I'm attempting to run Setuptools tests on Python 3.10 for the first time, but am unable to.

It looks like the issue is that (a) there are no pre-built binaries of typed_ast for Python 3.10 on macOS and (b) attempting to build from source stumbles into an error.

setuptools HEAD $ python3.10 -m pip-run typed_ast
Collecting typed_ast
  Using cached typed_ast-1.4.2.tar.gz (208 kB)
Using legacy 'setup.py install' for typed-ast, since package 'wheel' is not installed.
Installing collected packages: typed-ast
    Running setup.py install for typed-ast ... -    ERROR: Command errored out with exit status 1:
     command: /usr/local/bin/python3.10 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-install-xfkneiny/typed-ast_54fd239ecaeb4bfa94a09e250f4efa15/setup.py'"'"'; __file__='"'"'/private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-install-xfkneiny/typed-ast_54fd239ecaeb4bfa94a09e250f4efa15/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-record-j9896vcq/install-record.txt --single-version-externally-managed --home /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-target-j7qbo_ex --compile --install-headers /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-target-j7qbo_ex/include/python/typed-ast
         cwd: /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-install-xfkneiny/typed-ast_54fd239ecaeb4bfa94a09e250f4efa15/
error
    Complete output (34 lines):
    running install
    running build
    running build_py
    creating build
    creating build/lib.macosx-10.9-universal2-3.10
    creating build/lib.macosx-10.9-universal2-3.10/typed_ast
    copying typed_ast/conversions.py -> build/lib.macosx-10.9-universal2-3.10/typed_ast
    copying typed_ast/__init__.py -> build/lib.macosx-10.9-universal2-3.10/typed_ast
    copying typed_ast/ast3.py -> build/lib.macosx-10.9-universal2-3.10/typed_ast
    copying typed_ast/ast27.py -> build/lib.macosx-10.9-universal2-3.10/typed_ast
    package init file 'ast3/tests/__init__.py' not found (or not a regular file)
    creating build/lib.macosx-10.9-universal2-3.10/typed_ast/tests
    copying ast3/tests/test_basics.py -> build/lib.macosx-10.9-universal2-3.10/typed_ast/tests
    running build_ext
    building '_ast27' extension
    creating build/temp.macosx-10.9-universal2-3.10
    creating build/temp.macosx-10.9-universal2-3.10/ast27
    creating build/temp.macosx-10.9-universal2-3.10/ast27/Custom
    creating build/temp.macosx-10.9-universal2-3.10/ast27/Parser
    creating build/temp.macosx-10.9-universal2-3.10/ast27/Python
    gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch arm64 -arch x86_64 -g -Iast27/Include -I/Library/Frameworks/Python.framework/Versions/3.10/include/python3.10 -c ast27/Custom/typed_ast.c -o build/temp.macosx-10.9-universal2-3.10/ast27/Custom/typed_ast.o
    gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch arm64 -arch x86_64 -g -Iast27/Include -I/Library/Frameworks/Python.framework/Versions/3.10/include/python3.10 -c ast27/Parser/acceler.c -o build/temp.macosx-10.9-universal2-3.10/ast27/Parser/acceler.o
    gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch arm64 -arch x86_64 -g -Iast27/Include -I/Library/Frameworks/Python.framework/Versions/3.10/include/python3.10 -c ast27/Parser/bitset.c -o build/temp.macosx-10.9-universal2-3.10/ast27/Parser/bitset.o
    gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch arm64 -arch x86_64 -g -Iast27/Include -I/Library/Frameworks/Python.framework/Versions/3.10/include/python3.10 -c ast27/Parser/grammar.c -o build/temp.macosx-10.9-universal2-3.10/ast27/Parser/grammar.o
    gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch arm64 -arch x86_64 -g -Iast27/Include -I/Library/Frameworks/Python.framework/Versions/3.10/include/python3.10 -c ast27/Parser/grammar1.c -o build/temp.macosx-10.9-universal2-3.10/ast27/Parser/grammar1.o
    gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch arm64 -arch x86_64 -g -Iast27/Include -I/Library/Frameworks/Python.framework/Versions/3.10/include/python3.10 -c ast27/Parser/node.c -o build/temp.macosx-10.9-universal2-3.10/ast27/Parser/node.o
    gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch arm64 -arch x86_64 -g -Iast27/Include -I/Library/Frameworks/Python.framework/Versions/3.10/include/python3.10 -c ast27/Parser/parser.c -o build/temp.macosx-10.9-universal2-3.10/ast27/Parser/parser.o
    gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch arm64 -arch x86_64 -g -Iast27/Include -I/Library/Frameworks/Python.framework/Versions/3.10/include/python3.10 -c ast27/Parser/parsetok.c -o build/temp.macosx-10.9-universal2-3.10/ast27/Parser/parsetok.o
    gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch arm64 -arch x86_64 -g -Iast27/Include -I/Library/Frameworks/Python.framework/Versions/3.10/include/python3.10 -c ast27/Parser/tokenizer.c -o build/temp.macosx-10.9-universal2-3.10/ast27/Parser/tokenizer.o
    ast27/Parser/tokenizer.c:19:10: fatal error: 'pydebug.h' file not found
    #include "pydebug.h"
             ^~~~~~~~~~~
    1 error generated.
    error: command '/usr/bin/gcc' failed with exit code 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/local/bin/python3.10 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-install-xfkneiny/typed-ast_54fd239ecaeb4bfa94a09e250f4efa15/setup.py'"'"'; __file__='"'"'/private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-install-xfkneiny/typed-ast_54fd239ecaeb4bfa94a09e250f4efa15/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-record-j9896vcq/install-record.txt --single-version-externally-managed --home /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-target-j7qbo_ex --compile --install-headers /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-target-j7qbo_ex/include/python/typed-ast Check the logs for full command output.
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pip-run.py", line 4, in <module>
    __name__ == '__main__' and run()
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pip_run/__init__.py", line 18, in run
    with deps.load(*deps.not_installed(pip_args)) as home:
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/contextlib.py", line 133, in __enter__
    return next(self.gen)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pip_run/deps.py", line 50, in load
    _installable(args) and subprocess.check_call(cmd)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py", line 378, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '('/usr/local/bin/python3.10', '-m', 'pip', 'install', '-t', '/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-43tg8hwm', 'typed_ast')' returned non-zero exit status 1.

It's unclear to me why Python is requiring this module if it's only supported on Python 3.7 and older. It's unclear from that message if the package is unsupported on those Pythons or some (all) of its behavior is only meant for Python 3.7 and earlier ASTs. If this package is not intended to be installed on Python 3.8 or later, the package should probably declare a Requires-Python directive to signal that non-support.

I suspect this issue will likely be addressed by tracking down what's requiring typed_ast and why and to limit its dependency on later Pythons. In the meantime, I wanted to capture this issue here as the root cause.

[jaraco]

I see that mypy unconditionally requires this library, which means that mypy is non-viable on Python 3.10 without hand-holding unless some binaries are released.

[jaraco]

Looks like #155 may address the build failure.

[gvanrossum]

I see that mypy unconditionally requires this library, which means that mypy is non-viable on Python 3.10 without hand-holding unless some binaries are released.

That's because mypy still wants to support parsing Python 2, which requires the ast27 package included herein. I'm working on #155.

[hauntsaninja]

(re mypy specifically, see python/mypy#10124)

[jaraco]

Thanks @gvanrossum for the quick fix. I'd missed it earlier, but confirmed now the fix is working.