Module load fails with Python 3.9.8 (undefined symbol _PyUnicode_DecodeUnicodeEscape)

[rra]

Attempting to import the module as installed from PyPI (x86_64 architecture) on Python 3.9.8 produces the following exception:

Traceback (most recent call last):
  File "/home/runner/.cache/pre-commit/repo8e93dzwn/py_env-python3/bin/black", line 5, in <module>
    from black import patched_main
  File "/home/runner/.cache/pre-commit/repo8e93dzwn/py_env-python3/lib/python3.9/site-packages/black/__init__.py", line 52, in <module>
    from typed_ast import ast3, ast27
  File "/home/runner/.cache/pre-commit/repo8e93dzwn/py_env-python3/lib/python3.9/site-packages/typed_ast/ast3.py", line 40, in <module>
    from typed_ast import _ast3
ImportError: /home/runner/.cache/pre-commit/repo8e93dzwn/py_env-python3/lib/python3.9/site-packages/typed_ast/_ast3.cpython-39-x86_64-linux-gnu.so: undefined symbol: _PyUnicode_DecodeUnicodeEscape

This appears to have broken between Python 3.9.7 and Python 3.9.8.

[rra]

For others who may find this in a search, I ran into this problem via black because I had black pinned to an older version. The current version of black appears to no longer use typed-ast and thus won't encounter this issue.

[hroncok]

diff --git a/ast3/Python/ast.c b/ast3/Python/ast.c
index cfca73f..93da89c 100644
--- a/ast3/Python/ast.c
+++ b/ast3/Python/ast.c
@@ -56,6 +56,8 @@ _PyBytes_DecodeEscape(const char *s,
     return PyBytes_DecodeEscape(s, len, errors, unicode, recode_encoding);
 }
 
+#endif
+
 PyObject *
 _PyUnicode_DecodeUnicodeEscape(const char *s,
                                Py_ssize_t size,
@@ -66,7 +68,6 @@ _PyUnicode_DecodeUnicodeEscape(const char *s,
     return PyUnicode_DecodeUnicodeEscape(s, size, errors);
 }
 
-#endif
 
 static int validate_stmts(asdl_seq *);
 static int validate_exprs(asdl_seq *, expr_context_ty, int);

This seems to get the job done.

[anthrotype]

I actually encountered this problem not while running black, but while running pytype. So what's the fix? Is there one?

[rra]

It looks like both mypy and black previously used typed-ast and no longer do. black at least (I haven't checked mypy) has switched to using the ast module included in Python for Python 3.8 and later. I suspect that's also the correct fix for pytype given #170.

[zerocewl]

The last workaround is using python@3.9.7...

[hauntsaninja]

Thanks for reporting, @hroncok are you willing to make a PR?

[hroncok]

Here you go #171

[hauntsaninja]

Specifically, when running on Python 3.8 and later, black / mypy will only use typed-ast to check / format Python 2 code. Recent releases of black / mypy have made Python 2 support an "extra", so they no longer install typed-ast on Python 3.8+ unless you do e.g. pip install "mypy[python2]".

I'm not sure what exactly pytype's usage of typed-ast looks like, but I imagine it could do something similar. Later today I'll make a release with a fix (thanks to hroncok!), in which case the solution will be to install that newly released typed-ast.

[anthrotype]

Many thanks for the quick fix! I have no idea what exactly pytype does with typed_ast, hopefully they will be able to sort it out like black and mypy have.

[steve-mavens]

Thanks for the fix! Do you have an ETA on when a release with this change might hit pypi? Not asking for any commitment, but I'm currently deciding whether or not to move to the latest black. If you're likely to release soon then I need not bother! :-)

[srittau]

I believe @hauntsaninja wanted to do a release. If they have other commitments, I will see whether I can do a release today.

[steve-mavens]

That's fantastic, thank you. I've migrated one repo to the latest black, but this means I can leave the rest alone, since I don't think we need to do any other 3.9 builds imminently.

[hauntsaninja]

Released typed-ast 1.5.0 which includes this fix