improvement: Add AST subclass constructors #11880

bzoracler · 2024-05-09T12:10:49Z

Fixes #8378

ast.AST no longer owns attributes which may not exist on all subclasses (node locations, type_comment). These are all shifted to subclasses which actually own the attributes (d5e0838, 75a2e8e, f9a2cb9)
ast.AST no longer has the constructor __init__(self, *args: Any, **kwargs: Any).
All subclasses of ast.AST which have a non-empty _fields: ClassVar[tuple[str, ...]] or _attributes: ClassVar[tuple[str, ...]] now have their own __init__ constructor (d886a45). The constructor has:
- Positional-or-keyword arguments in the order in which they appear in _fields, and
- Non-mandatory keyword-only arguments from _attributes. These are expressed using a combination of typing.TypedDict and typing_extensions.Unpack.
  - These are non-mandatory because I expect most users won't pass explicit locations, but rather use something like ast.fix_missing_locations.

Updates:

Arguments of type ast.expr_context are not required to be passed to the constructor, but they will not exist on node instances for Python < 3.13.
Arguments of type list are not required to be passed to the constructor for Python >= 3.13.

improvement: shift `type_comment` attribute to classes which contain it

stdlib/_ast.pyi

JelleZijlstra · 2024-05-09T13:33:21Z

Thanks for this, I had been thinking of adding it but hadn't gotten around to it.

JelleZijlstra · 2024-05-09T15:56:58Z

stdlib/_ast.pyi


 class Module(mod):
    if sys.version_info >= (3, 10):
        __match_args__ = ("body", "type_ignores")
    body: list[stmt]
    type_ignores: list[TypeIgnore]
+    def __init__(self, body: list[stmt], type_ignores: list[TypeIgnore]) -> None: ...


These list fields should default to the empty list on 3.13+.

Addressed in c82a5fa

JelleZijlstra · 2024-05-09T16:00:02Z

stdlib/_ast.pyi

+            decorator_list: list[expr],
+            returns: expr | None = None,
+            *,
+            type_comment: str | None = None,


I don't think type_comment needs to be keyword-only on this overload.

Addressed in 9655775

…o constructor

bzoracler · 2024-05-10T06:04:06Z

I didn't add these other changes from Python 3.13 runtime to this PR:

AST._field_types - not sure what the intention is for the values of these dictionaries; are they meant to be for runtime introspection, or can they be anything?
The default_value parameter for ast.TypeVar, ast.ParamSpec, ast.TypeVarTuple - the runtime suggests that this defaults to None if not passed, but the grammar does not mark this field as optional. (I misread the grammar, they're definitely optional)

JelleZijlstra · 2024-05-10T12:50:59Z

AST._field_types - not sure what the intention is for the values of these dictionaries; are they meant to be for runtime introspection, or can they be anything?

It's used by the constructor of AST nodes to figure out what defaults to use. I did document it, so I think we should add it to the ast.AST class as ClassVar[dict[str, Any]].

github-actions · 2024-05-10T15:52:14Z

Diff from mypy_primer, showing the effect of this PR on open source code:

pyp (https://github.com/hauntsaninja/pyp)
+ pyp.py:585: error: "AST" has no attribute "lineno"  [attr-defined]
+ pyp.py:586: error: "AST" has no attribute "end_lineno"  [attr-defined]

sphinx (https://github.com/sphinx-doc/sphinx)
+ sphinx/ext/autodoc/preserve_defaults.py: note: In function "get_default_value":
+ sphinx/ext/autodoc/preserve_defaults.py:114:12: error: "AST" has no attribute "lineno"  [attr-defined]
+ sphinx/ext/autodoc/preserve_defaults.py:114:31: error: "AST" has no attribute "end_lineno"  [attr-defined]
+ sphinx/ext/autodoc/preserve_defaults.py:115:26: error: "AST" has no attribute "lineno"  [attr-defined]
+ sphinx/ext/autodoc/preserve_defaults.py:116:25: error: "AST" has no attribute "col_offset"  [attr-defined]
+ sphinx/ext/autodoc/preserve_defaults.py:116:45: error: "AST" has no attribute "end_col_offset"  [attr-defined]

mypy (https://github.com/python/mypy)
+ mypy/fastparse.py:1998: error: "Index" has no attribute "col_offset"  [attr-defined]
+ mypy/fastparse.py:2001: error: "Slice" has no attribute "col_offset"  [attr-defined]
+ mypy/fastparse.py:2002: error: Argument 1 to "Tuple" has incompatible type "List[slice]"; expected "List[expr]"  [arg-type]

pytest (https://github.com/pytest-dev/pytest)
+ src/_pytest/assertion/rewrite.py:839: error: Argument 1 to "Dict" has incompatible type "list[Constant]"; expected "list[expr | None]"  [arg-type]
+ src/_pytest/assertion/rewrite.py:839: note: "List" is invariant -- see https://mypy.readthedocs.io/en/stable/common_issues.html#variance
+ src/_pytest/assertion/rewrite.py:839: note: Consider using "Sequence" instead, which is covariant
+ src/_pytest/assertion/rewrite.py:932: error: Argument 3 to "If" has incompatible type "list[If]"; expected "list[stmt]"  [arg-type]
+ src/_pytest/assertion/rewrite.py:932: note: "List" is invariant -- see https://mypy.readthedocs.io/en/stable/common_issues.html#variance
+ src/_pytest/assertion/rewrite.py:932: note: Consider using "Sequence" instead, which is covariant
+ src/_pytest/assertion/rewrite.py:938: error: Argument 1 to "Assign" has incompatible type "list[Name]"; expected "list[expr]"  [arg-type]
+ src/_pytest/assertion/rewrite.py:938: note: "List" is invariant -- see https://mypy.readthedocs.io/en/stable/common_issues.html#variance
+ src/_pytest/assertion/rewrite.py:938: note: Consider using "Sequence" instead, which is covariant
+ src/_pytest/assertion/rewrite.py:963: error: Argument 1 to "Assign" has incompatible type "list[Name]"; expected "list[expr]"  [arg-type]
+ src/_pytest/assertion/rewrite.py:963: note: "List" is invariant -- see https://mypy.readthedocs.io/en/stable/common_issues.html#variance
+ src/_pytest/assertion/rewrite.py:963: note: Consider using "Sequence" instead, which is covariant
+ src/_pytest/assertion/rewrite.py:1145: error: Argument 1 to "Tuple" has incompatible type "list[Constant]"; expected "list[expr]"  [arg-type]
+ src/_pytest/assertion/rewrite.py:1145: note: "List" is invariant -- see https://mypy.readthedocs.io/en/stable/common_issues.html#variance
+ src/_pytest/assertion/rewrite.py:1145: note: Consider using "Sequence" instead, which is covariant
+ src/_pytest/assertion/rewrite.py:1146: error: Argument 1 to "Tuple" has incompatible type "list[Name]"; expected "list[expr]"  [arg-type]
+ src/_pytest/assertion/rewrite.py:1146: note: "List" is invariant -- see https://mypy.readthedocs.io/en/stable/common_issues.html#variance
+ src/_pytest/assertion/rewrite.py:1146: note: Consider using "Sequence" instead, which is covariant
+ src/_pytest/assertion/rewrite.py:1147: error: Argument 1 to "Tuple" has incompatible type "list[Constant]"; expected "list[expr]"  [arg-type]
+ src/_pytest/assertion/rewrite.py:1147: note: "List" is invariant -- see https://mypy.readthedocs.io/en/stable/common_issues.html#variance
+ src/_pytest/assertion/rewrite.py:1147: note: Consider using "Sequence" instead, which is covariant
+ src/_pytest/assertion/rewrite.py:1151: error: Argument 2 to "BoolOp" has incompatible type "list[Name]"; expected "list[expr]"  [arg-type]
+ src/_pytest/assertion/rewrite.py:1151: note: "List" is invariant -- see https://mypy.readthedocs.io/en/stable/common_issues.html#variance
+ src/_pytest/assertion/rewrite.py:1151: note: Consider using "Sequence" instead, which is covariant
+ testing/test_assertrewrite.py:133: error: "AST" has no attribute "lineno"  [attr-defined]
+ testing/test_assertrewrite.py:134: error: "AST" has no attribute "col_offset"  [attr-defined]
+ testing/test_assertrewrite.py:135: error: "AST" has no attribute "end_lineno"  [attr-defined]
+ testing/test_assertrewrite.py:136: error: "AST" has no attribute "end_col_offset"  [attr-defined]

werkzeug (https://github.com/pallets/werkzeug)
+ src/werkzeug/routing/rules.py:785: error: Argument 1 to "JoinedStr" has incompatible type "list[AST]"; expected "list[expr]"  [arg-type]
+ src/werkzeug/routing/rules.py:788: error: List item 0 has incompatible type "AST"; expected "expr"  [list-item]
+ src/werkzeug/routing/rules.py:788: error: List item 1 has incompatible type "AST"; expected "expr"  [list-item]
+ src/werkzeug/routing/rules.py:818: error: "AST" has no attribute "lineno"  [attr-defined]
+ src/werkzeug/routing/rules.py:820: error: "AST" has no attribute "end_lineno"  [attr-defined]
+ src/werkzeug/routing/rules.py:820: error: "AST" has no attribute "lineno"  [attr-defined]
+ src/werkzeug/routing/rules.py:822: error: "AST" has no attribute "col_offset"  [attr-defined]
+ src/werkzeug/routing/rules.py:824: error: "AST" has no attribute "end_col_offset"  [attr-defined]
+ src/werkzeug/routing/rules.py:824: error: "AST" has no attribute "col_offset"  [attr-defined]

ibis (https://github.com/ibis-project/ibis)
+ ibis/backends/bigquery/udf/core.py:66: error: Missing positional arguments "args", "keywords" in call to "Call"  [call-arg]
+ ibis/backends/bigquery/udf/core.py:79: error: Missing positional arguments "args", "keywords" in call to "Call"  [call-arg]
+ ibis/backends/bigquery/udf/core.py:85: error: Missing positional arguments "args", "keywords" in call to "Call"  [call-arg]
+ ibis/backends/bigquery/udf/core.py:85: error: Missing positional argument "value" in call to "Attribute"  [call-arg]
+ ibis/backends/bigquery/udf/core.py:95: error: Missing positional arguments "args", "keywords" in call to "Call"  [call-arg]

flake8-pyi (https://github.com/PyCQA/flake8-pyi)
+ pyi.py:2249: error: "AST" has no attribute "lineno"  [attr-defined]
+ pyi.py:2249: error: "AST" has no attribute "col_offset"  [attr-defined]

streamlit (https://github.com/streamlit/streamlit)
+ lib/streamlit/runtime/scriptrunner/magic.py: note: In function "_build_st_write_call":
+ lib/streamlit/runtime/scriptrunner/magic.py:166:12: error: Unexpected keyword argument "kwargs" for "Call"; did you mean "args"?  [call-arg]
+ note: "Call" defined here
+ lib/streamlit/runtime/scriptrunner/magic.py:166:12: error: Unexpected keyword argument "starargs" for "Call"  [call-arg]
+ note: "Call" defined here

JelleZijlstra

Thank you!

python/typeshed#11880 correctly points out that not all AST nodes have a line and column number.

python/typeshed#11880 adds more precise types for AST nodes. I'm submitting some changes to adapt pytest to these changes.

python/typeshed#11880 correctly points out that not all AST nodes have a line and column number.

Adapting to changes in python/typeshed#11880. This mostly adds more precise types for individual pieces of AST.

Followup from python/typeshed#11880. Not all AST nodes have a lineno; in this file only parameter defaults are passed to this function, which are always ast.expr.

python/typeshed#11880 adds more precise types for AST nodes. I'm submitting some changes to adapt pytest to these changes.

See https://docs.python.org/3.10/library/ast.html#abstract-grammar for a reference on the attributes `ast.Call` takes. A future version of mypy will give a type error for this code (python/typeshed#11880).

bzoracler added 5 commits May 9, 2024 23:56

improvement: shift node location attributes to classes which have them

d5e0838

Update _ast.pyi

75a2e8e

improvement: shift `type_comment` attribute to classes which contain it

improvement: add missing node field

f9a2cb9

improvement: add precise AST subclass constructors

d886a45

test: update allowlist

25d4dad

This comment has been minimized.

Sign in to view

bzoracler added 2 commits May 10, 2024 00:33

fix: delegate version-specific nodes to separate allowlists

70fdb52

fix: typo in parameter

c88c4aa

AlexWaygood reviewed May 9, 2024

View reviewed changes

stdlib/_ast.pyi Outdated Show resolved Hide resolved

AlexWaygood reviewed May 9, 2024

View reviewed changes

stdlib/_ast.pyi Outdated Show resolved Hide resolved

This comment has been minimized.

Sign in to view

style: Shorten signatures

1135b28

AlexWaygood mentioned this pull request May 9, 2024

Allow using typing_extensions.TypeVar in stubs PyCQA/flake8-pyi#486

Merged

This comment has been minimized.

Sign in to view

JelleZijlstra mentioned this pull request May 9, 2024

Default expr_context parameters in ast to Load() python/cpython#118851

Closed

JelleZijlstra reviewed May 9, 2024

View reviewed changes

bzoracler and others added 5 commits May 10, 2024 17:34

fix: remove unnecessary keyword-only constraint

9655775

improvement: don't require args of type expr_context to be passed t…

bb880f2

…o constructor

feat: empty list defaults in Python 3.13

c82a5fa

style: keep _SliceAttributes together with usage location

aea4c01

[pre-commit.ci] auto fixes from pre-commit.com hooks

1e489b1

This comment has been minimized.

Sign in to view

fix: add missing default_value field for Python 3.13

b82fc4e

This comment has been minimized.

Sign in to view

improvement: add new Python 3.13 class variable

cc50db4

This comment has been minimized.

Sign in to view

fix: wrong attribute conditional location

cc2cd65

JelleZijlstra approved these changes May 18, 2024

View reviewed changes

JelleZijlstra merged commit 5bd7150 into python:main May 18, 2024
62 checks passed

JelleZijlstra added a commit to PyCQA/flake8-pyi that referenced this pull request May 18, 2024

Fix new type error

a0370f1

python/typeshed#11880 correctly points out that not all AST nodes have a line and column number.

JelleZijlstra mentioned this pull request May 18, 2024

Fix new type error PyCQA/flake8-pyi#489

Merged

JelleZijlstra added a commit to JelleZijlstra/pytest that referenced this pull request May 18, 2024

Fix new typing issues in AST code

8bb493b

python/typeshed#11880 adds more precise types for AST nodes. I'm submitting some changes to adapt pytest to these changes.

JelleZijlstra mentioned this pull request May 18, 2024

Fix new typing issues in AST code pytest-dev/pytest#12337

Merged

AlexWaygood pushed a commit to PyCQA/flake8-pyi that referenced this pull request May 18, 2024

Fix new type error (#489)

f1230fc

python/typeshed#11880 correctly points out that not all AST nodes have a line and column number.

JelleZijlstra added a commit to JelleZijlstra/werkzeug that referenced this pull request May 18, 2024

Improve types for AST handling

8a672ef

Adapting to changes in python/typeshed#11880. This mostly adds more precise types for individual pieces of AST.

JelleZijlstra mentioned this pull request May 18, 2024

Improve types for AST handling pallets/werkzeug#2905

Open

JelleZijlstra mentioned this pull request May 18, 2024

More precise types in preserve_defaults.py sphinx-doc/sphinx#12385

Merged

bluetech pushed a commit to pytest-dev/pytest that referenced this pull request May 18, 2024

Fix new typing issues in AST code (#12337)

ee9ea70

python/typeshed#11880 adds more precise types for AST nodes. I'm submitting some changes to adapt pytest to these changes.

bzoracler deleted the ast-subclass-constructors branch May 18, 2024 20:12

JelleZijlstra mentioned this pull request May 18, 2024

Remove non-existent kwargs in ast.Call() call streamlit/streamlit#8711

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

improvement: Add AST subclass constructors #11880

improvement: Add AST subclass constructors #11880

bzoracler commented May 9, 2024 •

edited

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

JelleZijlstra commented May 9, 2024

JelleZijlstra May 9, 2024

bzoracler May 10, 2024

JelleZijlstra May 9, 2024

bzoracler May 10, 2024

This comment has been minimized.

bzoracler commented May 10, 2024 •

edited

This comment has been minimized.

JelleZijlstra commented May 10, 2024

This comment has been minimized.

github-actions bot commented May 10, 2024

JelleZijlstra left a comment

improvement: Add AST subclass constructors #11880

improvement: Add AST subclass constructors #11880

Conversation

bzoracler commented May 9, 2024 • edited

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

JelleZijlstra commented May 9, 2024

JelleZijlstra May 9, 2024

Choose a reason for hiding this comment

bzoracler May 10, 2024

Choose a reason for hiding this comment

JelleZijlstra May 9, 2024

Choose a reason for hiding this comment

bzoracler May 10, 2024

Choose a reason for hiding this comment

This comment has been minimized.

bzoracler commented May 10, 2024 • edited

This comment has been minimized.

JelleZijlstra commented May 10, 2024

This comment has been minimized.

github-actions bot commented May 10, 2024

JelleZijlstra left a comment

Choose a reason for hiding this comment

bzoracler commented May 9, 2024 •

edited

bzoracler commented May 10, 2024 •

edited