Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Crash involving dataclasses, generics, forward references #12685

Closed
NeilGirdhar opened this issue Apr 28, 2022 · 7 comments · Fixed by #12762
Closed

Crash involving dataclasses, generics, forward references #12685

NeilGirdhar opened this issue Apr 28, 2022 · 7 comments · Fixed by #12762
Assignees
@JukkaL
Labels
crash topic-dataclasses

Comments

@NeilGirdhar
Copy link
Contributor

NeilGirdhar commented Apr 28, 2022 •
edited

Running master MyPy 0.960 on the latest efax 1.6.3 gives

❯ git clone git@github.com:NeilGirdhar/efax.git
❯ cd efax
❯ mypy efax --show-traceback
efax/_src/distributions/dirichlet.py:29: error: INTERNAL ERROR -- Please try using mypy master on Github:
https://mypy.readthedocs.io/en/stable/common_issues.html#using-a-development-mypy-build
Please report a bug at https://github.com/python/mypy/issues
version: 0.960+dev.ee78afe325800f9266ccc0d143b49b8c0b29d8a3
Traceback (most recent call last):
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/bin/mypy", line 8, in <module>
    sys.exit(console_entry())
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/__main__.py", line 12, in console_entry
    main(None, sys.stdout, sys.stderr)
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/main.py", line 96, in main
    res, messages, blockers = run_build(sources, options, fscache, t0, stdout, stderr)
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/main.py", line 173, in run_build
    res = build.build(sources, options, None, flush_errors, fscache, stdout, stderr)
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/build.py", line 180, in build
    result = _build(
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/build.py", line 256, in _build
    graph = dispatch(sources, manager, stdout)
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/build.py", line 2733, in dispatch
    process_graph(graph, manager)
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/build.py", line 3081, in process_graph
    process_stale_scc(graph, scc, manager)
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/build.py", line 3173, in process_stale_scc
    mypy.semanal_main.semantic_analysis_for_scc(graph, scc, manager.errors)
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/semanal_main.py", line 78, in semantic_analysis_for_scc
    process_top_levels(graph, scc, patches)
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/semanal_main.py", line 199, in process_top_levels
    deferred, incomplete, progress = semantic_analyze_target(next_id, state,
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/semanal_main.py", line 326, in semantic_analyze_target
    analyzer.refresh_partial(refresh_node,
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/semanal.py", line 414, in refresh_partial
    self.refresh_top_level(node)
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/semanal.py", line 425, in refresh_top_level
    self.accept(d)
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/semanal.py", line 5340, in accept
    node.accept(self)
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/nodes.py", line 1028, in accept
    return visitor.visit_class_def(self)
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/semanal.py", line 1121, in visit_class_def
    self.analyze_class(defn)
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/semanal.py", line 1201, in analyze_class
    self.analyze_class_body_common(defn)
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/semanal.py", line 1210, in analyze_class_body_common
    self.apply_class_plugin_hooks(defn)
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/semanal.py", line 1256, in apply_class_plugin_hooks
    hook(ClassDefContext(defn, decorator, self))
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/plugins/dataclasses.py", line 524, in dataclass_class_maker_callback
    transformer.transform()
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/plugins/dataclasses.py", line 120, in transform
    attributes = self.collect_attributes()
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/plugins/dataclasses.py", line 408, in collect_attributes
    attr.expand_typevar_from_subtype(ctx.cls.info)
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/plugins/dataclasses.py", line 106, in expand_typevar_from_subtype
    self.type = map_type_from_supertype(self.type, sub_type, self.info)
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/typeops.py", line 151, in map_type_from_supertype
    inst_type = map_instance_to_supertype(inst_type, super_info)
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/maptype.py", line 24, in map_instance_to_supertype
    return map_instance_to_supertypes(instance, superclass)[0]
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/maptype.py", line 37, in map_instance_to_supertypes
    a.extend(map_instance_to_direct_supertypes(t, sup))
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/maptype.py", line 82, in map_instance_to_direct_supertypes
    t = expand_type(b, env)
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/expandtype.py", line 18, in expand_type
    return typ.accept(ExpandTypeVisitor(env))
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/types.py", line 1168, in accept
    return visitor.visit_instance(self)
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/expandtype.py", line 90, in visit_instance
    args = self.expand_types(t.args)
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/expandtype.py", line 215, in expand_types
    a.append(t.accept(self))
  File "/home/neil/.cache/pypoetry/virtualenvs/efax-ZssyUsLU-py3.10/lib/python3.10/site-packages/mypy/types.py", line 2513, in accept
    assert isinstance(visitor, SyntheticTypeVisitor)
AssertionError: 
efax/_src/distributions/dirichlet.py:29: : note: use --pdb to drop into pdb
@NeilGirdhar NeilGirdhar added the bug mypy got something wrong label Apr 28, 2022
@AlexWaygood AlexWaygood added crash and removed bug mypy got something wrong labels Apr 28, 2022
@NeilGirdhar
Copy link
Contributor Author

I briefly tried to make an MWE, but was unsuccessful.

@hauntsaninja
Copy link
Collaborator

Looks like this is a regression between 0.942 and 0.950

@hauntsaninja
Copy link
Collaborator

This bisects to #12656

@hauntsaninja hauntsaninja mentioned this issue Apr 28, 2022
Closed
@hauntsaninja hauntsaninja changed the title Crash Crash involving dataclasses Apr 28, 2022
@hauntsaninja
Copy link
Collaborator

Okay, took another look at this and got a small repro:

from dataclasses import dataclass
from typing import Generic, TypeVar

T = TypeVar("T")

@dataclass
class Parent(Generic[T]):
    key: str

@dataclass
class Child1(Parent["Child2"]): ...

@dataclass
class Child2(Parent["Child1"]): ...

@hauntsaninja hauntsaninja changed the title Crash involving dataclasses Crash involving dataclasses, generics, forward references Apr 28, 2022
@NYKevin
Copy link

NYKevin commented May 9, 2022

You can also repro it with just one class, which is probably still tickling the same bug:

import dataclasses
import typing

T = typing.TypeVar('T', bound='Foo')

@dataclasses.dataclass
class Foo(typing.Generic[T]):
	data: int

If you remove either of the following elements, the problem won't be reproducible:

  • The data field.
  • The bound on T.

The bound need not be Foo itself, as long as it is a necessary forward reference - that is, a forward reference to a class defined below T (and therefore "necessary" because, if you removed the quotes, it would fail with a NameError). If you put e.g. 'object', or the name of a class which already exists, then it won't reproduce. But you can just put something like class Bar: pass below the declaration of T, and then bound='Bar' will reproduce the bug.

The traceback is also slightly different:

example.py:8: error: INTERNAL ERROR -- Please try using mypy master on GitHub:
https://mypy.readthedocs.io/en/stable/common_issues.html#using-a-development-mypy-build
Please report a bug at https://github.com/python/mypy/issues
version: 0.960+dev.612074f05ba5d65e016c031dc94b13f962d9c6b4
Traceback (most recent call last):
  File "C:\Users\Kevin\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\Kevin\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\Kevin\poker\mypy-testing\Scripts\mypy.exe\__main__.py", line 7, in <module>
    sys.exit(console_entry())
  File "C:\Users\Kevin\poker\mypy-testing\lib\site-packages\mypy\__main__.py", line 12, in console_entry
    main(None, sys.stdout, sys.stderr)
  File "C:\Users\Kevin\poker\mypy-testing\lib\site-packages\mypy\main.py", line 96, in main
    res, messages, blockers = run_build(sources, options, fscache, t0, stdout, stderr)
  File "C:\Users\Kevin\poker\mypy-testing\lib\site-packages\mypy\main.py", line 173, in run_build
    res = build.build(sources, options, None, flush_errors, fscache, stdout, stderr)
  File "C:\Users\Kevin\poker\mypy-testing\lib\site-packages\mypy\build.py", line 181, in build
    result = _build(
  File "C:\Users\Kevin\poker\mypy-testing\lib\site-packages\mypy\build.py", line 257, in _build
    graph = dispatch(sources, manager, stdout)
  File "C:\Users\Kevin\poker\mypy-testing\lib\site-packages\mypy\build.py", line 2752, in dispatch
    process_graph(graph, manager)
  File "C:\Users\Kevin\poker\mypy-testing\lib\site-packages\mypy\build.py", line 3110, in process_graph
    process_stale_scc(graph, scc, manager)
  File "C:\Users\Kevin\poker\mypy-testing\lib\site-packages\mypy\build.py", line 3202, in process_stale_scc
    mypy.semanal_main.semantic_analysis_for_scc(graph, scc, manager.errors)
  File "C:\Users\Kevin\poker\mypy-testing\lib\site-packages\mypy\semanal_main.py", line 79, in semantic_analysis_for_scc
    process_functions(graph, scc, patches)
  File "C:\Users\Kevin\poker\mypy-testing\lib\site-packages\mypy\semanal_main.py", line 233, in process_functions
    process_top_level_function(analyzer,
  File "C:\Users\Kevin\poker\mypy-testing\lib\site-packages\mypy\semanal_main.py", line 274, in process_top_level_function
    deferred, incomplete, progress = semantic_analyze_target(target, state, node, active_type,
  File "C:\Users\Kevin\poker\mypy-testing\lib\site-packages\mypy\semanal_main.py", line 326, in semantic_analyze_target
    analyzer.refresh_partial(refresh_node,
  File "C:\Users\Kevin\poker\mypy-testing\lib\site-packages\mypy\semanal.py", line 417, in refresh_partial
    self.accept(node)
  File "C:\Users\Kevin\poker\mypy-testing\lib\site-packages\mypy\semanal.py", line 5344, in accept
    node.accept(self)
  File "C:\Users\Kevin\poker\mypy-testing\lib\site-packages\mypy\nodes.py", line 747, in accept
    return visitor.visit_func_def(self)
  File "C:\Users\Kevin\poker\mypy-testing\lib\site-packages\mypy\semanal.py", line 637, in visit_func_def
    self.analyze_func_def(defn)
  File "C:\Users\Kevin\poker\mypy-testing\lib\site-packages\mypy\semanal.py", line 669, in analyze_func_def
    self.defer(defn)
  File "C:\Users\Kevin\poker\mypy-testing\lib\site-packages\mypy\semanal.py", line 5044, in defer
    assert not self.final_iteration, 'Must not defer during final iteration'
AssertionError: Must not defer during final iteration
example.py:8: : note: use --pdb to drop into pdb

@hauntsaninja
Copy link
Collaborator

Thanks. I think that's actually a different issue (and one that is not a regression), see #12527

@JukkaL
Copy link
Collaborator

JukkaL commented May 10, 2022

The root cause seems to be that we can't safely run the dataclass transform in the main pass of the semantic analysis, since various references can still be unbound, and we can encounter placeholder types, etc. I'm going to attempt to move the whole transform to a later pass where all references are bound.

@JukkaL JukkaL self-assigned this May 10, 2022
@JukkaL JukkaL mentioned this issue May 10, 2022
Merged
JukkaL added a commit that referenced this issue May 11, 2022
@JukkaL
Running dataclass transform in a later pass to fix crashes (#12762)
03901ef 
The dataclass plugin could crash if it encountered a placeholder. Fix the issue by
running the plugin after the main semantic analysis pass, when all placeholders have
been resolved.

Also add a new hook called get_class_decorator_hook_2 that is used by the
dataclass plugin.

We may want to do a similar change to the attrs plugin, but let's change one thing
at a time.

Fix #12685.
@AlexWaygood AlexWaygood mentioned this issue May 15, 2022
Closed
@pranavrajpal pranavrajpal mentioned this issue May 31, 2022
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@JukkaL JukkaL

Labels
crash topic-dataclasses
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Running dataclass transform in a later pass to fix crashes python/mypy
5 participants
@NeilGirdhar @JukkaL @NYKevin @hauntsaninja @AlexWaygood