Announcement issue for plugin API changes

[msullivan]

The mypy plugin interface is experimental, unstable, and prone to change. In particular, there are no guarantees about backwards compatibility. Backwards incompatible changes may be made without a deprecation period.

We will, however, attempt to announce breaking changes in this issue, so that plugin developers can subscribe to this issue and be notified.

Breaking changes fall into three broad categories:

1. Changes to the actual plugin API itself
2. Changes to parts of the "implicit plugin API"---that is, internals that plugins are likely to use (representation of types, etc).
3. Changes to things that really aren't plausibly part of the plugin API (but, of course, that some plugins might be using anyway...)

Issues in category 1 will be consistently announced here, issues in category 3 will probably be announced here only if problems are reported, and issues in category 2 will be somewhere in the middle.

[msullivan]

And we'll start this off with a belated category 3 announcement:

The default wheels for mypy 0.700 are compiled with mypyc. This breaks monkey-patching of mypy internals.

If you are the author of a mypy plugin that relies on monkey-patching mypy internals, get in touch with us and we can probably find a better approach. (For example, #6598 added hooks needed by the django plugin.)

[gvanrossum]

(Sorry for the accidental unpin. Fixed now.)

[JukkaL]

The new semantic analyzer requires changes to some plugins, especially those that modify classes. In particular, hooks may be executed multiple times for the same definitions. PR #7135 added documentation about how to support the new semantic analyzer.

Note that mypy 0.720 (to be released soon) will enable the semantic analyzer by default, and the next release after that will remove the old semantic analyzer.

PRs #7136, #7132, #7096, #6987, #6984, #6724 and #6515 contain examples of changes that may be needed to plugins.

To test that a plugin works with the semantic analyzer, you should have test cases that cause mypy to analyze things twice. The easiest way to achieve is to add a forward reference to a type at module top level:

forwardref: C   # Forward reference to C causes deferral
class C: pass

# ... followed by whatever you want to test

[msullivan]

PR #7397 moves around some functions as part of untangling the cyclic imports in mypy.

The most prominent change and the most likely to impact plugins is:

• mypy.types.UnionType.make_simplified_union -> mypy.typeops.make_simplified_union

Additionally:

• mypy.types.TypeVarDef.erase_to_union_or_bound -> mypy.types.typeops.erase_def_to_union_or_bound
• mypy.types.TypeVarType.erase_to_union_or_bound -> mypy.types.typeops.erase_to_union_or_bound
• mypy.types.{true_only, false_only, true_or_false} -> mypy.typeops
• mypy.types.CallableType.corresponding_argument -> mypy.types.typeops.corresponding_argument
• Many functions from mypy.checkmember have been moved to mypy.typeops. The most prominent here is bind_self.

[msullivan]

PR #7829 makes all name and fullname methods in mypy into properties. This will unfortunately require changes to many plugins. We've decided that it is worth removing a long-standing pain point and that it is better to do it sooner than later.

sed can be used to update code to the new version with something like sed -i -e 's/\.name()/.name/g' -e 's/\.fullname()/.fullname/g'

If your plugin wishes to support older and newer versions during a transition period, this can be done with these helper functions:

from typing import Union
from mypy.nodes import FuncBase, SymbolNode


def fullname(x: Union[FuncBase, SymbolNode]) -> str:
    fn = x.fullname
    if callable(fn):
        return fn()
    return fn


def name(x: Union[FuncBase, SymbolNode]) -> str:
    fn = x.name
    if callable(fn):
        return fn()
    return fn

I don't have an automated way to convert code to use these, but if somebody produces one and sends it to me I will update this post.

Sorry for the inconvenience!

[ilevkivskyi]

PR #7923 changed the internal representation of type aliases in mypy. Previously, type aliases were always eagerly expanded. For example, in this case:

Alias = List[int]
x: Alias

the type of the Var node associated with x was Instance, now it will be a TypeAliasType. This change can cause subtle bugs in plugins that make decisions using calls like if isinstance(typ, Instance): ... as such calls will now return False for type aliases.

There are two helper functions mypy.types.get_proper_type() and mypy.types.get_proper_types() that return expansions for type aliases. Note: if after making the decision on the isinstance() call you pass on the original type (and not one of its component) it is recommended to always pass on the unexpanded alias.

There is also a mypy plugin to type-check your mypy plugins, see misc/proper_plugin.py, it will flag all "dangerous" isinstance() calls.

Sorry for the inconvenience!

[ilevkivskyi]

(An additional small reminder related to last two comments: don't forget that a plugin entry point gets the mypy version string, you can use it for more flexibility.)

[hauntsaninja]

[Category 2 change] PR #9951 gets rid of TypeVarDef; use TypeVarType instead. If you're wondering what the difference between them was, so was I, which is why there's only one of them now.
cc @samuelcolvin @sobolevn @oremanj @suned @seandstewart as people who have written code that would be affected.

[sobolevn]

Thanks! It affects some of my code:

• https://github.com/dry-python/classes/blob/3bfc2c9b7ad8cc59d2ddeea66acebc3ca12efa5a/classes/contrib/mypy/typeops/call_signatures.py#L104

[JukkaL]

#11541 causes mypy to kill the process at the end of a run, without cleaning things up properly. This might affect plugins that want to run something at the end of a run, or that assume that all files are flushed at the end of a run. --no-fast-exit can be used as a workaround, as it disables the new behavior. A better idea would be to flush files immediately.

If this change seems to cause many issues, we could consider a way of registering handlers that get run at the end of a mypy run.

[97littleleaf11]

Deprecated SemanticAnalyzer.builtin_type had been removed since 5bd2641 (0.930). Please use named_type instead.

We brought back the SemanticAnalyzer.builtin_type in 0.931 for backward compatible. It is still marked as deprecated.

[97littleleaf11]

PR #11332 changes SemanticAnalyzer.named_type to use fully_qualified_name. Now we can call it with builtins instead of __builtins__.

[JukkaL]

PR #14435 changes the runtime type of various (but not all!) fullname attributes/properties so that missing/empty values are represented using an empty string ("") instead of None. If a plugin guards against empty fullnames, it may need to updated. For example, consider a check like this:

    if n.fullname is not None:
        # do something with n.fullname

It can be updated like this, since an empty string is falsy (this also works with older mypy versions that use None):

    if n.fullname:
        # do something with n.fullname

[ikonst]

PR #15369 adds get_expression_type to the CheckerPluginInterface. This enables the common scenario in method/function signature hooks, where the actual type of an argument affects the rest of the signature.

This change is backwards compatible.

For example:

first_arg = ctx.args[0][0]
first_arg_type = ctx.api.get_expression_type(first_arg)

return ctx.default_signature.copy_modified(
  arg_types=[first_arg_type, first_arg_type],  # 1st arg affects 2nd arg's type
)

[cdce8p]

PR #14872 (v1.4.0) adds a new required argument - default - to all TypeVarLikeExpr and TypeVarLikeType types. This is in preparation for PEP 696 (TypeVar defaults) support.

If a plugin constructs these expression / types manually, a version guard needs to be added. E.g.

from mypy.nodes import TypeVarExpr
from mypy.types import TypeVarType, AnyType, TypeOfAny

def parse_mypy_version(version: str) -> tuple[int, ...]:
    return tuple(map(int, version.partition('+')[0].split('.')))

MYPY_VERSION_TUPLE = parse_mypy_version(mypy_version)

# ...

if MYPY_VERSION_TUPLE >= (1, 4):
    tvt = TypeVarType(
        self_tvar_name,
        tvar_fullname,
        -1,
        [],
        obj_type,
        AnyType(TypeOfAny.from_omitted_generics),  # <-- new!
    )
    self_tvar_expr = TypeVarExpr(
        self_tvar_name,
        tvar_fullname,
        [],
        obj_type,
        AnyType(TypeOfAny.from_omitted_generics),  # <-- new!
    )
else:
    tvt = TypeVarType(self_tvar_name, tvar_fullname, -1, [], obj_type)
    self_tvar_expr = TypeVarExpr(self_tvar_name, tvar_fullname, [], obj_type)

If no explicit default value is provided, AnyType(TypeOfAny.from_omitted_generics) should be used.