Skip to content

Commit

Permalink
Bug pylint 4206 (#921)
Browse files Browse the repository at this point in the history
* Takes into account the fact that inferring subscript when the node is a class may use the __class_getitem__ method of the current class instead of looking for __getitem__ in the metaclass.

* OrderedDict in the collections module inherit from dict which is C coded and thus have no metaclass but starting from python3.9 it supports subscripting thanks to the __class_getitem__ method.

* check_metaclass becomes a static class method because we need it in the class scope.
The brain_typing module does not add a ABCMeta_typing class thus there is no need to test it. Moreover it doesn't add neither a __getitem__ to the metaclass

* The brain_typing module does not add anymore _typing suffixed classes in the mro

* The OrderedDict class inherits from C coded dict class and thus doesn't have a metaclass.

* When trying to inherit from typing.Pattern the REPL says :  TypeError: type 're.Pattern' is not an acceptable base type

* The REPL says that Derived as ABCMeta for metaclass and the mro is Derived => Iterator => Iterable => object

* Adds comments

* Starting with Python39 some collections of the collections.abc module support subscripting thanks to __class_getitem__ method. However the wat it is implemented is not straigthforward and instead of complexifying the way __class_getitem__ is handled inside the getitem method of the ClassDef class, we prefer to hack a bit.

* Thanks to __class_getitem__ method there is no need to hack the metaclass

* SImplifies the inference system for typing objects before python3.9. Before python3.9 the objects of the typing module that are alias of the same objects in the collections.abc module have subscripting possibility thanks to the _GenericAlias metaclass. To mock the subscripting capability we add __class_getitem__ method on those objects.

* check_metaclass_is_abc become global to be shared among different classes

* Create a test class dedicated to the Collections brain

* Rewrites and adds test

* Corrects syntax error

* Deque, defaultdict and OrderedDict are part of the _collections module which is a pure C lib. While letting those class mocks inside collections module is fair for astroid it leds to pylint acceptance tests fail.

* Formatting according to black

* Adds two entries

* Extends the filter to determine what is subscriptable to include OrderedDict

* Formatting according to black

* Takes into account the fact that inferring subscript when the node is a class may use the __class_getitem__ method of the current class instead of looking for __getitem__ in the metaclass.

* OrderedDict in the collections module inherit from dict which is C coded and thus have no metaclass but starting from python3.9 it supports subscripting thanks to the __class_getitem__ method.

* check_metaclass becomes a static class method because we need it in the class scope.
The brain_typing module does not add a ABCMeta_typing class thus there is no need to test it. Moreover it doesn't add neither a __getitem__ to the metaclass

* The brain_typing module does not add anymore _typing suffixed classes in the mro

* The OrderedDict class inherits from C coded dict class and thus doesn't have a metaclass.

* When trying to inherit from typing.Pattern the REPL says :  TypeError: type 're.Pattern' is not an acceptable base type

* The REPL says that Derived as ABCMeta for metaclass and the mro is Derived => Iterator => Iterable => object

* Adds comments

* Starting with Python39 some collections of the collections.abc module support subscripting thanks to __class_getitem__ method. However the wat it is implemented is not straigthforward and instead of complexifying the way __class_getitem__ is handled inside the getitem method of the ClassDef class, we prefer to hack a bit.

* Thanks to __class_getitem__ method there is no need to hack the metaclass

* SImplifies the inference system for typing objects before python3.9. Before python3.9 the objects of the typing module that are alias of the same objects in the collections.abc module have subscripting possibility thanks to the _GenericAlias metaclass. To mock the subscripting capability we add __class_getitem__ method on those objects.

* check_metaclass_is_abc become global to be shared among different classes

* Create a test class dedicated to the Collections brain

* Rewrites and adds test

* Corrects syntax error

* Deque, defaultdict and OrderedDict are part of the _collections module which is a pure C lib. While letting those class mocks inside collections module is fair for astroid it leds to pylint acceptance tests fail.

* Formatting according to black

* Adds two entries

* Extends the filter to determine what is subscriptable to include OrderedDict

* Formatting according to black

* Takes into account AWhetter remarks

* Deactivates access to __class_getitem__ method

* OrderedDict appears in typing module with python3.7.2

* _alias function in the typing module appears with python3.7

* Formatting according to black

* _alias function is used also for builtins type and not only for collections.abc ones

* Adds tests for both builtins type that are subscriptable and typing builtin alias type that are also subscriptable

* No need to handle builtin types in this brain. It is better suited inside brain_bulitin_inference

* Adds brain to handle builtin types that are subscriptable starting with python39

* Formatting according to black

* Uses partial function instead of closure in order pylint acceptance to be ok

* Handling the __class_getitem__ method associated to EmptyNode for builtin types is made directly inside the getitem method

* infer_typing_alias has to be an inference_tip to avoid interferences between typing module and others (collections or builtin)

* Formatting

* Removes useless code

* Adds comment

* Takes into account cdce8p remarks

* Formatting

* Style changes

Co-authored-by: Marc Mueller <30130371+cdce8p@users.noreply.github.com>
  • Loading branch information
hippo91 and cdce8p committed Apr 6, 2021
1 parent 6cc2c66 commit 8e28720
Show file tree
Hide file tree
Showing 5 changed files with 413 additions and 101 deletions.
5 changes: 5 additions & 0 deletions ChangeLog
Expand Up @@ -11,6 +11,11 @@ What's New in astroid 2.5.3?
============================
Release Date: TBA

* Takes into account the fact that subscript inferring for a ClassDef may involve __class_getitem__ method

* Reworks the `collections` and `typing` brain so that `pylint`s acceptance tests are fine.

Closes PyCQA/pylint#4206

What's New in astroid 2.5.2?
============================
Expand Down
48 changes: 47 additions & 1 deletion astroid/brain/brain_collections.py
Expand Up @@ -68,7 +68,7 @@ def __rmul__(self, other): pass"""
if PY39:
base_deque_class += """
@classmethod
def __class_getitem__(self, item): pass"""
def __class_getitem__(self, item): return cls"""
return base_deque_class


Expand All @@ -77,7 +77,53 @@ def _ordered_dict_mock():
class OrderedDict(dict):
def __reversed__(self): return self[::-1]
def move_to_end(self, key, last=False): pass"""
if PY39:
base_ordered_dict_class += """
@classmethod
def __class_getitem__(cls, item): return cls"""
return base_ordered_dict_class


astroid.register_module_extender(astroid.MANAGER, "collections", _collections_transform)


def _looks_like_subscriptable(node: astroid.nodes.ClassDef) -> bool:
"""
Returns True if the node corresponds to a ClassDef of the Collections.abc module that
supports subscripting
:param node: ClassDef node
"""
if node.qname().startswith("_collections") or node.qname().startswith(
"collections"
):
try:
node.getattr("__class_getitem__")
return True
except astroid.AttributeInferenceError:
pass
return False


CLASS_GET_ITEM_TEMPLATE = """
@classmethod
def __class_getitem__(cls, item):
return cls
"""


def easy_class_getitem_inference(node, context=None):
# Here __class_getitem__ exists but is quite a mess to infer thus
# put an easy inference tip
func_to_add = astroid.extract_node(CLASS_GET_ITEM_TEMPLATE)
node.locals["__class_getitem__"] = [func_to_add]


if PY39:
# Starting with Python39 some objects of the collection module are subscriptable
# thanks to the __class_getitem__ method but the way it is implemented in
# _collection_abc makes it difficult to infer. (We would have to handle AssignName inference in the
# getitem method of the ClassDef class) Instead we put here a mock of the __class_getitem__ method
astroid.MANAGER.register_transform(
astroid.nodes.ClassDef, easy_class_getitem_inference, _looks_like_subscriptable
)
139 changes: 80 additions & 59 deletions astroid/brain/brain_typing.py
Expand Up @@ -8,7 +8,7 @@
"""Astroid hooks for typing.py support."""
import sys
import typing
from functools import lru_cache
from functools import partial

from astroid import (
MANAGER,
Expand All @@ -19,6 +19,7 @@
nodes,
context,
InferenceError,
AttributeInferenceError,
)
import astroid

Expand Down Expand Up @@ -116,37 +117,12 @@ def infer_typedDict( # pylint: disable=invalid-name
node.root().locals["TypedDict"] = [class_def]


GET_ITEM_TEMPLATE = """
CLASS_GETITEM_TEMPLATE = """
@classmethod
def __getitem__(cls, value):
def __class_getitem__(cls, item):
return cls
"""

ABC_METACLASS_TEMPLATE = """
from abc import ABCMeta
ABCMeta
"""


@lru_cache()
def create_typing_metaclass():
#  Needs to mock the __getitem__ class method so that
#  MutableSet[T] is acceptable
func_to_add = extract_node(GET_ITEM_TEMPLATE)

abc_meta = next(extract_node(ABC_METACLASS_TEMPLATE).infer())
typing_meta = nodes.ClassDef(
name="ABCMeta_typing",
lineno=abc_meta.lineno,
col_offset=abc_meta.col_offset,
parent=abc_meta.parent,
)
typing_meta.postinit(
bases=[extract_node(ABC_METACLASS_TEMPLATE)], body=[], decorators=None
)
typing_meta.locals["__getitem__"] = [func_to_add]
return typing_meta


def _looks_like_typing_alias(node: nodes.Call) -> bool:
"""
Expand All @@ -161,10 +137,43 @@ def _looks_like_typing_alias(node: nodes.Call) -> bool:
isinstance(node, nodes.Call)
and isinstance(node.func, nodes.Name)
and node.func.name == "_alias"
and isinstance(node.args[0], nodes.Attribute)
and (
# _alias function works also for builtins object such as list and dict
isinstance(node.args[0], nodes.Attribute)
or isinstance(node.args[0], nodes.Name)
and node.args[0].name != "type"
)
)


def _forbid_class_getitem_access(node: nodes.ClassDef) -> None:
"""
Disable the access to __class_getitem__ method for the node in parameters
"""

def full_raiser(origin_func, attr, *args, **kwargs):
"""
Raises an AttributeInferenceError in case of access to __class_getitem__ method.
Otherwise just call origin_func.
"""
if attr == "__class_getitem__":
raise AttributeInferenceError("__class_getitem__ access is not allowed")
else:
return origin_func(attr, *args, **kwargs)

if not isinstance(node, nodes.ClassDef):
raise TypeError("The parameter type should be ClassDef")
try:
node.getattr("__class_getitem__")
# If we are here, then we are sure to modify object that do have __class_getitem__ method (which origin is one the
# protocol defined in collections module) whereas the typing module consider it should not
# We do not want __class_getitem__ to be found in the classdef
partial_raiser = partial(full_raiser, node.getattr)
node.getattr = partial_raiser
except AttributeInferenceError:
pass


def infer_typing_alias(
node: nodes.Call, ctx: context.InferenceContext = None
) -> typing.Optional[node_classes.NodeNG]:
Expand All @@ -174,38 +183,48 @@ def infer_typing_alias(
:param node: call node
:param context: inference context
"""
if not isinstance(node, nodes.Call):
return None
res = next(node.args[0].infer(context=ctx))

if res != astroid.Uninferable and isinstance(res, nodes.ClassDef):
class_def = nodes.ClassDef(
name=f"{res.name}_typing",
lineno=0,
col_offset=0,
parent=res.parent,
)
class_def.postinit(
bases=[res],
body=res.body,
decorators=res.decorators,
metaclass=create_typing_metaclass(),
)
return class_def

if len(node.args) == 2 and isinstance(node.args[0], nodes.Attribute):
class_def = nodes.ClassDef(
name=node.args[0].attrname,
lineno=0,
col_offset=0,
parent=node.parent,
)
class_def.postinit(
bases=[], body=[], decorators=None, metaclass=create_typing_metaclass()
)
return class_def

return None
if not PY39:
# Here the node is a typing object which is an alias toward
# the corresponding object of collection.abc module.
# Before python3.9 there is no subscript allowed for any of the collections.abc objects.
# The subscript ability is given through the typing._GenericAlias class
# which is the metaclass of the typing object but not the metaclass of the inferred
# collections.abc object.
# Thus we fake subscript ability of the collections.abc object
# by mocking the existence of a __class_getitem__ method.
# We can not add `__getitem__` method in the metaclass of the object because
# the metaclass is shared by subscriptable and not subscriptable object
maybe_type_var = node.args[1]
if not (
isinstance(maybe_type_var, node_classes.Tuple)
and not maybe_type_var.elts
):
# The typing object is subscriptable if the second argument of the _alias function
# is a TypeVar or a tuple of TypeVar. We could check the type of the second argument but
# it appears that in the typing module the second argument is only TypeVar or a tuple of TypeVar or empty tuple.
# This last value means the type is not Generic and thus cannot be subscriptable
func_to_add = astroid.extract_node(CLASS_GETITEM_TEMPLATE)
res.locals["__class_getitem__"] = [func_to_add]
else:
# If we are here, then we are sure to modify object that do have __class_getitem__ method (which origin is one the
# protocol defined in collections module) whereas the typing module consider it should not
# We do not want __class_getitem__ to be found in the classdef
_forbid_class_getitem_access(res)
else:
# Within python3.9 discrepencies exist between some collections.abc containers that are subscriptable whereas
# corresponding containers in the typing module are not! This is the case at least for ByteString.
# It is far more to complex and dangerous to try to remove __class_getitem__ method from all the ancestors of the
# current class. Instead we raise an AttributeInferenceError if we try to access it.
maybe_type_var = node.args[1]
if isinstance(maybe_type_var, nodes.Const) and maybe_type_var.value == 0:
# Starting with Python39 the _alias function is in fact instantiation of _SpecialGenericAlias class.
# Thus the type is not Generic if the second argument of the call is equal to zero
_forbid_class_getitem_access(res)
return iter([res])
return iter([astroid.Uninferable])


MANAGER.register_transform(
Expand All @@ -223,4 +242,6 @@ def infer_typing_alias(
)

if PY37:
MANAGER.register_transform(nodes.Call, infer_typing_alias, _looks_like_typing_alias)
MANAGER.register_transform(
nodes.Call, inference_tip(infer_typing_alias), _looks_like_typing_alias
)
32 changes: 31 additions & 1 deletion astroid/scoped_nodes.py
Expand Up @@ -54,6 +54,8 @@
from astroid import util


PY39 = sys.version_info[:2] >= (3, 9)

BUILTINS = builtins.__name__
ITER_METHODS = ("__iter__", "__getitem__")
EXCEPTION_BASE_CLASSES = frozenset({"Exception", "BaseException"})
Expand Down Expand Up @@ -2617,7 +2619,22 @@ def getitem(self, index, context=None):
try:
methods = dunder_lookup.lookup(self, "__getitem__")
except exceptions.AttributeInferenceError as exc:
raise exceptions.AstroidTypeError(node=self, context=context) from exc
if isinstance(self, ClassDef):
# subscripting a class definition may be
# achieved thanks to __class_getitem__ method
# which is a classmethod defined in the class
# that supports subscript and not in the metaclass
try:
methods = self.getattr("__class_getitem__")
# Here it is assumed that the __class_getitem__ node is
# a FunctionDef. One possible improvement would be to deal
# with more generic inference.
except exceptions.AttributeInferenceError:
raise exceptions.AstroidTypeError(
node=self, context=context
) from exc
else:
raise exceptions.AstroidTypeError(node=self, context=context) from exc

method = methods[0]

Expand All @@ -2627,6 +2644,19 @@ def getitem(self, index, context=None):

try:
return next(method.infer_call_result(self, new_context))
except AttributeError:
# Starting with python3.9, builtin types list, dict etc...
# are subscriptable thanks to __class_getitem___ classmethod.
# However in such case the method is bound to an EmptyNode and
# EmptyNode doesn't have infer_call_result method yielding to
# AttributeError
if (
isinstance(method, node_classes.EmptyNode)
and self.name in ("list", "dict", "set", "tuple", "frozenset")
and PY39
):
return self
raise
except exceptions.InferenceError:
return util.Uninferable

Expand Down

0 comments on commit 8e28720

Please sign in to comment.