Wrong type inference for lists of a generic type with varying component types

[Zac-HD]

Using Mypy 0.610, default settings, and Python 3.6, I have found what I believe is a bug in inference about generic types.

# First off, we expect this to give `List[Union[None, int]]`, and it does.
reveal_type([None, 0])

import typing as t
T = t.TypeVar('T')  # same problem with covariant=True
class G(t.Generic[T]):
    def __init__(self, value: T) -> None:
        self.value = value
def nil() -> G[None]:
    return G(None)
def integer() -> G[int]:
    return G(0)
def nil_any() -> G[t.Any]:
    return G(None)
def integer_any() -> G[t.Any]:
    return G(0)

# but this is revealed as `List[object]` instead of `List[G[Union[None, int]]]`
# or at worst `List[Union[G[None], G[int]]]`.  Even `List[G[object]]` would be OK...
reveal_type([nil(), integer()])

# while this form reveals `List[G[Any]]` - making it a more accurate hint!
reveal_type([nil_any(), integer_any()])

This issue is blocking useful type hints for the one_of function, which takes the union of several strategies for generating test data with Hypothesis - see HypothesisWorks/hypothesis#1270 for discussion.

Our current workaround is to have one_of accept and return SearchStrategy[Any], so it does compose correctly but the checks are weaker than we'd like.

[gvanrossum]

Have you read up on invariance?

[Zac-HD]

I'm certainly familiar with the basics of in/co/contravariant types; if that's what you mean.

In this case though I have precisely the same types inferred whether I pass covariant=True, contravariant=True, or nothing at all to the type variable. In Hypothesis the generic type SearchStrategy is covariant in Ex (the type of the data it generates), but for the sake of minimal examples I omitted that here since it doesn't change the revealed type at all.

I don't see the connection to invariance though, unless you think it's wrong that a literal list [a, b] should have an inferred type List[Union[type(a), type(b)]]? That's why my example code leads with the equivalent no-generics case.

[ilevkivskyi]

I don't think there is an action item. Inferring nominal joins for instance types instead of unions is a deliberate choice. Both have some downsides, but we can't change the current behaviour at least for backwards compatibility. To better understand the situation, consider this example:

class B:
class C1(B): ...
class C2(B): ...
[C1(), C2()]  # what should this be, List[Union[C1, C2]] or List[B]?

Also there is a simple way to get a union by just giving it a union context:

def f(x: List[Union[str, int]]) -> None:
    ...
x: List[Union[int, str]]

x = [1, "hi"]  # OK
f([1, "hi"])  # Also OK

Finally, there is an issue to require an annotation whenever <built-in collection>[object] is inferred (since object is kind of corner case).

[Zac-HD]

@ilevkivskyi - surely it should still infer G[object] rather than object though? Otherwise why does G[Any] keep the generic type?

Discarding the generic type entirely makes more accurate type hints not just useless but actively counter-productive!

[ilevkivskyi]

surely it should still infer G[object] rather than object though?

No, because G is itself invariant, thus G[object] is not a supertype of G[int]. G[Any] however is a supertype (kind of) of G[int], but mypy avoids inferring Any unless an Any is already present among types.

I wanted to propose you making G covariant, because in that case the join would be G[object], but it looks like there is a bug in mypy. I looked at the code, and found a "shortcut" that misses some corner cases (in particular contravariance is totally lost). I will open a separate issue for this (inferring unions is not going to happen, see my previous comment, so I will not re-open this one).

Discarding the generic type entirely makes more accurate type hints not just useless but actively counter-productive!

First, we don't just "discard" anything, there are rules. Second, I suggest you to watch your words. You are using results of free work of many people, so calling it "worse than useless" is at least ungrateful.

[Zac-HD]

Thanks - #5269 will fix the issue that I have downstream, which does involve a covariant generic as noted in the code sample above.

I'm sorry to have come across as ungrateful - that was certainly not my intent, but I should have been more careful with my language. I do value and appreciate the work everyone puts in to Mypy and the ecosystem in general!

[ilevkivskyi]

@Zac-HD

that was certainly not my intent

OK, thanks. Written communication can lead to misunderstandings, I am glad it was just a misunderstanding.