Support narrowing literals and enums using the in operator in combination with tuple expressions.

[tyralla]

The general idea is to transform expressions like

(x is None) and (x in (1, 2)) and (x not in (3, 4))

into

(x is None) and (x == 1 or x == 2) and (x != 3 and x != 4)

This transformation circumvents the need to extend the (already complicated) narrowing logic further.

See the test testNarrowTypeAfterInTuple, which shows a slight change in the narrowing behaviour. I don't know if this is an improvement, but at least it seems to increase consistency.

The only downside of the transformation approach I am aware of is that it might cause some hard-too-understand error messages. The only such problem I encountered is the Right operand of "and" is never evaluated warning in the testNarrowLiteralsNotInTupleExpression test.

[davidhalter]

I generally like what you are trying to achieve.

But it feels a bit weird that this is the way you approach this. I have recently done a lot of research into the Mypy narrowing code and it feels like the most convoluted and complicated part of the whole Mypy code base. (Other parts like the recently added --new-type-inference are way more approachable even though they are more complex). I therefore understand that you wanted to special case this.

However, in this case it feels like the code would be more approachable if it was actually implemented in the in operator narrowing part of the code. It just feels super weird that there would be two places that narrow None in different ways.

Also you handled TupleExpr, which would imply a different behavior for TupleType, but that's fixable, I guess.

But then again I'm not a Mypy dev, so they might feel different about this PR.

[tyralla]

@davidhalter Thanks for your opinion on this, which I understand. I started with extending the mentioned in operator section and targeting TupleType but felt that a separate handling for TupleExpr would be simpler and maybe more correct.

Regarding the different ways to narrow None, I think this problem already exists. This is how the testNarrowTypeAfterInTuple test case looks like without my changes:

from typing import Optional
class A: pass
class B(A): pass
class C(A): pass

y: Optional[B]
if y in (B(), C()):
    reveal_type(y) # N: Revealed type is "__main__.B"
else:
    reveal_type(y) # N: Revealed type is "Union[__main__.B, None]"

But if one uses == instead of in, the result (again, without my changes) is:

if y == B() or y == C():
    reveal_type(y) # N: Revealed type is "Union[__main__.B, None]"
else:
    reveal_type(y) # N: Revealed type is "Union[__main__.B, None]"

So, for this aspect, I would say the proposed change increases consistency (always the second result).

I might have been too cautious about the TupleExpr vs. TupleType issue. I wanted to avoid handling code where users apply tuple subclasses with overwritten __contains__ methods. However, I now checked it and realised that Mypy models such subclasses with Instance and not TupleType. So, generally, targeting TupleType seems the right choice (although I expect 99 % of all potential use cases to work with tuples created on the fly so that targeting TupleExpr only could be satisfying enough).

Two Mypy primer responses are satisfying (discord), and one needs investigation (mkosi). However, for Mypyc, there seems to be a relevant problem. (I have no Mypyc experience, but my first impression is that the proposed transformation confuses Mypy. Maybe the left operand is only available for the first comparison with the items of the right operand?)

[github-actions]

Diff from mypy_primer, showing the effect of this PR on open source code:

discord.py (https://github.com/Rapptz/discord.py)
- discord/app_commands/tree.py:413: error: Missing return statement  [return]
- discord/app_commands/tree.py:533: error: Missing return statement  [return]

[tyralla]

The mkosi and mypyc problems are fixed. I also included a small documentation update in case this pull request gets accepted. It's clearly not the most general solution, but it should handle the most frequent use case (at least the one often relevant to me...), so I will wait for a review.