Adds `TypeGuard` to `filter` #6140

sobolevn · 2021-10-09T22:11:48Z

Couple of questsions:

How can I test that it works?

I was testing it with local code:

from typing import Iterator, Iterable, Callable, TypeVar, Generic, TypeGuard, Any, List, Optional, overload

_T = TypeVar('_T')
R = TypeVar('R')


class filter(Iterator[_T], Generic[_T]):
    @overload
    def __init__(self, __function: None, __iterable: Iterable[_T | None]) -> None: ...
    @overload
    def __init__(self, __function: Callable[[R], TypeGuard[_T]], __iterable: Iterable[R]) -> None: ...
    @overload
    def __init__(self, __function: Callable[[_T], Any], __iterable: Iterable[_T]) -> None: ...
    def __iter__(self) -> Iterator[_T]: ...
    def __next__(self) -> _T: ...

a: List[Optional[int]]
def is_int_typeguard(a: object) -> TypeGuard[int]: pass
cc = filter(is_int_typeguard, a)
reveal_type(cc)  # N: Revealed type is "ex.filter[Union[builtins.int, None]]"

This is the first TypeGuard in builtins.pyi, are there any extra steps?

Closes #5661

Refs #5661

JelleZijlstra · 2021-10-10T03:32:50Z

From the mypy-primer output, this seems to expose some bugs in mypy's TypeGuard implementation.

Also you can test with mypy's --custom-typeshed-dir option.

sobolevn · 2021-10-10T07:40:33Z

Almost all of the errors above happen because when calling filter(returns_bool, items) with def returns_bool(a) -> bool, it matches def __init__(self, __function: Callable[[R], TypeGuard[_T]], __iterable: Iterable[R]) -> None: ... overload for some reason. Look like a bug in mypy to me.

And that's why inference is broken.

hauntsaninja · 2021-10-10T08:16:14Z

IIRC at least at some point mypy treated TypeGuard exactly the same as bool, instead of as a subtype. Could be a pretty easy fix.

(Also worth noting that in typeshed's CI, we use latest mypy release instead of mypy master)

sobolevn · 2021-10-10T08:36:31Z

Also worth noting that in typeshed's CI, we use latest mypy release instead of mypy master

The same happens with master, I've just checked.
I will open a new issue in mypy. I hopefully will be able to fix it. 👍

sobolevn · 2021-10-10T09:14:39Z

Done: python/mypy#11307

sobolevn · 2021-10-30T13:35:53Z

@hauntsaninja @JelleZijlstra any chance to get this PR merged, please? python/mypy#11314 🙂

JelleZijlstra

The mypy PR got merged, thanks!

This looks good but I'm going to retrigger CI just in case.

JelleZijlstra · 2021-11-14T02:08:15Z

I'm a bit worried about the mypy-primer results here, I'll have to look at it more.

JelleZijlstra · 2021-11-14T02:52:04Z

Looking at the freqtrade ones:

the first one does len(list(filter(lambda x: x["name"] == searchname, columns))) == 1 where columns: List (they got to turn on --disallow-any-generics). It produces Argument 1 to "list" has incompatible type "filter[_T]"; expected "Iterable[_T]". That error makes little sense to me.
The others are around a line roi_list = list(filter(lambda x: x <= trade_dur, self.minimal_roi.keys())), where self.minimal_roi: Dict.
So maybe there's a bug in mypy around using Dict/List without generic arguments.

JelleZijlstra · 2021-12-21T02:50:09Z

Let's see if mypy got any better.

github-actions · 2021-12-21T03:07:31Z

Diff from mypy_primer, showing the effect of this PR on open source code:

zulip (https://github.com/zulip/zulip)
+ zerver/views/documentation.py:244: error: Argument 1 to "list" has incompatible type "filter[_T]"; expected "Iterable[_T]"  [arg-type]

jax (https://github.com/google/jax)
+ jax/core.py:798: error: Argument 1 to "list" has incompatible type "filter[_T]"; expected "Iterable[_T]"  [arg-type]
+ jax/core.py:799: error: Argument 1 to "list" has incompatible type "filter[_T]"; expected "Iterable[_T]"  [arg-type]
+ jax/experimental/host_callback.py:983: error: Argument 1 to "list" has incompatible type "filter[_T]"; expected "Iterable[_T]"  [arg-type]
+ jax/experimental/host_callback.py:1030: error: Argument 1 to "aval_to_xla_shapes" has incompatible type "_T"; expected "AbstractValue"  [arg-type]
+ jax/experimental/jax2tf/impl_no_xla.py:323: error: Argument 1 to "join" of "str" has incompatible type "List[bool]"; expected "Iterable[str]"  [arg-type]
+ jax/experimental/jax2tf/tests/tf_test_util.py:220: error: Argument 1 to "tuple" has incompatible type "filter[_T]"; expected "Iterable[_T]"  [arg-type]

aiortc (https://github.com/aiortc/aiortc)
+ src/aiortc/sdp.py:294: error: Incompatible return value type (got "_T", expected "RTCRtpCodecParameters")
+ src/aiortc/sdp.py:294: error: Argument 1 to "next" has incompatible type "filter[_T]"; expected "SupportsNext[_T]"
+ src/aiortc/rtcpeerconnection.py:66: error: "_T" has no attribute "mimeType"
+ src/aiortc/rtcpeerconnection.py:67: error: "_T" has no attribute "parameters"
+ src/aiortc/rtcpeerconnection.py:74: error: "bool" has no attribute "parameters"
+ src/aiortc/rtcpeerconnection.py:75: error: Argument 1 to "append" of "list" has incompatible type "bool"; expected "RTCRtpCodecParameters"
+ src/aiortc/rtcpeerconnection.py:657: error: Incompatible types in assignment (expression has type "_T", variable has type "RTCRtpTransceiver")

optuna (https://github.com/optuna/optuna)
+ optuna/cli.py:555: error: Argument 1 to "list" has incompatible type "filter[_T]"; expected "Iterable[_T]"
+ optuna/cli.py:556: error: Argument 1 to "_format_output" has incompatible type "List[_T]"; expected "Union[List[Dict[Tuple[str, str], Any]], Dict[Tuple[str, str], Any]]"
+ optuna/importance/_base.py:110: error: Incompatible return value type (got "OrderedDict[str, str]", expected "Dict[str, BaseDistribution]")
+ optuna/importance/_base.py:114: error: Argument 1 to "list" has incompatible type "filter[_T]"; expected "Iterable[_T]"
+ optuna/importance/_base.py:134: error: "_T" has no attribute "distributions"

freqtrade (https://github.com/freqtrade/freqtrade)
+ freqtrade/persistence/migrations.py:15: error: Argument 1 to "list" has incompatible type "filter[_T]"; expected "Iterable[_T]"
+ freqtrade/strategy/interface.py:808: error: Argument 1 to "list" has incompatible type "filter[_T]"; expected "Iterable[_T]"
+ freqtrade/strategy/interface.py:811: error: Value of type variable "SupportsRichComparisonT" of "max" cannot be "_T"
+ freqtrade/strategy/interface.py:812: error: Incompatible return value type (got "Tuple[_T, Any]", expected "Tuple[Optional[int], Optional[float]]")

sphinx (https://github.com/sphinx-doc/sphinx)
+ sphinx/ext/intersphinx.py: note: In function "_resolve_reference_in_domain_by_target":
+ sphinx/ext/intersphinx.py:308:40: error: Argument 1 to "list" has incompatible type "filter[_T]"; expected "Iterable[_T]"
+ sphinx/ext/intersphinx.py:311:43: error: Invalid index type "_T" for "Dict[str, Tuple[str, str, str, str]]"; expected type "str"

edgedb (https://github.com/edgedb/edgedb)
+ edb/schema/utils.py:787:9: error: List comprehension has incompatible type List[Tuple[_T, int]]; expected List[Tuple[Object, int]]
+ edb/schema/utils.py:787:66: error: Argument 2 to "get_nq_name" has incompatible type "_T"; expected "Object"
+ edb/schema/utils.py:793:20: error: Argument 1 to "list" has incompatible type "filter[_T]"; expected "Iterable[_T]"
+ edb/schema/utils.py:799:13: error: Value of type "object" is not indexable
+ edb/schema/utils.py:800:37: error: Value of type "object" is not indexable
+ edb/schema/utils.py:801:13: error: Value of type "object" is not indexable
+ edb/schema/utils.py:805:13: error: Value of type "_T" is not indexable

rclip (https://github.com/yurijmikhalevich/rclip)
+ rclip/main.py:138: error: Value of type "_T" is not indexable

rotki (https://github.com/rotki/rotki)
+ rotkehlchen/chain/ethereum/interfaces/ammswap/ammswap.py:719: error: Incompatible types in assignment (expression has type "_T", variable has type "ChecksumAddress")
+ rotkehlchen/chain/ethereum/modules/sushiswap/sushiswap.py:143: error: Incompatible types in assignment (expression has type "_T", variable has type "ChecksumAddress")
+ rotkehlchen/chain/ethereum/modules/uniswap/uniswap.py:186: error: Incompatible types in assignment (expression has type "_T", variable has type "ChecksumAddress")
+ rotkehlchen/chain/ethereum/modules/uniswap/uniswap.py:296: error: Incompatible types in assignment (expression has type "_T", variable has type "ChecksumAddress")
+ rotkehlchen/externalapis/covalent.py:214: error: Argument 1 to "convert_transaction_from_covalent" has incompatible type "bool"; expected "Dict[str, Any]"
+ rotkehlchen/chain/avalanche/manager.py:67: error: Value of type "bool" is not indexable

ibis (https://github.com/ibis-project/ibis)
+ ibis/backends/csv/__init__.py:78: error: "_T" has no attribute "name"
+ ibis/backends/csv/__init__.py:82: error: "_T" has no attribute "schema"
+ ibis/backends/csv/__init__.py:91: error: "_T" has no attribute "schema"
+ ibis/backends/csv/__init__.py:92: error: Dict entry 0 has incompatible type "_T": "Any"; expected "Node": "Any"

Tanjun (https://github.com/FasterSpeeding/Tanjun)
+ tanjun/conversion.py:646: error: Incompatible types in "yield from" (actual type "bool", expected type "Snowflake")
+ tanjun/conversion.py:650: error: Incompatible types in "yield from" (actual type "bool", expected type "Snowflake")

graphql-core (https://github.com/graphql-python/graphql-core)
+ src/graphql/utilities/print_schema.py:60: error: Argument 1 to "map" has incompatible type "Callable[[GraphQLDirective], str]"; expected "Callable[[bool], str]"
+ src/graphql/utilities/print_schema.py:61: error: Argument 1 to "map" has incompatible type "Callable[[GraphQLNamedType], str]"; expected "Callable[[bool], str]"
+ src/graphql/validation/rules/provided_required_arguments.py:52: error: "bool" has no attribute "name"
+ src/graphql/validation/rules/provided_required_arguments.py:52: error: Value expression in dictionary comprehension has incompatible type "bool"; expected type "Union[GraphQLArgument, InputValueDefinitionNode]"

spark (https://github.com/apache/spark)
+ python/pyspark/pandas/frame.py:2232: error: Argument 1 to "list" has incompatible type "filter[_T]"; expected "Iterable[_T]"  [arg-type]
+ python/pyspark/pandas/frame.py:2237: error: Argument 1 to "loads" has incompatible type "_T"; expected "Union[str, bytes]"  [arg-type]
+ python/pyspark/pandas/frame.py:2245: error: Argument 2 to "scol_for" has incompatible type "_T"; expected "str"  [arg-type]

core (https://github.com/home-assistant/core)
+ homeassistant/components/picnic/coordinator.py:97: error: Argument 1 to "next" has incompatible type "filter[_T]"; expected "SupportsNext[Dict[Any, Any]]"  [arg-type]
+ homeassistant/components/fitbit/sensor.py:408: error: Incompatible types in assignment (expression has type "_T", variable has type "Optional[Dict[str, str]]")  [assignment]
+ homeassistant/components/fitbit/sensor.py:409: error: Argument 1 to "list" has incompatible type "filter[_T]"; expected "Iterable[_T]"  [arg-type]

prefect (https://github.com/PrefectHQ/prefect)
+ src/prefect/core/flow.py:291: error: Argument 1 to "list" has incompatible type "filter[bool]"; expected "Iterable[Task]"
+ src/prefect/core/flow.py:1111: error: "_T" has no attribute "map_states"
+ src/prefect/core/flow.py:1114: error: "_T`2727" object is not iterable
+ src/prefect/core/flow.py:1148: error: "_T" has no attribute "is_cached"
+ src/prefect/core/flow.py:1150: error: "_T" has no attribute "is_mapped"
+ src/prefect/core/flow.py:1151: error: "_T" has no attribute "map_states"
+ src/prefect/core/flow.py:1155: error: "_T" has no attribute "map_states"
+ src/prefect/core/flow.py:1159: error: Incompatible types in assignment (expression has type "List[<nothing>]", variable has type "List[_T]")
+ src/prefect/core/flow.py:1159: note: "List" is invariant -- see https://mypy.readthedocs.io/en/stable/common_issues.html#variance
+ src/prefect/core/flow.py:1159: note: Consider using "Sequence" instead, which is covariant

porcupine (https://github.com/Akuli/porcupine)
+ porcupine/settings.py:916: error: Incompatible return value type (got "List[bool]", expected "List[str]")  [return-value]

dragonchain (https://github.com/dragonchain/dragonchain)
+ dragonchain/lib/dao/api_key_dao.py:89:25: error: Argument 1 to "list" has incompatible type "filter[_T]"; expected "Iterable[_T]"
+ dragonchain/lib/dao/api_key_dao.py:90:28: error: Argument 1 to "list" has incompatible type "filter[_T]"; expected "Iterable[_T]"
+ dragonchain/lib/dao/api_key_dao.py:93:78: error: Argument 1 to "get_json_from_object" has incompatible type "_T"; expected "str"
+ dragonchain/lib/dao/api_key_dao.py:95:5: error: Incompatible types in assignment (expression has type "_T", variable has type "_T")
+ dragonchain/lib/dao/api_key_dao.py:97:27: error: Value of type "_T" is not indexable
+ dragonchain/lib/dao/api_key_dao.py:97:31: error: "_T" has no attribute "find"
+ dragonchain/lib/dao/api_key_dao.py:98:78: error: Argument 1 to "get_json_from_object" has incompatible type "_T"; expected "str"

JelleZijlstra · 2021-12-21T03:12:43Z

Still no good. I'm guessing this is also affected by python/mypy#11428. We can reopen this once mypy's TypeGuard support improves.

seems to be a bug in mypy, see python/typeshed#6140 Signed-off-by: GRBurst <GRBurst@protonmail.com>

sobolevn added 2 commits October 10, 2021 01:08

Adds TypeGuard to filter

86fc066

Refs #5661

Update builtins.pyi

90be622

This comment has been minimized.

Sign in to view

sobolevn mentioned this pull request Oct 10, 2021

bool and TypeGuard incorrect match in @overload method python/mypy#11307

Closed

sobolevn mentioned this pull request Oct 11, 2021

How to disable all pylance rules? microsoft/pylance-release#1925

Closed

JelleZijlstra approved these changes Nov 7, 2021

View reviewed changes

JelleZijlstra closed this Nov 7, 2021

JelleZijlstra reopened this Nov 7, 2021

This comment has been minimized.

Sign in to view

Merge branch 'master' into patch-20

7a5c9f1

This comment has been minimized.

Sign in to view

This was referenced Nov 18, 2021

Issue 1114 dry-python/returns#1131

Open

Please consider making typing filter as good as comprehensions python/mypy#11591

Closed

JelleZijlstra closed this Dec 21, 2021

JelleZijlstra reopened this Dec 21, 2021

JelleZijlstra closed this Dec 21, 2021

layday mentioned this pull request Dec 28, 2021

builtins.filter support for PEP 647 TypeGuard microsoft/pyright#2768

Closed

Jasha10 mentioned this pull request Dec 29, 2021

builtins.filter compat with typing.TypeGuard #6726

Merged

GRBurst added a commit to GRBurst/slack-logger-python that referenced this pull request Jul 17, 2023

chore: rewrite filter to list comprehension to satisfy mypy

1d9ca12

seems to be a bug in mypy, see python/typeshed#6140 Signed-off-by: GRBurst <GRBurst@protonmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adds `TypeGuard` to `filter` #6140

Adds `TypeGuard` to `filter` #6140

sobolevn commented Oct 9, 2021

This comment has been minimized.

JelleZijlstra commented Oct 10, 2021

sobolevn commented Oct 10, 2021

hauntsaninja commented Oct 10, 2021

sobolevn commented Oct 10, 2021

sobolevn commented Oct 10, 2021

sobolevn commented Oct 30, 2021

JelleZijlstra left a comment

This comment has been minimized.

JelleZijlstra commented Nov 14, 2021

This comment has been minimized.

JelleZijlstra commented Nov 14, 2021

JelleZijlstra commented Dec 21, 2021

github-actions bot commented Dec 21, 2021

JelleZijlstra commented Dec 21, 2021

Adds TypeGuard to filter #6140

Adds TypeGuard to filter #6140

Conversation

sobolevn commented Oct 9, 2021

This comment has been minimized.

JelleZijlstra commented Oct 10, 2021

sobolevn commented Oct 10, 2021

hauntsaninja commented Oct 10, 2021

sobolevn commented Oct 10, 2021

sobolevn commented Oct 10, 2021

sobolevn commented Oct 30, 2021

JelleZijlstra left a comment

Choose a reason for hiding this comment

This comment has been minimized.

JelleZijlstra commented Nov 14, 2021

This comment has been minimized.

JelleZijlstra commented Nov 14, 2021

JelleZijlstra commented Dec 21, 2021

github-actions bot commented Dec 21, 2021

JelleZijlstra commented Dec 21, 2021

Adds `TypeGuard` to `filter` #6140

Adds `TypeGuard` to `filter` #6140