Add support for functools.partial #16939

hauntsaninja · 2024-02-23T06:36:09Z

Turns out that this is currently the second most popular mypy issue (and first most popular is a type system feature request that would need a PEP). I'm sure there's stuff missing, but this should handle most cases.

Fixes python#1484 This is currently the most popular mypy issue that does not need a PEP. I'm sure there's stuff missing, but this should handle most cases.

sobolevn

Looks awesome! Thanks for working on this.

I suggest to test that first / second runs work correctly as well.

hauntsaninja · 2024-02-23T08:05:32Z

Looking randomly through primer, looks pretty good:

aiohttp is a true positive (Middleware needs to be typed as callback protocol if you want to pass a kwarg to it)
ibis has a true positive (tests testing that code crashes in test_generic, couldn't really tell if data_types hits were true positive or not)
pip is true positive
CPython is probably a true positive? I think it'd be better off just using dict[str, Any] for those kwargs
tornado is true positive
graphql, extend_schema is true positive (mypy prevents passing TypedDict to dict[str, Any], hopefully PEP 705 will help), tests/validation is true positive (dict is not a GraphQLError)

JukkaL

Excellent! This is a really old feature request and many users will be happy about this. Didn't do a full review yet, but left some ideas about test cases.

JukkaL · 2024-02-23T16:11:12Z

test-data/unit/check-functools.test

+p1 = functools.partial(foo)
+p1(1, "a", 3)  # OK
+p1(1, "a", c=3)  # OK
+p1(1, b="a", c=3)  # OK


What about using reveal_type(p1) etc. in tests?

I added one of these, it's not very useful because it'll just show you partial[return_type]... would need a custom mypy type / some very involved generics logic / non-standard features to make it show up in the reveal

Ah makes sense. The current approach seems fine then.

JukkaL · 2024-02-23T16:16:23Z

test-data/unit/check-functools.test

+def foo(fn: typing.Callable[[typing.Unpack[Ts]], None], /, *arg: typing.Unpack[Ts], kwarg: str) -> None: ...
+# just test that this doesn't crash
+functools.partial(foo, kwarg="asdf")
+[builtins fixtures/dict.pyi]


Ideas for test cases (it's not important to support all of these cases precisely, but they at least shouldn't crash or behave erratically):

Test with a value of type[x] as an argument to partial.

Test with a class reference as an argument to partial (e.g. partial(Foo)). This is different from the above.

Test calling partial with *args and/or **kwargs.

Test passing Callable[..., Foo] to partial (with explicit ...).

Test passing a type guard to partial.

Test passing an instance that has a __call__ method to partial.

First, thank you for working on this issue 🙏
FWIW pyright implemented partial as a special case here and the accompanying test cases may be useful for ideas. For example, there are tests specific to an overloaded function being passed.

Pyright Tests:

https://github.com/microsoft/pyright/blob/main/packages/pyright-internal/src/tests/samples/partial1.py

https://github.com/microsoft/pyright/blob/main/packages/pyright-internal/src/tests/samples/partial2.py

https://github.com/microsoft/pyright/blob/main/packages/pyright-internal/src/tests/samples/partial3.py

https://github.com/microsoft/pyright/blob/main/packages/pyright-internal/src/tests/samples/partial4.py

https://github.com/microsoft/pyright/blob/main/packages/pyright-internal/src/tests/samples/partial5.py

Thank you for the review! Added the various tests and some small fixes. The TypeGuard one surfaces a pre-existing bug, I can make a separate PR edit: already fixed by TypeIs PR. There's also some work to be done with overloads, but similarly gets into some pre-existing behaviour

hauntsaninja · 2024-03-01T08:05:00Z

Looks like callback protocol support hurts overloads more than previously. Probably need to improve extract_callable_type

github-actions · 2024-03-01T20:31:07Z

Diff from mypy_primer, showing the effect of this PR on open source code:

anyio (https://github.com/agronholm/anyio)
+ src/anyio/_backends/_asyncio.py:2259: error: Argument 1 to "_forcibly_shutdown_process_pool_on_exit" has incompatible type "set[anyio.abc._subprocesses.Process]"; expected "set[anyio._backends._asyncio.Process]"  [arg-type]

pip (https://github.com/pypa/pip)
+ src/pip/_internal/utils/misc.py:136: error: "Callable[[FunctionType, Path, BaseException], Any] | Callable[[FunctionType, Path, tuple[type[BaseException], BaseException, TracebackType]], Any]" not callable  [misc]
+ src/pip/_internal/cli/progress_bars.py:66: error: Argument 2 to "_rich_progress_bar" has incompatible type "int | None"; expected "int"  [arg-type]

CPython (Argument Clinic) (https://github.com/python/cpython)
+ Tools/clinic/clinic.py:2799: error: "CConverterClassT" not callable  [misc]

porcupine (https://github.com/Akuli/porcupine)
+ porcupine/plugins/git_status.py:139: error: Argument 1 to "run_git_status" has incompatible type "Path | None"; expected "Path"  [arg-type]

pandas (https://github.com/pandas-dev/pandas)
+ pandas/core/window/rolling.py:1499: error: Argument 3 to "roll_apply" has incompatible type "bool | bool_"; expected "bool"  [arg-type]
+ pandas/core/tools/datetimes.py:997: error: Argument 6 to "_convert_listlike_datetimes" has incompatible type "bool | Literal[_NoDefault.no_default]"; expected "bool"  [arg-type]
+ pandas/core/groupby/groupby.py:3843: error: Argument "mask" to "group_fillna_indexer" has incompatible type "ndarray[Any, dtype[bool_]] | Any"; expected "ndarray[Any, dtype[unsignedinteger[_8Bit]]]"  [arg-type]
+ pandas/core/groupby/groupby.py:4368: error: Argument 3 to "group_quantile" has incompatible type "str"; expected "Literal['linear', 'lower', 'higher', 'nearest', 'midpoint']"  [arg-type]
+ pandas/core/groupby/groupby.py:4400: error: Argument "mask" to "group_quantile" has incompatible type "ndarray[Any, dtype[bool_]]"; expected "ndarray[Any, dtype[unsignedinteger[_8Bit]]]"  [arg-type]
+ pandas/tests/frame/test_ufunc.py:69: error: "_UFunc_Nin2_Nout1[Literal['add'], Literal[22], Literal[0]]" not callable  [misc]
+ pandas/tests/frame/test_ufunc.py:76: error: "_UFunc_Nin1_Nout1[Literal['negative'], Literal[19], None]" not callable  [misc]
+ pandas/tests/indexes/test_base.py:135: error: Unexpected keyword argument "data" for "RangeIndex"  [call-arg]
+ pandas/core/indexes/range.py:135: note: "RangeIndex" defined here

websockets (https://github.com/aaugustin/websockets)
+ src/websockets/legacy/server.py:1048: error: "type[WebSocketServerProtocol] | Callable[..., WebSocketServerProtocol]" not callable  [misc]
+ src/websockets/legacy/client.py:496: error: "type[WebSocketClientProtocol] | Callable[..., WebSocketClientProtocol]" not callable  [misc]
+ src/websockets/legacy/client.py:649: error: Redundant cast to "WebSocketClientProtocol"  [redundant-cast]
+ src/websockets/legacy/auth.py:180: error: "type[BasicAuthWebSocketServerProtocol] | Callable[..., BasicAuthWebSocketServerProtocol]" not callable  [misc]

prefect (https://github.com/PrefectHQ/prefect)
+ src/prefect/engine.py:1611: error: Argument 3 to "create_task_run_then_submit" has incompatible type "int"; expected "str"  [arg-type]
+ src/prefect/cli/worker.py:206: error: Argument 2 to "start_healthcheck_server" has incompatible type "float"; expected "int"  [arg-type]

koda-validate (https://github.com/keithasaurus/koda-validate)
+ koda_validate/signature.py:173: error: Argument 1 to "_get_validator" has incompatible type "tuple[Literal['return_key']]"; expected "str"  [arg-type]
- koda_validate/signature.py:384: error: Returning Any from function declared to return "_DecoratedFunc"  [no-any-return]
- koda_validate/signature.py:388: error: Returning Any from function declared to return "_DecoratedFunc | Callable[[_DecoratedFunc], _DecoratedFunc]"  [no-any-return]

isort (https://github.com/pycqa/isort)
+ isort/exceptions.py:13: error: "Type[ISortError]" not callable  [misc]

tornado (https://github.com/tornadoweb/tornado)
+ tornado/tcpclient.py:274: error: Argument 1 to "_create_stream" of "TCPClient" has incompatible type "Optional[int]"; expected "int"  [arg-type]
+ tornado/curl_httpclient.py:362: error: Argument 2 to "_curl_header_callback" of "CurlAsyncHTTPClient" has incompatible type "Optional[Callable[[str], None]]"; expected "Callable[[str], None]"  [arg-type]

vision (https://github.com/pytorch/vision)
+ torchvision/models/video/swin_transformer.py:518: error: Argument 2 to "VideoClassification" has incompatible type "tuple[int]"; expected "tuple[int, int]"  [arg-type]
+ torchvision/models/video/swin_transformer.py:549: error: Argument 2 to "VideoClassification" has incompatible type "tuple[int]"; expected "tuple[int, int]"  [arg-type]
+ torchvision/models/video/swin_transformer.py:580: error: Argument 2 to "VideoClassification" has incompatible type "tuple[int]"; expected "tuple[int, int]"  [arg-type]
+ torchvision/models/video/swin_transformer.py:607: error: Argument 2 to "VideoClassification" has incompatible type "tuple[int]"; expected "tuple[int, int]"  [arg-type]
+ torchvision/models/video/mvit.py:607: error: Argument 2 to "VideoClassification" has incompatible type "tuple[int]"; expected "tuple[int, int]"  [arg-type]
+ torchvision/models/video/mvit.py:640: error: Argument 2 to "VideoClassification" has incompatible type "tuple[int]"; expected "tuple[int, int]"  [arg-type]

kopf (https://github.com/nolar/kopf)
+ kopf/_core/actions/invocation.py:127: error: Incompatible types in assignment (expression has type "partial[partial[object | Coroutine[None, None, object]]]", variable has type "partial[object | Coroutine[None, None, object | None] | None]")  [assignment]
+ kopf/_core/actions/invocation.py:127: error: Argument 1 to "run" of "Context" has incompatible type "partial[object | Coroutine[None, None, object | None] | None]"; expected "Callable[[VarArg(Any), KwArg(Any)], partial[object | Coroutine[None, None, object]]]"  [arg-type]
+ kopf/_core/actions/invocation.py:127: note: "partial[object | Coroutine[None, None, object | None] | None].__call__" has type "Callable[[VarArg(Any), KwArg(Any)], object | Coroutine[None, None, object | None] | None]"
+ kopf/_core/reactor/orchestration.py:42: error: Unexpected keyword argument "resource" for "__call__" of "WatchStreamProcessor"  [call-arg]
+ kopf/_core/reactor/queueing.py:42: note: "__call__" of "WatchStreamProcessor" defined here

graphql-core (https://github.com/graphql-python/graphql-core)
+ src/graphql/utilities/extend_schema.py:360: error: Argument 1 to "extend_input_object_type_fields" of "ExtendSchemaImpl" has incompatible type "GraphQLInputObjectTypeKwargs"; expected "dict[str, Any]"  [arg-type]
+ src/graphql/utilities/extend_schema.py:427: error: Argument 1 to "extend_object_type_interfaces" of "ExtendSchemaImpl" has incompatible type "GraphQLObjectTypeKwargs"; expected "dict[str, Any]"  [arg-type]
+ src/graphql/utilities/extend_schema.py:429: error: Argument 1 to "extend_object_type_fields" of "ExtendSchemaImpl" has incompatible type "GraphQLObjectTypeKwargs"; expected "dict[str, Any]"  [arg-type]
+ src/graphql/utilities/extend_schema.py:467: error: Argument 1 to "extend_interface_type_interfaces" of "ExtendSchemaImpl" has incompatible type "GraphQLInterfaceTypeKwargs"; expected "dict[str, Any]"  [arg-type]
+ src/graphql/utilities/extend_schema.py:469: error: Argument 1 to "extend_interface_type_fields" of "ExtendSchemaImpl" has incompatible type "GraphQLInterfaceTypeKwargs"; expected "dict[str, Any]"  [arg-type]
+ src/graphql/utilities/extend_schema.py:491: error: Argument 1 to "extend_union_type_types" of "ExtendSchemaImpl" has incompatible type "GraphQLUnionTypeKwargs"; expected "dict[str, Any]"  [arg-type]
+ tests/validation/test_variables_in_allowed_position.py:174: error: List item 0 has incompatible type "dict[str, Sequence[Sequence[object]]]"; expected "GraphQLError"  [list-item]
+ tests/validation/test_variables_in_allowed_position.py:196: error: List item 0 has incompatible type "dict[str, Sequence[Sequence[object]]]"; expected "GraphQLError"  [list-item]
+ tests/validation/test_variables_in_allowed_position.py:222: error: List item 0 has incompatible type "dict[str, Sequence[Sequence[object]]]"; expected "GraphQLError"  [list-item]
+ tests/validation/test_variables_in_allowed_position.py:240: error: List item 0 has incompatible type "dict[str, Sequence[Sequence[object]]]"; expected "GraphQLError"  [list-item]
+ tests/validation/test_variables_in_allowed_position.py:258: error: List item 0 has incompatible type "dict[str, Sequence[Sequence[object]]]"; expected "GraphQLError"  [list-item]
+ tests/validation/test_variables_in_allowed_position.py:274: error: List item 0 has incompatible type "dict[str, Sequence[Sequence[object]]]"; expected "GraphQLError"  [list-item]
+ tests/validation/test_variables_in_allowed_position.py:290: error: List item 0 has incompatible type "dict[str, Sequence[Sequence[object]]]"; expected "GraphQLError"  [list-item]
+ tests/validation/test_variables_in_allowed_position.py:309: error: List item 0 has incompatible type "dict[str, Sequence[Sequence[object]]]"; expected "GraphQLError"  [list-item]
+ tests/validation/test_variables_in_allowed_position.py:328: error: List item 0 has incompatible type "dict[str, Sequence[Sequence[object]]]"; expected "GraphQLError"  [list-item]
+ tests/validation/test_variables_are_input_types.py:39: error: List item 0 has incompatible type "dict[str, Sequence[Sequence[object]]]"; expected "GraphQLError"  [list-item]
+ tests/validation/test_variables_are_input_types.py:43: error: List item 1 has incompatible type "dict[str, Sequence[Sequence[object]]]"; expected "GraphQLError"  [list-item]
+ tests/validation/test_variables_are_input_types.py:48: error: List item 2 has incompatible type "dict[str, Sequence[Sequence[object]]]"; expected "GraphQLError"  [list-item]
+ tests/validation/test_values_of_correct_type.py:176: error: List item 0 has incompatible type "dict[str, Sequence[Sequence[object]]]"; expected "GraphQLError"  [list-item]
+ tests/validation/test_values_of_correct_type.py:193: error: List item 0 has incompatible type "dict[str, Sequence[Sequence[object]]]"; expected "GraphQLError"  [list-item]
+ tests/validation/test_values_of_correct_type.py:210: error: List item 0 has incompatible type "dict[str, Sequence[Sequence[object]]]"; expected "GraphQLError"  [list-item]
+ tests/validation/test_values_of_correct_type.py:227: error: List item 0 has incompatible type "dict[str, Sequence[Sequence[object]]]"; expected "GraphQLError"  [list-item]
+ tests/validation/test_values_of_correct_type.py:245: error: List item 0 has incompatible type "dict[str, Sequence[Sequence[object]]]"; expected "GraphQLError"  [list-item]
+ tests/validation/test_values_of_correct_type.py:262: error: List item 0 has incompatible type "dict[str, Sequence[Sequence[object]]]"; expected "GraphQLError"  [list-item]
+ tests/validation/test_values_of_correct_type.py:280: error: List item 0 has incompatible type "dict[str, Sequence[Sequence[object]]]"; expected "GraphQLError"  [list-item]
+ tests/validation/test_values_of_correct_type.py:297: error: List item 0 has incompatible type "dict[str, Sequence[Sequence[object]]]"; expected "GraphQLError"  [list-item]
+ tests/validation/test_values_of_correct_type.py:314: error: List item 0 has incompatible type "dict[str, Sequence[Sequence[object]]]"; expected "GraphQLError"  [list-item]
+ tests/validation/test_values_of_correct_type.py:332: error: List item 0 has incompatible type "dict[str, Sequence[Sequence[object]]]"; expected "GraphQLError"  [list-item]
+ tests/validation/test_values_of_correct_type.py:349: error: List item 0 has incompatible type "dict[str, Sequence[Sequence[object]]]"; expected "GraphQLError"  [list-item]
+ tests/validation/test_values_of_correct_type.py:366: error: List item 0 has incompatible type "dict[str, Sequence[Sequence[object]]]"; expected "GraphQLError"  [list-item]
+ tests/validation/test_values_of_correct_type.py:384: error: List item 0 has incompatible type "dict[str, Sequence[Sequence[object]]]"; expected "GraphQLError"  [list-item]
+ tests/validation/test_values_of_correct_type.py:401: error: List item 0 has incompatible type "dict[str, Sequence[Sequence[object]]]"; expected "GraphQLError"  [list-item]
+ tests/validation/test_values_of_correct_type.py:418: error: List item 0 has incompatible type "dict[str, Sequence[Sequence[object]]]"; expected "GraphQLError"  [list-item]
+ tests/validation/test_values_of_correct_type.py:436: error: List item 0 has incompatible type "dict[str, Sequence[Sequence[object]]]"; expected "GraphQLError"  [list-item]
+ tests/validation/test_values_of_correct_type.py:454: error: List item 0 has incompatible type "dict[str, Sequence[Sequence[object]]]"; expected "GraphQLError"  [list-item]
+ tests/validation/test_values_of_correct_type.py:472: error: List item 0 has incompatible type "dict[str, Sequence[Sequence[object]]]"; expected "GraphQLError"  [list-item]
+ tests/validation/test_values_of_correct_type.py:490: error: List item 0 has incompatible type "dict[str, Sequence[Sequence[object]]]"; expected "GraphQLError"  [list-item]
+ tests/validation/test_values_of_correct_type.py:509: error: List item 0 has incompatible type "dict[str, Sequence[Sequence[object]]]"; expected "GraphQLError"  [list-item]
+ tests/validation/test_values_of_correct_type.py:527: error: List item 0 has incompatible type "dict[str, Sequence[Sequence[object]]]"; expected "GraphQLError"  [list-item]
+ tests/validation/test_values_of_correct_type.py:545: error: List item 0 has incompatible type "dict[str, Sequence[Sequence[object]]]"; expected "GraphQLError"  [list-item]
+ tests/validation/test_values_of_correct_type.py:564: error: List item 0 has incompatible type "dict[str, Sequence[Sequence[object]]]"; expected "GraphQLError"  [list-item]
+ tests/validation/test_values_of_correct_type.py:582: error: List item 0 has incompatible type "dict[str, Sequence[Sequence[object]]]"; expected "GraphQLError"  [list-item]
+ tests/validation/test_values_of_correct_type.py:600: error: List item 0 has incompatible type "dict[str, Sequence[Sequence[object]]]"; expected "GraphQLError"  [list-item]
+ tests/validation/test_values_of_correct_type.py:664: error: List item 0 has incompatible type "dict[str, Sequence[Sequence[object]]]"; expected "GraphQLError"  [list-item]
+ tests/validation/test_values_of_correct_type.py:681: error: List item 0 has incompatible type "dict[str, Sequence[Sequence[object]]]"; expected "GraphQLError"  [list-item]
+ tests/validation/test_values_of_correct_type.py:810: error: List item 0 has incompatible type "dict[str, Sequence[Sequence[object]]]"; expected "GraphQLError"  [list-item]
+ tests/validation/test_values_of_correct_type.py:814: error: List item 1 has incompatible type "dict[str, Sequence[Sequence[object]]]"; expected "GraphQLError"  [list-item]
+ tests/validation/test_values_of_correct_type.py:831: error: List item 0 has incompatible type "dict[str, Sequence[Sequence[object]]]"; expected "GraphQLError"  [list-item]
+ tests/validation/test_values_of_correct_type.py:848: error: List item 0 has incompatible type "dict[str, Sequence[Sequence[object]]]"; expected "GraphQLError"  [list-item]
+ tests/validation/test_values_of_correct_type.py:945: error: List item 0 has incompatible type "dict[str, Sequence[Sequence[object]]]"; expected "GraphQLError"  [list-item]
+ tests/validation/test_values_of_correct_type.py:966: error: List item 0 has incompatible type "dict[str, Sequence[Sequence[object]]]"; expected "GraphQLError"  [list-item]
+ tests/validation/test_values_of_correct_type.py:986: error: List item 0 has incompatible type "dict[str, Sequence[Sequence[object]]]"; expected "GraphQLError"  [list-item]
+ tests/validation/test_values_of_correct_type.py:1006: error: List item 0 has incompatible type "dict[str, Sequence[Sequence[object]]]"; expected "GraphQLError"  [list-item]
+ tests/validation/test_values_of_correct_type.py:1035: error: List item 0 has incompatible type "dict[str, Sequence[Sequence[object]]]"; expected "GraphQLError"  [list-item]
+ tests/validation/test_values_of_correct_type.py:1067: error: List item 0 has incompatible type "dict[str, Sequence[Sequence[object]]]"; expected "GraphQLError"  [list-item]

... (truncated 428 lines) ...

jax (https://github.com/google/jax)
+ jax/_src/linear_util.py:339: error: Missing positional argument "tree" in call to "tree_map"  [call-arg]
+ jax/_src/interpreters/partial_eval.py:2652: error: Argument 2 to "_eval_jaxpr_padded" has incompatible type "Sequence[Any]"; expected "list[Any]"  [arg-type]
+ jax/_src/custom_derivatives.py:245: error: Missing positional argument "tree" in call to "tree_map"  [call-arg]
+ jax/_src/nn/functions.py:483: error: Argument 1 to "custom_jvp" has incompatible type "Callable[[Array | Any | Any | Any | bool | int | float | complex, int | tuple[int, ...] | None, Array | Any | Any | Any | bool | int | float | complex | None, Array | Any | Any | Any | bool | int | float | complex | None], Array]"; expected "Callable[..., Never]"  [arg-type]
+ jax/_src/scipy/special.py:579: error: Argument 1 to "custom_jvp" has incompatible type "Callable[[Array | Any | Any | Any | bool | int | float | complex, int], Array]"; expected "Callable[..., Never]"  [arg-type]
+ jax/_src/scipy/special.py:1601: error: Argument 1 to "_expn3" has incompatible type "Array"; expected "int"  [arg-type]
+ jax/_src/scipy/special.py:1602: error: Argument 1 to "_expn2" has incompatible type "Array"; expected "int"  [arg-type]
+ jax/_src/scipy/special.py:1603: error: Argument 1 to "_expn1" has incompatible type "Array"; expected "int"  [arg-type]
+ jax/_src/scipy/signal.py:430: error: Argument 1 to "detrend" has incompatible type "Literal[True] | str"; expected "str"  [arg-type]
+ jax/experimental/rnn.py:245: error: Argument 1 to "custom_vjp" has incompatible type "Callable[[Array, Array, Array, Array, Array, int, int, int, float, bool, str | Precision | tuple[str, str] | tuple[Precision, Precision] | None], tuple[Array, Array, Array]]"; expected "Callable[..., Never]"  [arg-type]
+ jax/_src/pallas/pallas_call.py:421: error: Argument 1 to "_batch_block_mapping" has incompatible type "tuple[int | None, ...]"; expected "tuple[int, ...]"  [arg-type]
+ jax/_src/cudnn/fused_attention_stablehlo.py:747: error: Need type annotation for "output"  [var-annotated]
+ jax/experimental/pallas/ops/rms_norm.py:225: error: Argument 1 to "custom_vjp" has incompatible type "list[int]"; expected "tuple[int, ...]"  [arg-type]
+ jax/experimental/pallas/ops/layer_norm.py:244: error: Argument 1 to "custom_vjp" has incompatible type "list[int]"; expected "tuple[int, ...]"  [arg-type]
+ jax/experimental/pallas/ops/attention.py:149: error: Argument 1 to "custom_vjp" has incompatible type "list[int]"; expected "tuple[int, ...]"  [arg-type]
+ jax/experimental/pallas/ops/tpu/flash_attention.py:203: error: Argument 1 to "custom_vjp" has incompatible type "range"; expected "tuple[int, ...]"  [arg-type]
+ jax/experimental/pallas/ops/tpu/flash_attention.py:1088: error: Argument 1 to "_flash_attention_dkv_kernel" has incompatible type "int | None"; expected "int"  [arg-type]
+ jax/experimental/pallas/ops/tpu/flash_attention.py:1089: error: Argument 2 to "_flash_attention_dkv_kernel" has incompatible type "int | None"; expected "int"  [arg-type]
+ jax/experimental/pallas/ops/tpu/flash_attention.py:1432: error: Argument 4 to "_flash_attention_dq_kernel" has incompatible type "int | None"; expected "int"  [arg-type]
+ jax/experimental/pallas/ops/tpu/flash_attention.py:1568: error: Need type annotation for "res"  [var-annotated]

cwltool (https://github.com/common-workflow-language/cwltool)
+ cwltool/workflow_job.py: note: In member "try_make_job" of class "WorkflowJob":
+ cwltool/workflow_job.py:616:57: error: Argument 3 to "receive_output" of "WorkflowJob" has incompatible type "OutputCallbackType | None"; expected "OutputCallbackType"  [arg-type]
+ cwltool/command_line_tool.py: note: In member "job" of class "CommandLineTool":
+ cwltool/command_line_tool.py:960:52: error: Argument 1 to "update_status_output_callback" has incompatible type "OutputCallbackType | None"; expected "OutputCallbackType"  [arg-type]
+ cwltool/workflow.py: note: In member "job" of class "WorkflowStep":
+ cwltool/workflow.py:463:56: error: Argument 1 to "receive_output" of "WorkflowStep" has incompatible type "OutputCallbackType | None"; expected "OutputCallbackType"  [arg-type]

aiohttp (https://github.com/aio-libs/aiohttp)
+ aiohttp/web_app.py:56:37: error: Unexpected keyword argument "handler"  [call-arg]
+ aiohttp/web_app.py:56:37: note: See https://mypy.rtfd.io/en/stable/_refs.html#code-call-arg for more info

ibis (https://github.com/ibis-project/ibis)
+ ibis/expr/operations/tests/test_generic.py:147: error: Too many arguments for "Literal"  [call-arg]
+ ibis/expr/operations/tests/test_generic.py:147: error: Unexpected keyword argument "name" for "Literal"  [call-arg]
+ ibis/expr/operations/generic.py:147: note: "Literal" defined here
+ ibis/expr/operations/tests/test_generic.py:147: error: "Literal" gets multiple values for keyword argument "dtype"  [misc]
+ ibis/backends/sql/datatypes.py:31: error: Argument 1 to "Map" has incompatible type "String"; expected "bool"  [arg-type]
+ ibis/backends/sql/datatypes.py:48: error: Argument 1 to "Map" has incompatible type "String"; expected "bool"  [arg-type]
+ ibis/backends/sql/datatypes.py:66: error: Argument 1 to "Array" has incompatible type "String"; expected "bool"  [arg-type]

bokeh (https://github.com/bokeh/bokeh)
+ src/bokeh/driving.py: note: In function "bounce":
+ src/bokeh/driving.py:97:36: error: Argument 1 to "force" has incompatible type "Iterable[int]"; expected "Iterator[Any]"  [arg-type]
+ src/bokeh/driving.py:97:36: note: "Iterable" is missing following "Iterator" protocol member:
+ src/bokeh/driving.py:97:36: note:     __next__
+ src/bokeh/driving.py: note: In function "cosine":
+ src/bokeh/driving.py:116:36: error: Argument 1 to "force" has incompatible type "Iterable[float]"; expected "Iterator[Any]"  [arg-type]
+ src/bokeh/driving.py:116:36: note: "Iterable" is missing following "Iterator" protocol member:
+ src/bokeh/driving.py:116:36: note:     __next__
+ src/bokeh/driving.py: note: In function "count":
+ src/bokeh/driving.py:122:36: error: Argument 1 to "force" has incompatible type "Iterable[int]"; expected "Iterator[Any]"  [arg-type]
+ src/bokeh/driving.py:122:36: note: "Iterable" is missing following "Iterator" protocol member:
+ src/bokeh/driving.py:122:36: note:     __next__
+ src/bokeh/driving.py: note: In function "linear":
+ src/bokeh/driving.py:154:36: error: Argument 1 to "force" has incompatible type "Iterable[float]"; expected "Iterator[Any]"  [arg-type]
+ src/bokeh/driving.py:154:36: note: "Iterable" is missing following "Iterator" protocol member:
+ src/bokeh/driving.py:154:36: note:     __next__
+ src/bokeh/driving.py: note: In function "repeat":
+ src/bokeh/driving.py:172:36: error: Argument 1 to "force" has incompatible type "Iterable[int]"; expected "Iterator[Any]"  [arg-type]
+ src/bokeh/driving.py:172:36: note: "Iterable" is missing following "Iterator" protocol member:
+ src/bokeh/driving.py:172:36: note:     __next__
+ src/bokeh/driving.py: note: In function "sine":
+ src/bokeh/driving.py:191:36: error: Argument 1 to "force" has incompatible type "Iterable[float]"; expected "Iterator[Any]"  [arg-type]
+ src/bokeh/driving.py:191:36: note: "Iterable" is missing following "Iterator" protocol member:
+ src/bokeh/driving.py:191:36: note:     __next__

hydra-zen (https://github.com/mit-ll-responsible-ai/hydra-zen)
+ src/hydra_zen/wrapper/_implementations.py:58: error: Too few arguments for "run" of "Context"  [call-arg]

JukkaL

I found some potential issues, but overall looks good. At least the incremental mode bug seems important to fix, since this will likely cause issues.

JukkaL · 2024-03-08T14:24:28Z

mypy/plugins/functools.py

+    )
+
+    ret = ctx.api.named_generic_type("functools.partial", [ret_type])
+    ret = ret.copy_with_extra_attr("__mypy_partial", partially_applied)


extra_attrs is currently a bit poorly thought out, so this will cause some problems. First, it isn't serialized, so storing the additional information there breaks in incremental mode. For example, consider a module like this:

... p = partial(f, 1)

Now if we import p from another module, the second run which uses serialized data will produce a different type for p compared to the first run.

I'm not sure what is the best way to fix this. Probably the simplest option would be to serialize extra_attrs. It seems that everything we put there can be serialized -- it just hasn't been implemented.

Also it would be good to have an incremental mode test case.

It looks like mypy daemon doesn't keep track of extra_attrs. mypy.server.astdiff should look into extra_attrs to detect changes in extra_attrs, as these may need to be propagated.

JukkaL · 2024-03-08T14:42:04Z

mypy/plugins/functools.py

+        if len(bound.arg_types) == len(fn_type.arg_types):
+            arg_type = bound.arg_types[i]
+            if isinstance(get_proper_type(arg_type), UninhabitedType):
+                arg_type = fn_type.arg_types[i]  # bit of a hack


Hmm this could refer to a type variable type, so I wonder if this can leak type variables, in the target function is a generic one, and one of the provided arguments can be used to bind the type variable.

JukkaL · 2024-03-08T14:42:52Z

mypy/plugins/functools.py

+        else:
+            # TODO: I assume that bound and fn_type have the same arguments. It appears this isn't
+            # true when PEP 646 things are happening. See testFunctoolsPartialTypeVarTuple
+            arg_type = fn_type.arg_types[i]


Similar to above.

JukkaL · 2024-03-08T14:44:00Z

mypy/plugins/functools.py

+
+    ret_type = bound.ret_type
+    if isinstance(get_proper_type(ret_type), UninhabitedType):
+        ret_type = fn_type.ret_type  # same kind of hack as above


Similar to above -- it seems that this might leak type variables. If that is the case, it would probably be better to fall back to the default return type.

I think this doesn't leak type variables, because partially_applied has the same variables as fn_type (from copy_modified). partially_applied just remains generic. See the test cases in testFunctoolsPartialGeneric. I can add some more comments to the code

But let me know if I'm off the mark!

I think you are right, and this is good already.

JukkaL · 2024-03-08T14:45:26Z

test-data/unit/check-functools.test

+p1 = functools.partial(foo)
+p1(1, "a", 3)  # OK
+p1(1, "a", c=3)  # OK
+p1(1, b="a", c=3)  # OK


Ah makes sense. The current approach seems fine then.

kevxwan-zv · 2024-04-16T03:01:02Z

Hey folks, this looks amazing, but when are you planning to release the change to mypy?

hauntsaninja requested a review from sobolevn February 23, 2024 06:37

This comment has been minimized.

Sign in to view

Add support for functools.partial

5b56460

Fixes python#1484 This is currently the most popular mypy issue that does not need a PEP. I'm sure there's stuff missing, but this should handle most cases.

hauntsaninja force-pushed the functools-partial branch from ce0d0b6 to 5b56460 Compare February 23, 2024 07:05

This comment has been minimized.

Sign in to view

sobolevn reviewed Feb 23, 2024

View reviewed changes

hauntsaninja added 2 commits February 22, 2024 23:36

fix mypyc

ff4914f

fix crash

c0084eb

This comment has been minimized.

Sign in to view

hauntsaninja marked this pull request as ready for review February 23, 2024 08:06

sobolevn mentioned this pull request Feb 23, 2024

Remove ConverterKeywordDict alias in clinic.py python/cpython#115843

Merged

fix context

a847234

This comment has been minimized.

Sign in to view

JukkaL reviewed Feb 23, 2024

View reviewed changes

hauntsaninja added 3 commits February 29, 2024 21:43

test more unpack support

2a5f3f1

fix callback protocol, fix line numbers

325cae4

more tests

1ee376d

hauntsaninja mentioned this pull request Mar 1, 2024

Implement TypeIs (PEP 742) #16898

Merged

This comment has been minimized.

Sign in to view

Merge branch 'master' into functools-partial

b7ca434

This comment has been minimized.

Sign in to view

hauntsaninja added 2 commits March 1, 2024 11:31

fix typeguard

03b397e

ignore overload, as previously

d2886ac

This comment has been minimized.

Sign in to view

.

65e356a

hauntsaninja added the upnext label Mar 7, 2024

JukkaL reviewed Mar 8, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for functools.partial #16939

Add support for functools.partial #16939

hauntsaninja commented Feb 23, 2024 •

edited

This comment has been minimized.

This comment has been minimized.

sobolevn left a comment •

edited

This comment has been minimized.

hauntsaninja commented Feb 23, 2024 •

edited

This comment has been minimized.

JukkaL left a comment

JukkaL Feb 23, 2024

hauntsaninja Mar 1, 2024

JukkaL Mar 8, 2024

JukkaL Feb 23, 2024

kbehlers Feb 23, 2024

hauntsaninja Mar 1, 2024 •

edited

This comment has been minimized.

hauntsaninja commented Mar 1, 2024

This comment has been minimized.

This comment has been minimized.

github-actions bot commented Mar 1, 2024

JukkaL left a comment

JukkaL Mar 8, 2024

JukkaL Mar 8, 2024

JukkaL Mar 8, 2024

JukkaL Mar 8, 2024

hauntsaninja Mar 9, 2024 •

edited

JukkaL Mar 13, 2024

JukkaL Mar 8, 2024

kevxwan-zv commented Apr 16, 2024

Add support for functools.partial #16939

Are you sure you want to change the base?

Add support for functools.partial #16939

Conversation

hauntsaninja commented Feb 23, 2024 • edited

This comment has been minimized.

This comment has been minimized.

sobolevn left a comment • edited

Choose a reason for hiding this comment

This comment has been minimized.

hauntsaninja commented Feb 23, 2024 • edited

This comment has been minimized.

JukkaL left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hauntsaninja Mar 1, 2024 • edited

Choose a reason for hiding this comment

This comment has been minimized.

hauntsaninja commented Mar 1, 2024

This comment has been minimized.

This comment has been minimized.

github-actions bot commented Mar 1, 2024

JukkaL left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hauntsaninja Mar 9, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kevxwan-zv commented Apr 16, 2024

hauntsaninja commented Feb 23, 2024 •

edited

sobolevn left a comment •

edited

hauntsaninja commented Feb 23, 2024 •

edited

hauntsaninja Mar 1, 2024 •

edited

hauntsaninja Mar 9, 2024 •

edited