Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

functools.partial support #1484

Open
berdario opened this issue May 5, 2016 · 30 comments · May be fixed by #16939
Open

functools.partial support #1484

berdario opened this issue May 5, 2016 · 30 comments · May be fixed by #16939
Labels

Comments

@berdario
Copy link

berdario commented May 5, 2016

I haven't been able to find another issue for this, weird that this hasn't been reported already.

from typing import *
from functools import partial

def inc(a:int) -> int:
    return a+1

def foo(f: Callable[[], int]) -> int:
    return f()

foo(partial(inc, 1))

This will fail with

error: Argument 1 to "foo" has incompatible type partial[int]; expected Callable[[], int]

a similar code by using a dyadic add instead of inc will yield:

error: Argument 1 to "foo" has incompatible type partial[int]; expected Callable[[int], int]

(so, partial apparently doesn't currently retain any type information on the non-applied inputs... I guess this'll make the fix non trivial)

@JukkaL
Copy link
Collaborator

JukkaL commented May 5, 2016

The error message is generated because a class with a __call__ method (in this case, partial) isn't considered a subtype of Callable, even though it should be. See #797.

Also, partial only retains information about the return type, since the type system can't represent a more precise type.

@berdario
Copy link
Author

berdario commented May 5, 2016

the type system can't represent a more precise type.

You mean that the partial class doesn't represent it at runtime, right?
I think mypy could special case partial, and coerce it to the appropriate Callable type transparently

@JukkaL
Copy link
Collaborator

JukkaL commented May 5, 2016

I meant that mypy would have to special case partial, since we can't write a good enough stub using PEP 484 features only. Mypy already does some special casing, but for this I'd rather have a more general mypy plugin/extension system instead of even more ad-hoc special case logic in the type checker.

@gvanrossum
Copy link
Member

Maybe this could be supported by #3299

@JukkaL
Copy link
Collaborator

JukkaL commented Jun 13, 2017

Yes, this looks like a good candidate for function plugins. Increasing priority since this is may be pretty easy to implement now. Just supporting positional arguments for partial should cover the majority of uses and would probably be good enough.

@ilevkivskyi
Copy link
Member

This particular case and probably other basic use cases are already supported by protocols (see tests in PR #3132), simply because partial has __call__ and therefore is a structural subtype of Callable.

@JukkaL
Copy link
Collaborator

JukkaL commented Jun 13, 2017

Protocols won't help with inferring the argument types of the result, though?

@ilevkivskyi
Copy link
Member

Protocols won't help with inferring the argument types of the result, though?

Yes, I didn't do any special-casing, partial in typeshed is only generic in one type _T which is the original return type. Maybe with the new support for flexible Callable this might be improved, but probably a plugin will still be required if we want precise types.

@JukkaL JukkaL added the topic-plugins The plugin API and ideas for new plugins label Jun 14, 2017
@ilevkivskyi
Copy link
Member

The original example is now fixed by #3132. This issue can be kept open to track better support of partial using a plugin.

@astrojuanlu
Copy link
Contributor

Not sure if this has been fixed entirely or perhaps it's a different issue:

(py34) juanlu@centauri /tmp $ cat test.py 
from functools import reduce, partial

def sum_two(a, b, c=0):
  return a + b + c

f_0 = partial(sum_two, c=0)
print(reduce(f_0, [1, 2, 3], 10))
(py34) juanlu@centauri /tmp $ mypy --version
mypy 0.550
(py34) juanlu@centauri /tmp $ mypy --ignore-missing-imports --check-untyped-defs test.py 
test.py:7: error: No overload variant of "reduce" matches argument types [functools.partial[Any], builtins.list[builtins.int*], builtins.int]

bhtucker pushed a commit to harrystech/arthur-redshift-etl that referenced this issue Jan 8, 2018
@nicktimko
Copy link

nicktimko commented Apr 18, 2018

I think this crops up also when trying to use a partial for a defaultdict factory:

from collections import defaultdict
from functools import partial
from typing import cast, Callable, Mapping

# error: No overload variant of "defaultdict" matches argument types [functools.partial[builtins.int*]]
x: Mapping[str, int] = defaultdict(partial(int, 10))

# (ok, but feels like it shouldn't be needed)
y: Mapping[str, int] = defaultdict(cast(Callable, partial(int, 10)))

@JukkaL
Copy link
Collaborator

JukkaL commented May 14, 2019

python/typeshed#2878 changed how partial works. Now mypy can often preserve the argument types, but it loses keyword argument names. It may still be worth it to special case partial using a plugin.

macisamuele added a commit to macisamuele/bravado-core that referenced this issue May 26, 2019
NOTE: Usage of partials has been removed due to typing issues
      More details could be found on python/mypy#1484
macisamuele added a commit to macisamuele/bravado-core that referenced this issue May 26, 2019
NOTE: Usage of partials has been removed due to typing issues
      More details could be found on python/mypy#1484
@wkschwartz
Copy link
Contributor

wkschwartz commented Jun 20, 2019

+1 for supporting partial's instance attributes: func, args, and keywords.

@finite-state-machine
Copy link

Another issue caused by this is that overloads don't work. It seems that the return type of partial(func, ...) is always the return type of the last @overload of func, regardless of arguments.

Not sure how TypeVars are supposed to work here, but it'd probably be safer to infer the type as Any or a Union of all @overloads, rather than taking an arbitrary return type.

My recent tests show that mypy is currently using the first @overload signature, rather than the last.

@AlexWaygood has reasonably judged #12675 to be a duplicate of the problem described in @Dreamsorcerer's comment. For anyone working on that problem: the now-closed issue (which I assume will continue to be available) contains some code to exercise the undesired behaviour.

@raphCode
Copy link

raphCode commented Feb 15, 2023

I stumbled across this:

from typing import reveal_type
from functools import partial

class A:
    pass

p = partial(A)

reveal_type(p)  # Revealed type is "functools.partial[A]"
reveal_type(p.func)  # Revealed type is "def (*Any, **Any) -> A"
print(issubclass(p.func, A))
# error: Argument 1 to "issubclass" has incompatible type "Callable[..., A]"; expected "type"  [arg-type]

I don't know if this is considered a mypy bug or even related to partials, but the code runs fine, and issubclass returns True as expected.

@NeilGirdhar
Copy link
Contributor

NeilGirdhar commented Feb 15, 2023

I don't know if this is considered a mypy bug or even related to partials, but the code runs fine, and issubclass returns True as expected.

It's because in the typeshed, you have

class partial(Generic[_T]):
    func: Callable[..., _T]

So, the class is transformed into a callable on assignment. MyPy is doing the right thing here.

@raphCode
Copy link

I see.
Is this the desired behavior? The typeshed does not seem to reflect what partial does at runtime in this case.

@NeilGirdhar
Copy link
Contributor

NeilGirdhar commented Feb 16, 2023

Is this the desired behavior?

Yes, assigning to a member of a class can lose type information. Just like tuple(list(('a', 'b'))) gives you tuple[str, ...] and not tuple[str, str].

This issue is about partial matching callable.

@Dreamsorcerer
Copy link
Contributor

Should this be a separate issue?

A partial of a type is not recognised as a type:

def foo(a: Type[int], **kwargs):
    x = a(1)

foo(functools.partial(int, base=10))  # Argument 1 to "foo" has incompatible type "partial[int]"; expected "Type[int]"

Admittedly, this could also be not type-safe:

foo(functools.partial(int, 5))

Not sure if there's a better way to handle this. My actual use case is to accept any subclass of an abstract type, but some implementations may need positional arguments and some not, so I use partial to include the positional arguments needed before passing into the function.

@erictraut
Copy link

@Dreamsorcerer, I don't think your example above should type check. In other words, I think mypy is doing the right thing because partial[int] is not type compatible with type[int]. For example, if you attempt to pass a base keyword argument to partial[int], you will receive an error because base was already provided. You can make your code type safe by changing the definition of foo to the following, which type checks without error:

def foo(a: Callable[..., int], **kwargs): ...

@Dreamsorcerer
Copy link
Contributor

Right, but then we lose all typing for arguments.

My actual use case is:

def init_app(session_storage: Type[aiohttp_session.AbstractStorage]):
    ...
    storage = session_storage(cookie_name="SESSION", max_age=3600, secure=True)

# First use case
init_app(aiohttp_session.SimpleCookieStorage)
# Second use case, which needs a positional key parameter
init_app(functools.partial(EncryptedCookieStorage, Fernet(config["fernet_key"]))

Would be nice not to lose typing on the kwargs (and I get an explicit Any error too).

@Dreamsorcerer
Copy link
Contributor

Dreamsorcerer commented Feb 25, 2023

Although, that's interesting that I can do init_app(EncryptedCookieStorage), which then type checks, but would fail at runtime...

So, actually, maybe the problem is that it allows a subclass of the abstract type to require an argument which the abstract class does not require...
https://github.com/aio-libs/aiohttp-session/blob/master/aiohttp_session/cookie_storage.py#L18

@NeilGirdhar
Copy link
Contributor

NeilGirdhar commented Feb 25, 2023

So, actually, maybe the problem is that it allows a subclass of the abstract type to require an argument which the abstract class does not require...

In Python, constructors don't obey LSP. That's why you shouldn't annotate the function with type[...]: there's no way to know how with which parameters to construct it. You should annotate with Callable.

Right, but then we lose all typing for arguments.

You can type the arguments by providing them to Callable or by creating a protocol.

hauntsaninja added a commit to hauntsaninja/mypy that referenced this issue Feb 23, 2024
Fixes python#1484

This is currently the most popular mypy issue that does not need a PEP.
I'm sure there's stuff missing, but this should handle most cases.
@hauntsaninja hauntsaninja linked a pull request Feb 23, 2024 that will close this issue
hauntsaninja added a commit to hauntsaninja/mypy that referenced this issue Feb 23, 2024
Fixes python#1484

This is currently the most popular mypy issue that does not need a PEP.
I'm sure there's stuff missing, but this should handle most cases.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.