Beartype and TypeVar for type consistency accross arguments

[Louis-Pujol]

Hello,

First of all thanks for beartype ! It helps me a lot for detecting bugs and increasing the confidence in the consistency of my code.

My question is about the interaction between TypeVar and beartype. The following code defines an add function that accepts strings or numbers:

from typing import TypeVar, Union
from beartype import beartype

number = Union[int, float]
T = TypeVar('T', number, str)

@beartype
def add(a: T, b: T) -> T:
    return a + b

According to my understanding of PEP484, I read this as : (str, str) or (number, number) are valid input types for add whereas (number, str) is incorrect.

If I try to call add with a=3 and b="cat", mypy detects an error while beartype does not (and if I modify the definition of the function with return a, the code executes without raising any error).

I expected a beartype error when calling add with arguments of inconsistent types. Can you help me figure out why it is not the case ? Is my understanding of TypeVar correct and does it exist a workaround to assess at runtime the consistency of the arguments with beartype ?

Thanks in advance for your help

[leycec]

tl;dr: Plum. Plum is what you need in your life.

A longer and tedious explanation now follows that will exhaust you and make you punch the air in frustration. Also, I provide working code that solves your problem below! "Please keep reading" is what I'm saying.

First of all thanks for beartype ! It helps me a lot for detecting bugs and increasing the confidence in the consistency of my code.

Aww. 🤗

Thanks so much for the kind words, @Louis-Pujol. Python really is the Wild West of code, isn't it? Everything's anarchic, ill-defined, duck-typed, and constantly on fire. This is why we @beartype.

Sadly, even @beartype sometimes falls down on the job. Case in point: this discussion.

My question is about the interaction between TypeVar and @beartype.

...heh. Yeah. TypeVar is a known weak point for all runtime type-checkers – @beartype, @pydantic, and typeguard alike.

Currently, @beartype reduces all constrained type variables (i.e., TypeVar objects passed two or more types as in your excellent example) to a union of those types. @beartype thus reduces your useful example to this less useful example:

from typing import TypeVar, Union
from beartype import beartype

T = Union[int, float, str]

@beartype
def add(a: T, b: T) -> T:
    return a + b

And... now we understand why the following erroneous call raises the following unexpected non-human-readable TypeError exception (rather than the human-readable @beartype type violation you were expecting):

>>> add(3, 'cat')
Traceback (most recent call last):
  File "/home/leycec/tmp/mopy.py", line 13, in <module>
    add(3, 'cat')
  File "<@beartype(__main__.add) at 0x7ff9745fdf80>", line 55, in add
  File "/home/leycec/tmp/mopy.py", line 11, in add
    return a + b
           ~~^~~
TypeError: unsupported operand type(s) for +: 'int' and 'str'

You are now thinking to yourself: "@beartype sucks. Mypy rocks. Everybody, throw rocks at @beartype until it bleeds!"

You aren't necessarily wrong. @beartype should (and will eventually) do better here. In fact, our feature request #236 documents exactly the functionality you were expecting here: "Type-stable callable declarations (i.e., TypeVar constraints isolated to single calls)."

Oh My Gods. @beartype Sucks, Everybody!

...heh. @beartype may suck for this use case – but Plum doesn't. Plum is built on the back of @beartype. Therefore, by extension, if we squint our eyes, @beartype doesn't suck either! ^{everybody pats everybody on the back}

^{the 80's: they kinda sucked.}

Okay. Look. What I'm trying (and failing) to say is that your use case can be refactored into multiple dispatch. Python's primary competitor, Julia, comes with multiple dispatch built-in. Although Python currently fails to ship with out-of-box support for multiple dispatch, fear not! Plum succeeds where Python itself fails.

Let's refactor your @beartype-unfriendly example into a @beartype-friendly example driven by Plum's multiple dispatch mechanism, the @plum.dispatch decorator:

from plum import dispatch
from typing import Union

number = Union[int, float]

@dispatch  # <-- awesome magic
def add(a: number, b: number) -> number:
    '''
    Numerically add the passed two integers and/or floating-point numbers.
    '''

    return a + b

@dispatch  # <-- more awesome magic
def add(a: str, b: str) -> str:
    '''
    Concatenate the passed two strings.
    '''

    return a + b

That's actually considerably easier to both read and reason about than the original TypeVar-based example. That said...

There's a Tradeoff and It Kinda Sucks (But It's Not Too Bad)

There always is, right? This is Python. Gotta be fire-breathing tradeoffs lurking around every corner or it isn't Python. The tradeoff here is that mypy will probably now hate you.

I consider this a worthwhile tradeoff. After all: what's better? Type hints that are enforced at runtime via @beartype + Plum or type hints that are ignored at runtime but complained about in a whining voice inside your IDE by mypy or pyright or PyCharm or whatevahs? Yes. I think we all know the correct answer to that question.

Runtime >>>>>> IDE. At the end of the working day, IDEs are ignorable. Runtime is not. That said...

This Is an Open Question in the Typing Community, Actually

Mypy integration is actually on open question over at the Plum issue tracker. I recall reading that @wesselb (i.e., the awesome Plum guy) and @gabrieldemarmiesse (i.e., the awesome Python-on-whales guy) are tag-teaming this on Python's official discussion forum to standardize a new Python Enhancement Proposal (PEP).

Lastly, since:

• You already require @beartype as a mandatory dependency in your workflow, requiring Plum as an additional mandatory dependency should (theoretically) be no hardship? Hopefully? Maybe? I promise nothing.
• Plum is built on the back of @beartype and since @beartype is fast, Plum is also fast by extension. Ergo, efficiency is not a concern here. Plum guarantees multiple dispatch to be fast in Python, too.

@wesselb: Pinging you here because you are Plum incarnate and I probably got everything wrong. 😩

[Louis-Pujol]

Hi !

For the moment it's not a bottleneck in my work, I'm 100% happy with beartype only. I was just trying to decipher python's typing world reading PEPs/beartype/mypy/typing... docs and do some small tests to understand how things are working. I opened this discussion as I wasn't sure to understand the role of TypeVar. Your answer goes further from my expectations, thanks for the workaround and for pointing me out Plum, I didn't know about it. I'll have a closer look and follow the discussions about overloading !

Not easy for a newcomer to figure out the current state of typing in Python and the possibilities it offers ! The fact that the python documentation is 100% static typing oriented doesn't help... I only do runtime type checking for the moment, I consider it sufficient to be confident in my codebase in combination with tests, and my priority is to raise meaningful errors to the user.

Thanks a lot for the detailed answer and the work you are doing on beartype, I'll continue to explore these topics !

[wesselb]

Just briefly dipping into the discussion here to say that, of course, @leycec is right on all fronts. :)

Whereas beartype can be used to ensure that the types of arguments are correct, plum builds on beartype to enable multiple dispatch. Multiple dispatch essentially is an advanced form of function overloading where you implement the same function multiple times with different types for the arguments, as in @leycec's example.

Unfortunately, at the moment neither beartype nor plum fully support type variables. But @leycec is definitely right that, sometimes, a signature involving a type variable can be simplified into multiple definitions without type variables.

@Louis-Pujol should you have a question about any of this, about Beartype, or about Plum, please feel free to ask it here or elsewhere. :)

[Louis-Pujol]

Many thanks @leycec and @wesselb for the explanations ! 👍