[Feature Request] Support for os.PathLike

[kaparoo]

TL;DR

How can I make this code work?

@beartype
def for_any_pathlike(str | os.PathLike[str]):
    ...

for_any_pathlike("awesome_text.txt")  # it works!
for_any_pathlike(pathlib.Path("useful_data.csv"))  # raises `beartype.roar.BeartypeDecorHintNonpepException` 

Content

There is a type alias StrPath defined in typeshed as

StrPath = str | os.PathLike[str]

Because the typeshed is not accessible at runtime, I tried to define StrPath myself and use it with @beartype like this:

from os import PathLike
from pathlib import Path

from beartype import beartype

StrPath = str | PathLike[str]

@beartype
def foo(path: StrPath) -> None:
    print(f"{path=!s}")

foo("test.py")  # "test.py"
foo(Path("test.py"))  # raises `beartype.roar.BeartypeDecorHintNonpepException`

The function foo worked as expected for "test.py" (string), but it raised beartype.roar.BeartypeDecorHintNonpepException for Path("test.py") which is evaluated - at least in mypy - as PathLike[str]:

beartype.roar.BeartypeDecorHintNonpepException: Function __main__.foo() parameter "path" type hint os.PathLike[str] either PEP-noncompliant or currently unsupported by @beartype.

I presume this is because os.PathLike uses the way below to provide an interface for Generic:

GenericAlias = type(list[int])

class PathLike(abc.ABC):
    ...
    __class_getitem__ = classmethod(GenericAlias)

Therefore I hope beartype supports ABCs like os.PathLike. Currently, I use StrPath = str | Path and there is no problem in 99% of my cases, but it would be better to base it on os.PathLike.

[leycec]

...heh. Oh, os.PathLike – the red-headed stepchild of the typing world. Officially, the semantics of os.PathLike[...] have yet to be standardized; since there's no existing standard, everyone just makes up their own interpretation of what subscripting os.PathLike actually means. Unofficially, mypy currently enforces the following ad-hoc constraints on os.PathLike subscriptions:

S = TypeVar('S', bound=Union[str, bytes])
class PathLike(Generic[S], extra=os.PathLike): # or Protocol[S] in future
    def __fspath__(self) -> S:
        ...

Given that, @beartype can effectively type-check a type hint os.PathLike[T] as follows:

• An arbitrary object obj satisfies a type hint os.PathLike[T] if and only if:

isinstance(obj, os.PathLike) and
isinstance(obj.__fspath__(), T)

That is to say:

1. obj must be an instance of os.PathLike (i.e., must define an __fspath__() dunder method).
2. The value returned by calling the obj.__fspath__() dunder method must be an instance of the same type T subscripting the original type hint os.PathLike[T].

Complications Arise, Yo

Of course, there are complications. The os.PathLike generic is way too permissive at runtime. Notably, os.PathLike allows itself to be subscripted by arbitrary type hints that make no sense: e.g.,

>>> from os import PathLike
>>> PathLike[list[int]]
PathLike[list[int]]  # <-- uhh. wut? this makes no sense, Python. *facepalm*

Supporting os.PathLike subscriptions thus also requires @beartype to handle erroneous subscriptions that make no semantic sense. Indeed, I think there's only four valid parametrizations of os.PathLike:

• os.PathLike[str], implying isinstance(obj, os.PathLike) and isinstance(obj.__fspath__(), str).
• os.PathLike[bytes], implying isinstance(obj, os.PathLike) and isinstance(obj.__fspath__(), bytes).
• os.PathLike[typing.AnyStr], which reduces to just os.PathLike, implying just isinstance(obj, os.PathLike). The child type hint typing.AnyStr is semantically meaningless.
• os.PathLike[typing.Any], which reduces to just os.PathLike, implying just isinstance(obj, os.PathLike). The child type hint typing.Any is semantically meaningless. Unsurprisingly, this is basically a duplicate of the prior subscription.

That's it. I think? Still, that's pretty annoying. That's yet more bureaucratic minutiae that @beartype has to manually implement, because CPython refused to do its job here.

"Urgh!"*

Definitely Feasible, Bro

That's... definitely feasible. Rejoice, everyone. Sadly, I also kinda have no volunteer time to do this. It doesn't help that CPython itself fails to validate os.PathLike[...] subscriptions and just lets anybody subscript os.PathLike by literally anything. 🤦

For now, would you just mind dropping the subscription [str]? That is, would you mind just doing this:

StrPath = str | PathLike

That should absolutely work out-of-the-box without requiring any changes to @beartype itself.

Therefore I hope beartype supports ABCs like os.PathLike.

@beartype definitely supports ABCs like os.PathLike – and always has, thankfully. It's just subscripting ABCs with arbitrary type hints like os.PathLike[str] and os.PathLike[list[int]] where complications arise. None of that has been standardized. So, @beartype has to just make up its own semantics and pretend that it knows what it's talking about. </sigh>

[kaparoo]

Thank you for your kind comment @leycec!

For now, would you just mind dropping the subscription [str]? That is, would you mind just doing this:

StrPath = str | PathLike

The StrPath I want is either str or an object with a __fspath__ method that returns str, so it seems a little different from what you suggested. 😅

import os
from pathlib import Path
from beartype import beartype

StrPath = str | os.PathLike

@beartype
def foo(path: StrPath):
    print(os.fspath(path))

foo("test")  # it works (obviously)
foo(Path("test"))  # okay

class MyPath:
    def __init__(self, path):
        self.path = path

    def __fspath__(self):
        return self.path
    
foo(MyPath(b"test"))  # it also works...

So, I implemented a custom PathLike like mypy's issue:

from typing import TypeVar, Protocol, runtime_checkable

T = TypeVar("T", str, bytes)

@runtime_checkable
class PathLike(Protocol[T]):
    def __fspath__(self) -> T:
        ...

StrPath = str | PathLike[str]

@beartype
def foo(path: StrPath):
    print(os.fspath(path))

foo("test")
foo(Path("test"))

# foo(b"test")  # `foo` blocked `bytes` (finally!)

class BytePath:
    def __init__(self, path: str):
        self.path = bytes(path, encoding="UTF-8")

    def __fspath__(self) -> bytes:
        return self.path

foo(BytePath("test"))  # no..! it works!

I know that all methods of the @runtime_checkable() Protocol are checked only for existence.
Is there any way to solve this without extra validators in beartype?

[leycec]

Yuppers. The custom Protocol approach also totally works. Clever workaround there. Sadly...

Is there any way to solve this without extra validators in beartype?

Tragically, not at the moment. When @beartype fails to do what you want out-of-the-box, extra validators save you from @beartype's personal failings. Validators save you from us.

Thankfully, extra validators would allow you to both have your typing cake and eat it too. Since there are only two meaningful subscriptions of PathLike (i.e., PathLike[str] and PathLike[bytes]), you only need two beartype validators to make this magic happen. In fact, since you don't even care about PathLike[bytes], that's just one beartype validator. Pure static type-checkers like mypy and pyright will continue to support this as expected. Moreover, this approach avoids the need to define your own custom runtime Protocol subclass. It's simpler, faster, and simply magical.

Behold! PathLikeStr, arise like our Bengal cat from a crusty-eyed 14-hour slumber in our cat tree:

from beartype.vale import Is
from os import PathLike
from typing import TYPE_CHECKING, Annotated

# PathLike[str], but actually supported by everybody.
if TYPE_CHECKING:
    PathLikeStr = PathLike[str]
else:
    PathLikeStr = Annotated[PathLike, Is[
        lambda path_like: isinstance(path_like.__fspath__(), str)]]

# This is the way.
StrPath = str | PathLikeStr

Now, let's prove that actually does what @leycec promises that does:

# Import boring stuff.
>>> from beartype import beartype
>>> from os import fspath
>>> from pathlib import Path

# Prove that good always prevails.
>>> @beartype
... def foo(path: StrPath) -> str:
...     return fspath(path)
>>> foo("test")
'test'  # <-- good.
>>> foo(Path("more test"))
'more test'  # <-- GOOD.

# Now for the coup de grace.
>>> class BytePath(object):
...     def __init__(self, path: str):
...         self.path = bytes(path, encoding="UTF-8")
... 
...     def __fspath__(self) -> bytes:
...         return self.path
>>> foo(BytePath("ugly test"))
Traceback (most recent call last):  # <-- Suck it, "BytePath". Just suck it.
  File "/home/leycec/tmp/mopy.py", line 31, in <module>
    foo(BytePath("ugly test"))
  File "<@beartype(__main__.foo) at 0x7f3acf351c60>", line 32, in foo
beartype.roar.BeartypeCallHintParamViolation: Function __main__.foo() parameter path=<__main__.BytePath object at 0x7f3acec014c0> violates type hint typing.Union[str, typing.Annotated[os.PathLike, Is[lambda path_like: isinstance(path_like.__fspath__(), str)]]], as <class "__main__.BytePath"> <__main__.BytePath object at 0x7f3acec014c0>:
* Not str.
* <class "__main__.BytePath"> <__main__.BytePath object at 0x7f3acec014c0> violates validator Is[lambda path_like: isinstance(path_like.__fspath__(), str)]:
    False == Is[lambda path_like: isinstance(path_like.__fspath__(), str)].

Typing cake: it tastes delicious. 😋

[kaparoo]

Oh... this is the best way in the current situation...
Okay. I'll take this 🥲
Thanks @leycec ! 🙇

[leycec]

You're most welcome. As a gentle reminder to myself to eventually implement this properly for everybody, would you mind if I quietly reopen this feature request? With any luck, I'll tackle stuff like this in 2024. Let the hype train begin! 🎆

[kaparoo]

Sure! Thank you for your attention! 🙏

[leycec]

Partially resolved by fe5b23d. @beartype's upcoming 0.17.0 release ^{...to be released before Santa Bear Claws gives us all the video games} shallowly type-checks os.PathLike[...] type hints.

In fact, @beartype 0.17.0 shallowly type-checks all weirdo non-standard type hints in Python's typeshed. This includes os.PathLike[...], weakref.weakref[...], and even more hideous monsters as yet unknown to man and beaver alike. @beartype 0.17.0 no longer raises exceptions when confronted with this sort of typing gore.

This means that your above verbose definition of PathLikeStr can be reduced to this terse one-liner:

from beartype.vale import Is
from os import PathLike
from typing import Annotated

# PathLike[str], but actually supported by everybody.
PathLikeStr = Annotated[PathLike[str], Is[  # <-- much simpler, much terser, much gooder
    lambda path_like: isinstance(path_like.__fspath__(), str)]]

# This is the way.
StrPath = str | PathLikeStr

Rejoice! Santa Bear Claws has heard your prayers. 🎅

[kaparoo]

Brilliant! I miss Santa already 🎅