[Claw Bug] beartype.claw unusable from submodules run via Python's -m CLI option

[MaxSchoenau]

Hello,

I am a huge fan of beartype and convinced our projects' team to rely on beartype as well. :D

When Beartypes claw was introduced, I immediately updated all our projects to the glorious 0.15 release. However, after trying out beartypes claw for a bit, I started feeling a bit stupid because I don't get how I get beartype to work.

Is there someone that could help me?

I have attached a minimal (poetry) project. In the modules init.py, I included

from beartype.claw import beartype_this_package  # <-- boilerplate for victory
beartype_this_package()

If I understood the release notes of 0.15 right, then this function call would mean, that my modules functions are now annotated. Howwever, the function I provided in the my_functions module, do not raise any beartype error.

What am I doing wrong?

Thank you very much for your help in advance :)
Uploading beartypeproject…
beartypeproject.zip

[leycec]

I am a huge fan of beartype and convinced our projects' team to rely on beartype as well. :D

Aww! You so nice, @MaxSchoenau. One good turn deserves another, so let's do your exciting team justice by solving the beartype.claw conundrum for you. That said...

I started feeling a bit stupid because I don't get how I get beartype to work.

Not only are you absolutely smart, but you're absolutely already there. You got it. You got beartype.claw to work. Your work here is done. I can prove this with a trivial unit test, too:

• Define a new tests/test_my_functions.py unit test module resembling:

#!/usr/bin/env python3

def test_my_functions() -> None:
    '''
    Unit test the *awesome* :func:`beartypeproject.my_functions.foo` function.
    '''

    from beartype.roar import BeartypeCallHintParamViolation
    from beartypeproject.my_functions import foo
    from pytest import raises

    with raises(BeartypeCallHintParamViolation):
        foo('bar')

• Run your the test_my_functions unit test with pytest in the usual way: e.g.,

[/tmp/beartypeproject] $ pwd
/tmp/beartypeproject
[/tmp/beartypeproject] $ ls
.  ..  beartypeproject  .idea  .pytest_cache  tests  poetry.lock  pyproject.toml  README.md
[/tmp/beartypeproject] $ pytest tests
===================================================== test session starts =====================================================
platform linux -- Python 3.11.4, pytest-7.3.2, pluggy-1.0.0
rootdir: /tmp/beartypeproject
plugins: typeguard-4.0.0, pkgcore-0.12.21, cov-4.0.0, asyncio-0.21.0, timeout-2.1.0, anyio-3.7.0, forked-1.6.0
asyncio: mode=Mode.STRICT
collected 1 item                                                                                                              

tests/test_my_functions.py .                                                                                            [100%]

====================================================== 1 passed in 0.11s ======================================================

In other words, beartype.claw is indeed correctly augmenting your wonderfully named beartypeproject with runtime type-checking fueled by @beartype. This must be true, because the test_my_functions() unit test defined above passes; that unit test clearly demonstrates that beartype.claw has properly decorated your beartypeproject.my_functions.foo() function by the @beartype decorator, which then raises the expected type violation when that function is passed an invalid parameter violating the type hint annotating that function.

I Got a Bad Feeling About This

That said... there still might be an issue here. I'm unsure what exactly that issue might be, but you don't appear to be seeing the same behaviour that I am seeing. So... that's the issue.

How are you running your beartypeproject project? I note the use of an if __name__ == '__main__': conditional in your beartypeproject.my_functions submodule. If you're attempting to run that directly as a low-level script (e.g., as python3 beartypeproject/my_functions.py), then that probably explains the anomalous behaviour you're seeing. Why? Because, in that case, Python won't have imported and thus ran your beartypeproject.__init__ submodule.

You probably instead want to run the project via Python's -m command-line option: e.g.,

[/tmp/beartypeproject] $ python3 -m beartypeproject.my_functions
Traceback (most recent call last):
  File "/home/leycec/py/beartype/beartype/claw/_importlib/clawimpcache.py", line 101, in __getitem__
    return super().__getitem__(module_name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: '__main__'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/tmp/beartypeproject/beartypeproject/my_functions.py", line 2, in <module>
    def foo(bar: float):
  File "/home/leycec/py/beartype/beartype/claw/_importlib/clawimpcache.py", line 105, in __getitem__
    raise BeartypeClawImportConfException(
beartype.roar.BeartypeClawImportConfException: Beartype configuration associated with module "__main__" hooked by "beartype.claw" not found. Existing beartype configurations associated with hooked modules include:
	{'beartypeproject.my_functions': BeartypeConf(claw_is_pep526=True, is_color=None, is_debug=False, is_pep484_tower=False, strategy=<BeartypeStrategy.O1: 2>warning_cls_on_decorator_exception=<class 'beartype.roar._roarwarn.BeartypeClawDecorWarning'>, )}

...heh. And, as we now see, there does appear to be a @beartype issue here. The beartype_this_package() correctly installed an import hook for your beartypeproject package but then refused to accept that the __main__ module is actually just an alias for your beartypeproject.my_functions submodule.

Sadly, I'm unsure whether or not @beartype can resolve this particular issue or not. My Python-Fu goes deep – but it doesn't go quite that deep. When you run a specific submodule of your project, Python itself forcefully changes the name of that submodule to __main__ (which is fine) but then fails to make __main__ importable elsewhere. Specifically, if I embed the following code somewhere in our beartype.claw API – like, say, our private beartype.claw._importlib._clawimppload submodule:

class BeartypeSourceFileLoader(SourceFileLoader):
    ...

    def __init__(self, *args, **kwargs) -> None:
        ...
        from sys import argv
        print(f'Python arguments: "{repr(argv)}"')
        print(f'Main module name: "{self._main_module_name_beartype}"')

...then the following suspicious output is printed:

Python arguments: "['-m']"
Main module name: "None"
Python arguments: "['/tmp/beartypeproject/beartypeproject/my_functions.py']"
Main module name: "beartypeproject.my_functions"
Python arguments: "['/tmp/beartypeproject/beartypeproject/my_functions.py']"
Main module name: "beartypeproject.my_functions"
Python arguments: "['/tmp/beartypeproject/beartypeproject/my_functions.py']"
Main module name: "beartypeproject.my_functions"
Python arguments: "['/tmp/beartypeproject/beartypeproject/my_functions.py']"
Main module name: "beartypeproject.my_functions"
Python arguments: "['/tmp/beartypeproject/beartypeproject/my_functions.py']"
Main module name: "beartypeproject.my_functions"
Python arguments: "['/tmp/beartypeproject/beartypeproject/my_functions.py']"
Main module name: "beartypeproject.my_functions"
Python arguments: "['/tmp/beartypeproject/beartypeproject/my_functions.py']"
Main module name: "beartypeproject.my_functions"
Python arguments: "['/tmp/beartypeproject/beartypeproject/my_functions.py']"
Main module name: "beartypeproject.my_functions"
Python arguments: "['/tmp/beartypeproject/beartypeproject/my_functions.py']"
Main module name: "beartypeproject.my_functions"
Python arguments: "['/tmp/beartypeproject/beartypeproject/my_functions.py']"
Main module name: "beartypeproject.my_functions"
Python arguments: "['/tmp/beartypeproject/beartypeproject/my_functions.py']"
Main module name: "beartypeproject.my_functions"
Python arguments: "['/tmp/beartypeproject/beartypeproject/my_functions.py']"
Main module name: "beartypeproject.my_functions"
Python arguments: "['/tmp/beartypeproject/beartypeproject/my_functions.py']"
Main module name: "beartypeproject.my_functions"
Python arguments: "['/tmp/beartypeproject/beartypeproject/my_functions.py']"
Main module name: "beartypeproject.my_functions"
Python arguments: "['/tmp/beartypeproject/beartypeproject/my_functions.py']"
Main module name: "beartypeproject.my_functions"
Python arguments: "['/tmp/beartypeproject/beartypeproject/my_functions.py']"
Main module name: "beartypeproject.my_functions"
Python arguments: "['/tmp/beartypeproject/beartypeproject/my_functions.py']"
Main module name: "beartypeproject.my_functions"
Python arguments: "['/tmp/beartypeproject/beartypeproject/my_functions.py']"
Main module name: "beartypeproject.my_functions"
Python arguments: "['/tmp/beartypeproject/beartypeproject/my_functions.py']"
Main module name: "beartypeproject.my_functions"
Python arguments: "['/tmp/beartypeproject/beartypeproject/my_functions.py']"
Main module name: "beartypeproject.my_functions"
Python arguments: "['/tmp/beartypeproject/beartypeproject/my_functions.py']"
Main module name: "beartypeproject.my_functions"
Python arguments: "['/tmp/beartypeproject/beartypeproject/my_functions.py']"
Main module name: "beartypeproject.my_functions"

😨

So much weirdness there. Honestly, I have no idea what is going on there. Python appears to be iteratively processing the passed command-line arguments and then blatantly lying to everybody (including beartype.claw) about that processing.

...wait. Wait a dag gum minute. This simply cannot be. But it is! Python itself appears to be broken in this edge case, as confirmed by this six year-old StackOverflow question on this exact topic.

Someone who is not me, please file a new issue with Python's bug tracker politely requesting that this six year-old issue finally be fixed. How frustrating! Do something, Python devs. Your runpy API is busted.

So, We Have Confirmed That Python Itself is Broken. What Now?

Now, we play video games until we pass out. Just kidding! ^{...not kidding.}

What we do now is accept that your architectural design is broken by dark influences outside our control and then find another parallel architecture whose design is not broken. Thankfully, there is one! Pick your poison, for tonight we ride the dragon:

• Define a new bearypeproject.__main__ submodule that serves as the main entry-point for your app. This is really what you want to be doing anyway. Old-school logic like if __name__ == '__main__': is really only appropriate for toy scripts and projects. The contents of this submodule should resemble:

from beartypeproject.my_functions import foo
foo('baz')

• Drop the if __name__ == '__main__': conditional from your beartypeproject.my_functions submodule, which should now just resemble:

def foo(bar: float):
    pass

• Lastly, run beartypeproject as a top-level package: e.g.,

/tmp/beartypeproject $ python3 -m beartypeproject
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/tmp/beartypeproject/beartypeproject/__main__.py", line 2, in <module>
    foo('baz')
  File "<@beartype(beartypeproject.my_functions.foo) at 0x7fa5f2038040>", line 21, in foo
beartype.roar.BeartypeCallHintParamViolation: Function beartypeproject.my_functions.foo() parameter bar='baz' violates type hint <class 'float'> under non-default configuration BeartypeConf(claw_is_pep526=True, is_color=None, is_debug=False, is_pep484_tower=False, strategy=<BeartypeStrategy.O1: 2>warning_cls_on_decorator_exception=<class 'beartype.roar._roarwarn.BeartypeClawDecorWarning'>, ), as str 'baz' not instance of float.

Success! The bear roars with glee while dipping its paw in the maple syrup jar. That said, let's leave this open until either Python itself fixes its broken behaviour or we figure out to circumvent that broken behaviour. Thus, I sigh. 😮‍💨

[leycec]

Since this metastasized from a low-key discussion into a full-blown issue, let's quietly redirect this over to our issue tracker.

Thanks so much for submitting this sheer madness, @MaxSchoenau. @beartype will get this straightened out for you and your wondrous team in no time flat or my username isn't @TheBearBro. 👍

[MaxSchoenau]

As always, thank you for your extensive and fast help!

You are right of course; I find if __name__ == '__main__': unpythonic as well and should not have used it. I only use it for debugging, as I did to determine if my functions are still decorated under the claw. Beartypes works like a charm in my normal workflow; I just gave up to early! :)

Thank you nevertheless for also tracking down and solving this edge case!