Modify check_type to work without memo

[Kentzo]

In my project I would like to check arbitrary object against PEP 484 definition.
check_type seems to be what I need, but it requires a memo object which I do not have in my use case.

[agronholm]

How would that work? Where would the annotation come from?

[agronholm]

Also, what is the use case here?

[Kentzo]

@agronholm In my case I'm working with PEP 526 annotations and get them directly from a class which is like typing.NamedTuple but enforces type match at runtime.

[agronholm]

Ok, what about type variables? I'm not certain how they work with variable annotations.

[Kentzo]

class X:
    a: int = 10

print(X.__annotations__['a'])  # int

[Kentzo]

My request is general though: assuming one has both value and expected_type he would like a function that checks just that.

[agronholm]

Yes, I understand, but I want to make sure type variables are handled correctly. That is the entire reason for the memo argument.

[Kentzo]

I don't think I understand well enough the problem memo solves. Once you resolved expected type and value, what else can go wrong?

[agronholm]

Do you know how type variables work? Have you ever used them?

[Kentzo]

My best guess they are alike C++'s template parameters.

[agronholm]

Something like that. Consider the following code:

T_Element = TypeVar('T_Element')

def extract(list_: List[T_Element], index: int) -> T_Element:
    return list_[index]

T_Element is "locked" for list_ so the function must return whatever type the list is being used to hold. The purpose of memo is to keep track of the variable bindings.

[Kentzo]

But you cannot define function's type without defining function itself, can you? Therefore it cannot be assigned as variable's annotation.

You can with typing.Callable.

[agronholm]

You do know about generic classes, right? Classes like:

class Foo(Generic[T_Element]):
    ...

[Kentzo]

Indeed, without memo the following code does not work:

T = typing.TypeVar('T')
typeguard.check_type('', ('', 1, ''), typing.Tuple[T, int, T], None)

But seems that check_type has everything to test against that. Can you think of an example when definition of expected_type is not enough other than when you need to extract that from function's signature?

[agronholm]

Why are you talking about functions now? I thought you were after variable annotations here.

[Kentzo]

@agronholm I'm trying to understand if _CallMemo is useful outside of a function and how code can be adjusted to handle an example like one above.

[Kentzo]

That seems to work:

class _Memo:
    def __init__(self):
        self.typevars = {}  # type: Dict[Any, type]

T = typing.TypeVar('T')
typeguard.check_type('', ('', 1, '123'), typing.Tuple[T, int, T], _Memo())
typeguard.check_type('', ('', 1, 123), typing.Tuple[T, int, T], _Memo())  # TypeError

[agronholm]

Where would those type variables get their bound values?

[agronholm]

In the case of classes, type variables in variable annotations would get their values from the type used to instantiate the class, if I'm correct. I need to further study the PEP to be sure.

[agronholm]

Ok, consider the following:

from typing import TypeVar, Generic, List

T = TypeVar('T')

class Foo(Generic[T]):
    bar: T

    def __init__(self, bars: List[T]):
        self.bar = bars[0]

foo = Foo(['xyz'])

Now according to PEP 484, the type of foo is now Foo[str]. Now I'm wondering if there is any way to get the parameter at run time.

[agronholm]

I could maybe modify the code to work without a memo and just skip the type variable binding if the information is not available.

[Kentzo]

Now according to PEP 484, the type of foo is now Foo[str].

It doesn't seem that the resolved type is stored anywhere at runtime and needs to be manually resolved.

[agronholm]

It doesn't seem that the resolved type is stored anywhere at runtime.

Yes, so it seems. I recall pytypes has an ugly workaround for this, but I don't want to do that. Oh well, I guess treating type variables as their bound types (or Any in their absence) is acceptable?

[Kentzo]

@agronholm By bound you mean first encountered type?

[agronholm]

No, those are erased at run time. I mean: T = TypeVar('T', bound=dict) where we would then substitute T for dict.

[Kentzo]

What about constraints?

[Stewori]

I recall pytypes has an ugly workaround for this

This workaround is only applied to older typing module, before __orig_bases__ was introduced. I found inconsistent behaviour between versions of typing even uglier. typing 3.5.2.2 is the last version where the monkeypatch applies to (shipped with older CPython 3.5).

[Stewori]

Note that pytypes gets examples like in #21 right (feature not released yet). However, I recognized faulty behavior with examples like in #24 (comment) where the typevar is not bound a-priori. I'm looking into this right now and hope to fix it soon.

[agronholm]

Now that attrs 17.3 has been released with PEP 526 support, this feature will become much more important.

[Stewori]

What do you think how should the following behave?

T_ = TypeVar('T_')

@typechecked
def test(a: T_, b: T_) -> None:
    #whatever
    return None

test(1, 1.5)
test(1.5, 1)

Should the calls fail because int and float are not the same?
Or should only the first call fail because T_ is first bound to int?
Or should both pass and T_ be bound to float and in general to union of bindings accross arguments, then only fail regarding return type maybe?
The last would imply e.g. for

test(4, 'a')

T_ to become Union[int, str].
Or should it be configurable? How would typguard do it?

[Kentzo]

An interesting read from the C++ world: Deduction from a function call.

I think either Python's typing syntax must be extend to allow explicit resolve of the parameters:

test(1, 1.5)  # type: T = int # error
test[float](1, 1.5)  # type: T = float # ok

or interpreter implementation specific forward analysis of the byte code (compilation?) needs to be done. The latter will not cover cases because Python :)

Or should both past and T_ be bound to float and in general to union of bindings accross arguments, then only fail regarding return type maybe?

I think type checker should warn unless user explicitly configured e.g. using pseudocode as above:

test(4, 'a') # type: T = Union[int, str]

@agronholm

Oh well, I guess treating type variables as their bound types (or Any in their absence) is acceptable?

Class annotations can be used for that in a hope that instance of the class has some of the class variables assigned:

T = TypeVar('T')

class Foo(Generic[T]):
    bar: T

    def __init__(self, bars: List[T]):
        self.bar = bars[0]

foo = Foo([42])

resolved_params = {}

for param in foo.__parameters__:
    for attr, attr_param in get_type_hints(foo).items():
        if attr_param is param:
            try:
                param_type = type(getattr(foo, 'bar'))
            except AttributeError:
                continue
            else:
                break
    else:
        param_type = Any

    resolved_params[param] = param_type

assert resolved_params == {T: int}

Don't know how subclasses should be treated though.

[agronholm]

@Stewori In your example, I believe both calls should fail because the T_ is invariant and the argument types differ. But if T_ was covariant, then the second call should work.

[Stewori]

@agronholm I also had this idea regarding covariant, but I'm not sure whether that is actually the meaning of covariant. AFAIK covariant means for a parameter of a generic class that if the parameter becomes more specific, the generic class also becomes more specific. A function call is - however - not a generic type, so I'm not sure if the principle should be applied here. (Should variance direction be inverted regarding return type? In subclassing sense, return types behave contravariantly.)

Anyway, doing it like you suggest is probably the most consistent approach for this.
In that manner, I guess for a contravariant T_ only the first call should work and in no case both calls should work (no bivariant types allowed in PEP 484). So, the order of parameters inherently matters here.

[Kentzo]

@agronholm Another pice that I found while re-reading PEP 526 is that ClassVar annotations cannot be parametrized.

[agronholm]

Yep, that I know but non-classvar variable annotations can be.

[Stewori]

Just wanted to update here that the above examples concerning TypeVars are now supported in pytypes. This version was released to PyPI yesterday.

[Kentzo]

Here is the complete use case where I wanted to use typeguard: async-app.config.

[Stewori]

But that should work now that None is acceptable for memo, shouldn't it?

You can alternatively use pytypes. Replace

import typeguard
...
typeguard.check_type(name, value, expected_type, None)

by

import pytypes
...
pytypes.is_of_type(value, expected_type)

I suppose this is a kind of thing both frameworks can handle. Or is the type specifically nasty, e.g. containing TypeVars or whatever? In that case maybe file another issue about that case.

[Kentzo]

I'm okay with typeguard for now as I'm not using TypeVars.

It's very simple to add support for pytypes. Supply will follow demand and PRs are welcome :)