Improve inference in builds() and from_type() for attrs classes #1282

Zac-HD · 2018-05-13T15:09:59Z

Closes #954. If you have well-behaved classes defined using attrs, and use the various validation, typing, and conversion features without abusing them too badly, Hypothesis will be a little more magical ✨

The trick for users - as usual - is to handle everything that static analysis thinks might be a valid input! It's a good habit in general, and I actually kinda like the tooling pushing me that way 😄

(as an implementer, I now remember why I haven't touched inference in a while...)

kxepal · 2018-05-13T16:40:09Z

hypothesis-python/RELEASE.rst

+RELEASE_TYPE: minor
+
+This release adds a new mechanism to infer strategies for classes
+defined using pypi:`attrs`, based on the the type, converter, or


kxepal · 2018-05-13T16:42:13Z

hypothesis-python/src/hypothesis/searchstrategy/attrs.py

+    # Try inferring type or exact values from attrs provided validators
+    if atrib.validator is not None:
+        validator = atrib.validator
+        if isinstance(validator, attr.validators._OptionalValidator):


That seems not very legal to access third party privates. Isn't there any other ways to get those via public api?

We could use type(attr.validators.optional([])), but IMO it's actually cleaner to peek inside here. I think the alternative form is about as likely to break as the explicit form, and I strongly prefer the look of the latter.

kxepal · 2018-05-13T16:43:02Z

hypothesis-python/src/hypothesis/searchstrategy/attrs.py

+    )
+
+
+def from_attrs_attribute(atrib):


kxepal · 2018-05-13T16:44:37Z

hypothesis-python/src/hypothesis/searchstrategy/attrs.py

+    # Various things we know.  Updated below, then inferred to a strategy
+    base = st.nothing()  # updated to none() if None is a possibility
+    default = st.nothing()  # A strategy for the default value, if any
+    in_ = []  # list of in_ validator collections to sample from


May be name it validator_collections instead? in_ doesn't looks informative. Or there is a reason for such naming?

Zac-HD · 2018-05-16T14:04:25Z

Hey @DRMacIver, can you manually cancel some of the huge stack of Appveyor builds and maybe look into auto-cancellation? (Apparently Appveyor calls this "rolling builds")

DRMacIver · 2018-05-16T16:29:34Z

Hey @DRMacIver, can you manually cancel some of the huge stack of Appveyor builds and maybe look into auto-cancellation? (Apparently Appveyor calls this "rolling builds")

Done and done. I thought we got your permissions on appveyor working though?

Zac-HD · 2018-05-17T03:03:06Z

I thought we got your permissions on appveyor working though?

We've looked at it a couple of times but without success; and with auto-cancellation it's NBD really.

Zac-HD · 2018-05-17T15:21:17Z

Hey @HypothesisWorks/hypothesis-python-contributors, this is ready for review 🎉

(and hi @hynek, if you have opinions on Hypothesis-for-Attrs I'd love to hear them too)

DRMacIver

I don't seem to be quite awake enough today to do a full review of this, but I've added some comments with thoughts and some stuff that confused me.

In general I'm finding a lot of the inference code quite inscrutable. I get why it's like this - there are a lot of annoying special cases to deal with - but I think we might need better commenting at a minimum (something like the mini-essays in engine.py) and ideally some way to do a lot of this more cleanly.

DRMacIver · 2018-05-21T11:44:47Z

hypothesis-python/src/hypothesis/searchstrategy/attrs.py

+    default = st.nothing()  # A strategy for the default value, if any
+    in_collections = []  # list of in_ validator collections to sample from
+    # value must be instance of all these types or tuples thereof
+    types = defaultdict(list)  # maps type to list of locations, for error msgs


I got really confused about what this was doing because of the name. Maybe something like types_to_locations?

Actually where do we even use this?

Er, I actually ended up abandoning the reporting part and this could simply be a set instead.

DRMacIver · 2018-05-21T11:46:19Z

hypothesis-python/src/hypothesis/searchstrategy/attrs.py

+    if attrib.validator is not None:
+        validator = attrib.validator
+        if isinstance(validator, attr.validators._OptionalValidator):
+            base, validator = st.none(), validator.validator


nit: I think having these on the same line makes it less clear.

DRMacIver · 2018-05-21T11:51:28Z

hypothesis-python/src/hypothesis/searchstrategy/attrs.py

+
+
+def from_attrs_attribute(attrib):
+    """Infer a strategy from an attr.Attribute object."""


I think I need some more high level commenting about what's going here, separate from the code. I'm not having a very awake day, but I'm really struggling to follow the logic here.

DRMacIver · 2018-05-21T11:53:06Z

hypothesis-python/src/hypothesis/searchstrategy/attrs.py

+    # Try inferring from the converter, if any
+    converter = getattr(attrib, 'converter', None)
+    if isinstance(converter, type):
+        # Could this ever work but give incorrect inference?


This comment is pretty inscrutable to me.

DRMacIver · 2018-05-21T11:58:57Z

hypothesis-python/src/hypothesis/strategies.py

@@ -1047,6 +1049,12 @@ def builds(
    value :const:`hypothesis.infer` as a keyword argument to
    builds, instead of a strategy for that argument to the callable.

+    If the callable is a class defined with :pypi:`attrs`, missing required
+    arguments may be inferred from the type, converter, or validator (for


I wonder if it would be better to be no-specific about this and just say "on a best-effort basis"? The details are kinda uninteresting - either it works and people barely have to know it's there, or it doesn't and people will write their own strategy here rather than tinkering with their class definitions.

DRMacIver · 2018-05-21T12:00:46Z

hypothesis-python/RELEASE.rst

+This release adds a new mechanism to infer strategies for classes
+defined using :pypi:`attrs`, based on the the type, converter, or
+validator of each attribute.  This inference is now built in to
+:func:`~hypothesis.strategies.builds` and :func:`~hypothesis.strategies.from_type`.


Is it worth calling out that this will change the behaviour of strategies using builds for attrs where there's a default type they're omitting?

e.g. if I have

@attr.s() class Foo(object): bar = attr.ib(default=0, converter=int)

Then previously this would have generated only Foo(0), but now can generate Foo(n) for any n.

(I think it's fine to change this, I just think it's worth explicitly mentioning)

It will only change the behaviour in this case if you call builds(Foo, bar=infer); or succeed instead of failing if you try to build something with a required argument but no strategy.

The inference about the default value is so that inferred strategies can "shrink away" to the default value, and then None if that would be valid.

Ah, right! Sorry I was forgetting how defaults and inference interacted.

Zac-HD · 2018-05-21T12:35:42Z

Thanks for the review! If it's hard for you to follow now it'll be hard for me too later, so best to comment heavily and clarify where I can now! I'll make the docs less specific too.

Zac-HD · 2018-05-21T14:43:02Z

OK, that should be easier to follow now.

In the process I've realised that I also need to reconcile the types in a way that supports subtypes better (i.e. at all), so that section is not likely to stay a two-liner.

Otherwise pretty much done, assuming test pass 😆

hynek · 2018-05-23T07:38:22Z

I cannot review this code but I’m very excited by it!

Zac-HD · 2018-05-23T14:33:34Z

I've restructured and commented the inference code, so it should be easier to follow now.
The type-driven inference is reasonably principled too.

The final upgrade is to write something that deals with subtypes, so that [instance_of(object), instance_of(str)] can resolve to from_type(str) instead of nothing(). I'll need to sleep on it to write something elegant though, so just ignore the marked section if you're reviewing before I get to that. Done.

Zac-HD · 2018-05-24T12:52:21Z

Ping @DRMacIver; ready for final review and maybe even merging 😉

DRMacIver · 2018-05-24T12:53:33Z

Ping @DRMacIver

Sorry. ETA for review capacity probably not until Monday. I might be able to get some time to do review tomorrow morning but no promises. 🤞

DRMacIver · 2018-05-24T12:57:19Z

(Based on what I've seen already, if someone else wants to do the review on this and is happy to sign off on it, I'm perfectly happy for them to do so! I have no problem with the feature itself - all of my questions would be implementation level. But I'm also happy to do the review on this, just temporarily time constrained)

Zac-HD · 2018-05-24T12:57:29Z

hypothesis-python/src/hypothesis/internal/compat.py

        while i and j >= 0:
            result[j] = i & 255
            i >>= 8
            j -= 1
        if i:
-            raise OverflowError('int too big to convert')
+            raise OverflowError('i=%r cannot be represented in %r bytes'


I had a transient failure here, and couldn't replicate it. Of course I haven't seen it since changing the message either, but maybe someday this will help 😕

Worth a ticket about the transient failure, pointing to this line? At least we’d have a record of the last time we (sort of) saw this failure.

Do you still have the stack trace around? Would be useful to have it in an issue for posterity.

Alas, no - I expected to get a better one and haven't seen it since 😭

Zac-HD · 2018-05-24T13:03:33Z

No worries then, and no pressure!

If you have a little time, I'd actually prefer a review of #1270 - it's basically "is David happy with the documentation of novel public API".

@HypothesisWorks/hypothesis-python-contributors it would be nice to get this (and/or the easier #1295) merged in time for my May 31st workshop if any of you are interested in reading it 😉

Zac-HD · 2018-06-24T09:03:45Z

Ping @HypothesisWorks/hypothesis-python-contributors - it would be lovely to get a review of this!
It's been a month now, and it's blocking an external contributor from working on #1309 😭

DRMacIver

Some minor comments. Generally looks good. I'd be lying if I said I fully understood the inference code, but it's definitely a lot more readable now!

DRMacIver · 2018-06-24T10:22:34Z

hypothesis-python/src/hypothesis/searchstrategy/attrs.py

+    if strat.is_empty:
+        raise ResolutionFailed(
+            'Cannot infer a strategy from the default, vaildator, type, or '
+            'converter for %r' % (attrib,))


What does the repr for a an attrib object look like? Does this actually end up as a useful error message? (e.g. I'd like to see both the attribute name and the class name in this).

The minimal attrib repr is Attribute(name='a', default=NOTHING, validator=None, repr=True, cmp=True, hash=None, init=True, metadata=mappingproxy({}), type=None, converter=None). I'll add the class repr to the message though.

DRMacIver · 2018-06-24T10:26:21Z

hypothesis-python/src/hypothesis/searchstrategy/types.py

-        return -1
-    return issubclass(t, collections.abc.Container)
+        return (-1, repr(t))
+    if PY2:


I'm pretty 👎 on if PY2 checks outside of compat.py. Couldn't this be a from hypothesis.internal.compat import Container?

DRMacIver · 2018-06-24T10:28:16Z

hypothesis-python/src/hypothesis/internal/compat.py

        while i and j >= 0:
            result[j] = i & 255
            i >>= 8
            j -= 1
        if i:
-            raise OverflowError('int too big to convert')
+            raise OverflowError('i=%r cannot be represented in %r bytes'


Do you still have the stack trace around? Would be useful to have it in an issue for posterity.

alexwlchan · 2018-06-16T07:40:39Z

hypothesis-python/src/hypothesis/internal/compat.py

        while i and j >= 0:
            result[j] = i & 255
            i >>= 8
            j -= 1
        if i:
-            raise OverflowError('int too big to convert')
+            raise OverflowError('i=%r cannot be represented in %r bytes'


Worth a ticket about the transient failure, pointing to this line? At least we’d have a record of the last time we (sort of) saw this failure.

alexwlchan · 2018-06-24T11:19:42Z

hypothesis-python/src/hypothesis/searchstrategy/attrs.py

+    # when we try to get a value but have lost track of where this was created.
+    if strat.is_empty:
+        raise ResolutionFailed(
+            'Cannot infer a strategy from the default, vaildator, type, or '


sp: “vaildator”

Zac-HD · 2018-06-25T01:44:13Z

Blocked by #1348 (test_can_flatmap_to_recursive_data).

Zac-HD · 2018-06-26T05:30:11Z

And it's released! Long live testing things with attrs 🎉

Zac-HD force-pushed the infer-attrs branch 3 times, most recently from 3fc0435 to ded2110 Compare May 14, 2018 06:46

kxepal reviewed May 14, 2018

View reviewed changes

Zac-HD force-pushed the infer-attrs branch 4 times, most recently from adc55e1 to 30a29b9 Compare May 16, 2018 13:41

Zac-HD force-pushed the infer-attrs branch from 30a29b9 to b76db56 Compare May 17, 2018 03:02

Zac-HD force-pushed the infer-attrs branch from b76db56 to 3afa464 Compare May 17, 2018 04:54

DRMacIver reviewed May 21, 2018

View reviewed changes

Zac-HD force-pushed the infer-attrs branch 2 times, most recently from 5874a44 to 99d677f Compare May 21, 2018 23:36

Zac-HD force-pushed the infer-attrs branch from 99d677f to 44113f7 Compare May 23, 2018 14:00

Zac-HD force-pushed the infer-attrs branch 3 times, most recently from 82ad134 to b8a716b Compare May 24, 2018 10:01

Zac-HD commented May 24, 2018

View reviewed changes

Zac-HD mentioned this pull request Jun 1, 2018

Fix class xx(NamedTuple) introspection #1309

Closed

Zac-HD mentioned this pull request Jun 15, 2018

Deprecate the use of settings objects as context managers #1327

Closed

DRMacIver requested changes Jun 24, 2018

View reviewed changes

alexwlchan reviewed Jun 24, 2018

View reviewed changes

Zac-HD force-pushed the infer-attrs branch 4 times, most recently from d8d9250 to bf12de9 Compare June 24, 2018 14:03

Zac-HD mentioned this pull request Jun 24, 2018

Ruby tests failing for Python PR #1351

Closed

Zac-HD force-pushed the infer-attrs branch from bf12de9 to e12ab35 Compare June 24, 2018 15:16

DRMacIver approved these changes Jun 25, 2018

View reviewed changes

Zac-HD and others added 7 commits June 25, 2018 19:08

Implementation of strategy inference for attrs

c1fb9e9

Tests for attrs inference

0720de0

Add docs and release note

5c8f8a3

Handle validator for "value in string"

5a42a15

Clarify and comment inference code for review

463ae0f

Improve inference from attrs type metadata

7dc8612

Fix Py2 int/long in from_type

76e5b74

Zac-HD force-pushed the infer-attrs branch from e12ab35 to f0fd9c0 Compare June 25, 2018 09:08

Zac-HD and others added 2 commits June 25, 2018 23:04

Handle subtypes in instance_of validator

4fc71dc

Improve error message for attribute

abdaa2d

Zac-HD force-pushed the infer-attrs branch from f0fd9c0 to abdaa2d Compare June 25, 2018 13:30

Zac-HD merged commit bf77d59 into HypothesisWorks:master Jun 26, 2018

Zac-HD deleted the infer-attrs branch June 26, 2018 04:54



		def from_attrs_attribute(attrib):
		"""Infer a strategy from an attr.Attribute object."""

Improve inference in builds() and from_type() for attrs classes #1282

Improve inference in builds() and from_type() for attrs classes #1282

Conversation

Zac-HD commented May 13, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Zac-HD commented May 16, 2018

DRMacIver commented May 16, 2018

Zac-HD commented May 17, 2018

Zac-HD commented May 17, 2018

DRMacIver left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Zac-HD commented May 21, 2018

Zac-HD commented May 21, 2018

hynek commented May 23, 2018

Zac-HD commented May 23, 2018 • edited

Zac-HD commented May 24, 2018

DRMacIver commented May 24, 2018

DRMacIver commented May 24, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Zac-HD Jun 24, 2018 • edited

Choose a reason for hiding this comment

Zac-HD commented May 24, 2018

Zac-HD commented Jun 24, 2018

DRMacIver left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Zac-HD commented Jun 25, 2018 • edited

Zac-HD commented Jun 26, 2018

Zac-HD commented May 13, 2018 •

edited

Zac-HD commented May 23, 2018 •

edited

Zac-HD Jun 24, 2018 •

edited

Zac-HD commented Jun 25, 2018 •

edited