Support @example(...).via("your string here")

hypothesis-python/RELEASE.rst

@@ -0,0 +1,5 @@
+RELEASE_TYPE: minor
+
+The :obj:`@example(...) <hypothesis.example>` decorator now has a ``.via()``
+method, which future tools will use to track automatically-added covering
+examples (:issue:`3506`).

hypothesis-python/docs/changes.rst

@@ -144,7 +144,7 @@ the standard.
 6.54.6 - 2022-09-18
 -------------------
 
-If multiple explicit examples (from :func:`@example() <hypothesis.example>`)
+If multiple explicit examples (from :obj:`@example() <hypothesis.example>`)
 raise a Skip exception, for consistency with generated examples we now re-raise
 the first instead of collecting them into an ExceptionGroup (:issue:`3453`).
 
@@ -293,7 +293,7 @@ working on this at the EuroPython sprints!
 6.50.1 - 2022-07-09
 -------------------
 
-This patch improves the error messages in :func:`@example() <hypothesis.example>`
+This patch improves the error messages in :obj:`@example() <hypothesis.example>`
 argument validation following the recent release of :ref:`6.49.1 <v6.49.1>`.
 
 .. _v6.50.0:
@@ -317,7 +317,7 @@ which led us to revert :ref:`Hypothesis 5.2 <v5.2.0>` last time!
 -------------------
 
 This patch fixes some inconsistency between argument handling for
-:func:`@example <hypothesis.example>` and :func:`@given <hypothesis.given>`
+:obj:`@example <hypothesis.example>` and :func:`@given <hypothesis.given>`
 (:issue:`2706 <2706#issuecomment-1168363177>`).
 
 .. _v6.49.0:
@@ -538,7 +538,7 @@ or :func:`~hypothesis.strategies.register_type_strategy` with types that have no
 6.46.2 - 2022-05-03
 -------------------
 
-This patch fixes silently dropping examples when the :func:`@example <hypothesis.example>`
+This patch fixes silently dropping examples when the :obj:`@example <hypothesis.example>`
 decorator is applied to itself (:issue:`3319`).  This was always a weird pattern, but now it
 works.  Thanks to Ray Sogata, Keeri Tramm, and Kevin Khuong for working on this patch!
 
@@ -2500,7 +2500,7 @@ Permitted values are ``None``, and instances of
 -------------------
 
 This patch fixes :issue:`2696`, an internal error triggered when the
-:func:`@example <hypothesis.example>` decorator was used and the
+:obj:`@example <hypothesis.example>` decorator was used and the
 :obj:`~hypothesis.settings.verbosity` setting was ``quiet``.
 
 .. _v5.43.2:
@@ -2767,7 +2767,7 @@ which makes ``import hypothesis`` around 200 milliseconds faster
 -------------------
 
 This patch adds some helpful suggestions to error messages you might see
-while learning to use the :func:`@example() <hypothesis.example>` decorator
+while learning to use the :obj:`@example() <hypothesis.example>` decorator
 (:issue:`2611`) or the :func:`~hypothesis.strategies.one_of` strategy.
 
 .. _v5.36.0:
@@ -2891,7 +2891,7 @@ Thanks to Nikita Sobolev for fixing :issue:`2604`!
 -------------------
 
 When reporting failing examples, or tried examples in verbose mode, Hypothesis now
-identifies which were from :func:`@example(...) <hypothesis.example>` explicit examples.
+identifies which were from :obj:`@example(...) <hypothesis.example>` explicit examples.
 
 .. _v5.32.1:
 
@@ -3556,7 +3556,7 @@ report to help us improve the shrinker for difficult but realistic workloads.
 -------------------
 
 This release improves the interaction between :func:`~hypothesis.assume`
-and the :func:`@example() <hypothesis.example>` decorator, so that the
+and the :obj:`@example() <hypothesis.example>` decorator, so that the
 following test no longer fails with ``UnsatisfiedAssumption`` (:issue:`2125`):
 
 .. code-block:: python
@@ -4094,7 +4094,7 @@ Miscellaneous
 ~~~~~~~~~~~~~
 - The ``.example()`` method of strategies (intended for interactive
   exploration) no longer takes a ``random`` argument.
-- It is now an error to apply :func:`@example <hypothesis.example>`,
+- It is now an error to apply :obj:`@example <hypothesis.example>`,
   :func:`@seed <hypothesis.seed>`, or :func:`@reproduce_failure <hypothesis.reproduce_failure>`
   without also applying :func:`@given <hypothesis.given>`.
 - You may pass either the ``target`` or ``targets`` argument to stateful rules, but not both.
@@ -4364,7 +4364,7 @@ It is unlikely to have much user visible impact.
 4.51.0 - 2019-12-07
 -------------------
 
-This release deprecates use of :func:`@example <hypothesis.example>`,
+This release deprecates use of :obj:`@example <hypothesis.example>`,
 :func:`@seed <hypothesis.seed>`, or :func:`@reproduce_failure <hypothesis.reproduce_failure>`
 without :func:`@given <hypothesis.given>`.
 
@@ -7710,7 +7710,7 @@ setting ``allow_infinity=False`` and exactly one of ``min_value`` and
 3.66.32 - 2018-08-09
 --------------------
 
-This release adds type hints to the :func:`~hypothesis.example` and
+This release adds type hints to the :obj:`@example() <hypothesis.example>` and
 :func:`~hypothesis.seed` decorators, and fixes the type hint on
 :func:`~hypothesis.strategies.register_type_strategy`. The second argument to
 :func:`~hypothesis.strategies.register_type_strategy` must either be a
@@ -10122,7 +10122,7 @@ This release fixes some minor bugs in argument validation:
 -------------------
 
 This release fixes a bug where test failures that were the result of
-an :func:`@example <hypothesis.example>` would print an extra stack trace before re-raising the
+an :obj:`@example <hypothesis.example>` would print an extra stack trace before re-raising the
 exception.
 
 .. _v3.21.0:
@@ -10673,7 +10673,7 @@ properties such as indexing support or repeated iteration.
 ------------------
 
 This patch fixes a bug in :ref:`3.7.3 <v3.7.3>`, where using
-:func:`@example <hypothesis.example>` and a pytest fixture in the same test
+:obj:`@example <hypothesis.example>` and a pytest fixture in the same test
 could cause the test to fail to fill the arguments, and throw a TypeError.
 
 .. _v3.7.3:

hypothesis-python/docs/data.rst

@@ -330,7 +330,7 @@ strategies interactively. Rather than having to specify everything up front in
 This is similar to :func:`@composite <hypothesis.strategies.composite>`, but
 even more powerful as it allows you to mix test code with example generation.
 The downside of this power is that :func:`~hypothesis.strategies.data` is
-incompatible with explicit :func:`@example(...) <hypothesis.example>`\ s -
+incompatible with explicit :obj:`@example(...) <hypothesis.example>`\ s -
 and the mixed code is often harder to debug when something goes wrong.
 
 If you need values that are affected by previous draws but which *don't* depend

hypothesis-python/docs/quickstart.rst

@@ -97,7 +97,7 @@ a passing test to).
         return []
 
 If we wanted to make sure this example was always checked we could add it in
-explicitly by using the :func:`@example <hypothesis.example>` decorator:
+explicitly by using the :obj:`@example <hypothesis.example>` decorator:
 
 .. code:: python
 
@@ -115,7 +115,7 @@ of data are valid inputs, or to ensure that particular edge cases such as
 although Hypothesis will :doc:`remember failing examples <database>`,
 we don't recommend distributing that database.
 
-It's also worth noting that both :func:`@example <hypothesis.example>` and
+It's also worth noting that both :obj:`@example <hypothesis.example>` and
 :func:`@given <hypothesis.given>` support keyword arguments as
 well as positional. The following would have worked just as well:
 

hypothesis-python/docs/reproducing.rst

@@ -39,7 +39,7 @@ was printed you can decorate ``test`` with ``@example(n=1)``.
 as a regression test or to cover some edge case - basically combining a
 Hypothesis test and a traditional parametrized test.
 
-.. autofunction:: hypothesis.example
+.. autoclass:: hypothesis.example
 
 Hypothesis will run all examples you've asked for first. If any of them fail it
 will not go on to look for more examples.
@@ -76,6 +76,8 @@ Either are fine, and you can use one in one example and the other in another
 example if for some reason you really want to, but a single example must be
 consistent.
 
+.. automethod:: hypothesis.example.via
+
 .. _reproducing-with-seed:
 
 -------------------------------------

hypothesis-python/src/hypothesis/core.py

@@ -136,25 +136,80 @@ class Example:
     kwargs = attr.ib()
 
 
-def example(*args: Any, **kwargs: Any) -> Callable[[TestFunc], TestFunc]:
+class example:
     """A decorator which ensures a specific example is always tested."""
-    if args and kwargs:
-        raise InvalidArgument(
-            "Cannot mix positional and keyword arguments for examples"
-        )
-    if not (args or kwargs):
-        raise InvalidArgument("An example must provide at least one argument")
 
-    hypothesis_explicit_examples: List[Example] = []
+    def __init__(self, *args: Any, **kwargs: Any) -> None:
+        if args and kwargs:
+            raise InvalidArgument(
+                "Cannot mix positional and keyword arguments for examples"
+            )
+        if not (args or kwargs):
+            raise InvalidArgument("An example must provide at least one argument")
 
-    def accept(test):
+        self.hypothesis_explicit_examples: List[Example] = []
+        self._this_example = Example(tuple(args), kwargs)
+
+    def __call__(self, test: TestFunc) -> TestFunc:
         if not hasattr(test, "hypothesis_explicit_examples"):
-            test.hypothesis_explicit_examples = hypothesis_explicit_examples
-        test.hypothesis_explicit_examples.append(Example(tuple(args), kwargs))
+            test.hypothesis_explicit_examples = self.hypothesis_explicit_examples  # type: ignore
+        test.hypothesis_explicit_examples.append(self._this_example)  # type: ignore
         return test
 
-    accept.hypothesis_explicit_examples = hypothesis_explicit_examples  # type: ignore
-    return accept
+    def via(self, *whence: str) -> "example":
+        """Attach a machine-readable label noting whence this example came.
+
+        The idea is that tools will be able to add ``@example()`` cases for you, e.g.
+        to maintain a high-coverage set of explicit examples, but also *remove* them
+        if they become redundant - without ever deleting manually-added examples:
+
+        .. code-block:: python
+
+            # You can choose to annotate examples, or not, as you prefer
+            @example(...)
+            @example(...).via("regression test for issue #42")
+
+            # The `hy-` prefix is reserved for automated tooling
+            @example(...).via("hy-failing")
+            @example(...).via("hy-coverage")
+            @example(...).via("hy-target-$label")
+            def test(x):
+                pass
+
+        Note that this "method chaining" syntax requires Python 3.9 or later, for
+        :pep:`614` relaxing grammar restrictions on decorators.  If you need to
+        support older versions of Python, you can use an identity function:
+
+        .. code-block:: python
+
+            def identity(x):
+                return x
+
+
+            @identity(example(...).via("label"))
+            def test(x):
+                pass
+
+        """
+        if len(whence) != 1 or not isinstance(whence[0], str):
+            raise InvalidArgument(".via() must be passed a string")
+        # This is deliberately a no-op at runtime; the tools operate on source code.
+        return self
+
+    if sys.version_info[:2] >= (3, 8):  # pragma: no branch
+        # We want a positional-only argument, and on Python 3.8+ we'll get it.
+        __sig = get_signature(via)
+        via = define_function_signature(
+            name=via.__name__,
+            docstring=via.__doc__,
+            signature=__sig.replace(
+                parameters=[
+                    p.replace(kind=inspect.Parameter.POSITIONAL_ONLY)
+                    for p in __sig.parameters.values()
+                ]
+            ),
+        )(via)
+        del __sig
 
 
 def seed(seed: Hashable) -> Callable[[TestFunc], TestFunc]:

hypothesis-python/tests/cover/test_example.py

@@ -18,6 +18,7 @@
 from hypothesis import example, find, given, strategies as st
 from hypothesis.errors import (
     HypothesisException,
+    InvalidArgument,
     NonInteractiveExampleWarning,
     Unsatisfiable,
 )
@@ -84,3 +85,22 @@ def test_interactive_example_does_not_emit_warning():
     child.sendline("from hypothesis.strategies import none")
     child.sendline("none().example()")
     child.sendline("quit(code=0)")
+
+
+def identity(decorator):
+    # The "identity function hack" from https://peps.python.org/pep-0614/
+    # Method-chaining decorators are otherwise a syntax error in Python <= 3.8
+    return decorator
+
+
+@identity(example(False).via("Manually specified"))
+@given(st.booleans())
+def test_ok_example_via(x):
+    pass
+
+
+def test_invalid_example_via():
+    with pytest.raises(InvalidArgument):
+        example(x=False).via(100)  # not a string!
+    with pytest.raises(TypeError):
+        example(x=False).via("abc", "def")  # too many args