ENH/TST: Refactor refguide-check, take 3 #20127

ev-br · 2024-02-21T15:22:13Z

Reference issue

Continues and closes gh-16391, supersedes and closes gh-19242

What does this implement/fix?

Continue gh-19242 by @Sheila-nk : refactor the doctesting part of the refguide-check into a standalone tool, use pytest for running and orchestration.

So this PR has essentially two parts:

heavy lifting done by Sheila (see commit attributions)
additional fixes to docstring examples from [WIP]: ENH/TST: Integrate pytest plugin for enhanced doctesting #19242. A part with pytest ignores was suggested by Lucas in [WIP]: ENH/TST: Integrate pytest plugin for enhanced doctesting #19242 (comment), also commit-attributed.

The remaining TODOs follow #19242 (comment). What's relevant now is

finish plumbing the external tool as git submodule
activate checking the code in tutorials
drop the obsoleted part of refguide-check. I'd prefer to do it in a quick follow-up PR.

On the 'scpdt' tool side, we need to

get a better name (doctest_sp, doctest_scipy, scipy_doctest, something else?) --- doctest-scipy unless there are other suggestions
get out of my GH --- will propose to move it to the scipy GH org on the ML

The tool itself is in a bit of a flux at the moment, nothing blocking though. Here's the tracker which is being actively worked on: https://github.com/ev-br/scpdt/issues

The pytest plugin is written by @Sheila-nk with inputs from @melissawm and myself. Thank you Sheila and Melissa!

Additional information

I am starting to dislike tagging this all as doctesting. The goal is not ---never was and will never be --- to mix doctests into unit testing. The goal is to keep the examples in the docs current. If somebody comes up with a catchy name for this, I'm all ears :-).

ev-br

Let me flag several tweaks to the docstrings. The fixes are mainly mechanical, may very well be completely wrong, so would appreciate a sanity check from subject experts.

ev-br · 2024-02-21T15:28:55Z

scipy/optimize/_milp.py

@@ -322,7 +322,7 @@ def milp(c, *, integrality=None, bounds=None, constraints=None, options=None):

    >>> A = np.array([[-1, 1], [3, 2], [2, 3]])
    >>> b_u = np.array([1, 12, 12])
-    >>> b_l = np.full_like(b_u, -np.inf)
+    >>> b_l = np.full_like(b_u, -np.inf, dtype=float)


@mdhaber git blame points at you, so I'd ask about this change. Without it, running the example from the milp docstring generates a warning from numpy; adding a dtype above changes the result from [1.0, 2.0] to [2.0, 2.0]:

In [1]: import numpy as np In [2]: c = -np.array([0, 1]) In [3]: A = np.array([[-1, 1], [3, 2], [2, 3]]) ...: >>> b_u = np.array([1, 12, 12]) ...: >>> b_l = np.full_like(b_u, -np.inf) /home/br/mambaforge/envs/scipy-dev/lib/python3.10/site-packages/numpy/core/numeric.py:407: RuntimeWarning: invalid value encountered in cast multiarray.copyto(res, fill_value, casting='unsafe') In [4]: from scipy.optimize import LinearConstraint ...: >>> constraints = LinearConstraint(A, b_l, b_u) In [5]: integrality = np.ones_like(c) In [6]: from scipy.optimize import milp ...: >>> res = milp(c=c, constraints=constraints, integrality=integrality) ...: >>> res.x Out[6]: array([1., 2.])

ev-br · 2024-02-21T15:29:46Z

scipy/sparse/__init__.py

@@ -230,7 +230,6 @@

 >>> A = lil_array((1000, 1000))
 >>> A[0, :100] = rand(100)
->>> A[1, 100:200] = A[0, :100]


@dschult had to remove this line because otherwise it fails and compains about 1D slices.

scipy/stats/qmc.py

scipy/stats/sampling.py

ev-br · 2024-02-21T15:55:56Z

Testing dev.py options:

$ python dev.py test --doctests -j 4 seems to give all workers all tests instead of distributing them? Needs checking, can be done in a follow-up (refguide-check never supported this). Is tracked at pytest-xdist breaks warning filters scipy_doctest#116
$ python dev.py test --doctests -t path/to/file does not work. Can be done in a follow-up, as refguide-check never supported this.

One other bikeshed item is the name of the switch: dev.py test --doctests (current state) or dev.py test --doctests-only?

With regular pytest pytest --doctests means run both unit tests and doctests; Here dev.py test --doctests skips regular unit tests entirely. OTOH --doctests-only is more typing. Thoughts?

tupui

Really cool 🚀

I am starting to dislike tagging this all as doctesting. The goal is not ---never was and will never be --- to mix doctests into unit testing. The goal is to keep the examples in the docs current. If somebody comes up with a catchy name for this, I'm all ears :-).

We have smoke-tests, what about smoke-doc? 😅

scipy/stats/qmc.py

lucascolley · 2024-02-21T16:38:48Z

The commit attribution to me hasn't quite worked due to extra quotation marks, but no biggie :) FWIW I'm a fan of smoke-doc

lucascolley · 2024-02-21T16:40:22Z

scipy/_lib/_testutils.py

+                "--ignore=scipy/interpolate/_interpnd_info.py",
+                "--ignore=scipy/_lib/array_api_compat",
+                "--ignore=scipy/_lib/highs",
+                "--ignore=scipy/_lib/unuran",
+                "--ignore=scipy/_lib/_gcutils.py",
+                "--ignore=scipy/_lib/doccer.py",
+                "--ignore=scipy/_lib/_uarray",


can we get rid of the errors from the files which aren't submodules / vendored?

Probably. Together with scipy/scipy_doctest#94
I'd defer it to a follow-up though.

That said, if you feel like tackling it in this PR Lucas, I won't mind at all :-).

don't tempt me... (no promises but maybe)

Some of them are broken (io.rst? sampling_XXX?); others need a tooling update. This way, the output of $ pytest doc/source/tutorial/ --doctest-glob=*rst --ignore=doc/source/tutorial/examples is green.

…ctests`

[docs only]

While at it, use csr_array not csr_matrix (cosmetics).

Had to remove Ellipsis matching on numeric values in several places. [docs only]

[docs only]

Doctestrings were updated for NumPy 2.0, scalar reprs have changed. This is a minimal fix, just to avoid `dev.py refguide-check` erroring out. [docs only]

[docs only]

ev-br · 2024-05-15T14:43:37Z

No further comments, CI is happy, the plugin installs from PyPI -- let's roll with this.

A huge thank you to Sheila @Sheila-nk for doing the heavy lifting, and thanks to all reviewers.

Use $ python dev.py smoke-docs and dev.py smoke-tutorial instead Both ways of running doctests, smoke-docs and refguide-check, are avaiable in scipy#20127. [docs only]

ev-br requested review from larsoner, tupui, perimosocordiae, andyfaff and rgommers as code owners February 21, 2024 15:22

github-actions bot added scipy.stats scipy.sparse.linalg scipy.sparse scipy.optimize scipy._lib scipy.sparse.csgraph CI Items related to the CI tools such as CircleCI, GitHub Actions or Azure labels Feb 21, 2024

ev-br commented Feb 21, 2024

View reviewed changes

lucascolley added maintenance Items related to regular maintenance tasks and removed scipy.stats scipy.sparse.linalg scipy.sparse scipy.optimize scipy._lib scipy.sparse.csgraph CI Items related to the CI tools such as CircleCI, GitHub Actions or Azure labels Feb 21, 2024

ev-br force-pushed the doctest_plugin branch from fbb1fd4 to ba25862 Compare February 21, 2024 16:30

tupui reviewed Feb 21, 2024

View reviewed changes

scipy/stats/qmc.py Outdated Show resolved Hide resolved

ev-br mentioned this pull request Feb 21, 2024

Run plugin on the scipy codebase scipy/scipy_doctest#77

Closed

3 tasks

lucascolley reviewed Feb 21, 2024

View reviewed changes

ev-br mentioned this pull request Feb 21, 2024

plumb through to scipy scipy/scipy_doctest#42

Closed

lucascolley removed request for perimosocordiae and rgommers February 21, 2024 19:20

ev-br added 21 commits May 15, 2024 11:30

MAINT: doctests: skip broken tutorials

f254697

Some of them are broken (io.rst? sampling_XXX?); others need a tooling update. This way, the output of $ pytest doc/source/tutorial/ --doctest-glob=*rst --ignore=doc/source/tutorial/examples is green.

MAINT: doctests: a more detailed ignore/skip/xfail doctests

36a1111

DEV: refguide-check: add dev.py smoke-docs command, remove `test --do…

d40a32b

…ctests`

DEV: doctests: add dev.py smoke-tutorial command

5366504

CI: run smoke-docs on CircleCI

41fc6e7

[docs only]

MAINT: appease the linter

fc66dcc

DOC: tutorial: fix MPL deprecation warnings

f16be46

CI: split smoke-docs CI run into separate rubrics

2ace6d1

[docs only]

DOC: io: adapt to NumPy 2.0 scalar repr

1cbe995

While at it, use csr_array not csr_matrix (cosmetics).

DOC: sparse.csgraph: adjust docstrings to NumPy 2.0 scalar repr

87c0373

DOC: ndimage: adjust docstring to NumPy 2.0 scalar repr

a812c2d

DOC: stats: adjust docstring to NumPy 2.0 scalar repr

335f0ad

Had to remove Ellipsis matching on numeric values in several places. [docs only]

DOC: tutorial: fix dev.py smoke-tutorial

ba82736

[docs only]

MAINT: stats: fix doctesting infra for NumPy < 2

ca6c1c7

MAINT: refguide-check: add skips to make it run under numpy < 2

210e813

Doctestrings were updated for NumPy 2.0, scalar reprs have changed. This is a minimal fix, just to avoid `dev.py refguide-check` erroring out. [docs only]

lint

0a065ed

TST: do not fail if scpdt is not available

649efaf

TST: _lib: smoke-docs may use warnings

6f4126d

MAINT: appease mypy for smoke-docs

8f9f0d4

DOC: smoke-docs: adjust the plugin repo path/name

89bee46

MAINT: smoke-tutorials (plural)

9c8e8c9

[docs only]

ev-br force-pushed the doctest_plugin branch 2 times, most recently from 9d2311f to fff11c2 Compare May 15, 2024 10:01

ev-br added 2 commits May 15, 2024 14:02

DOC: update to the sparse repr change

9044c1a

CI: install the released scipy-doctest

1b291ca

[docs only]

ev-br force-pushed the doctest_plugin branch from fff11c2 to 1b291ca Compare May 15, 2024 11:02

ev-br merged commit 93e0aec into scipy:main May 15, 2024
28 checks passed

ev-br mentioned this pull request Jun 10, 2024

refguide-check with pytest as a runner numpy/numpy#26604

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ENH/TST: Refactor refguide-check, take 3 #20127

ENH/TST: Refactor refguide-check, take 3 #20127

ev-br commented Feb 21, 2024 •

edited

ev-br left a comment

ev-br Feb 21, 2024

ev-br Feb 21, 2024

ev-br commented Feb 21, 2024 •

edited

tupui left a comment

lucascolley commented Feb 21, 2024

lucascolley Feb 21, 2024

ev-br Feb 21, 2024

ev-br Feb 21, 2024

lucascolley Feb 21, 2024

ev-br commented May 15, 2024

ENH/TST: Refactor refguide-check, take 3 #20127

ENH/TST: Refactor refguide-check, take 3 #20127

Conversation

ev-br commented Feb 21, 2024 • edited

Reference issue

What does this implement/fix?

Additional information

ev-br left a comment

Choose a reason for hiding this comment

ev-br Feb 21, 2024

Choose a reason for hiding this comment

ev-br Feb 21, 2024

Choose a reason for hiding this comment

ev-br commented Feb 21, 2024 • edited

tupui left a comment

Choose a reason for hiding this comment

lucascolley commented Feb 21, 2024

lucascolley Feb 21, 2024

Choose a reason for hiding this comment

ev-br Feb 21, 2024

Choose a reason for hiding this comment

ev-br Feb 21, 2024

Choose a reason for hiding this comment

lucascolley Feb 21, 2024

Choose a reason for hiding this comment

ev-br commented May 15, 2024

ev-br commented Feb 21, 2024 •

edited

ev-br commented Feb 21, 2024 •

edited