DOC: Bump Python version on RTD build and fix failure

[pllim]

Description

RTD complained about astropy/table/table.py:docstring of astropy.table.TableColumns.keys::py:class reference target not found: a set-like object providing a view on D's keys on latest build even though the PR merged reported successful RTD build. Hoping that bumping the Python version will fix the problem if it is due to some incompatibilities with older dependencies that cannot be upgraded because Python being used is too old.

UPDATE: I think the failure is real. I think we inherited something from Python that doesn't play well with Sphinx.

Looks like there is also astropy/timeseries/__init__.py:docstring of astropy.timeseries.BoxLeastSquaresResults.clear::py:class reference target not found: None. Remove all items from D..

UPDATE: Comparing the logs from last successful build and the failed build on master branch, I noticed the following:

• numpydoc 1.0.0 -> 1.1.0 (biggest suspect!) -- Let's pin until we get an answer from Sphinx emits "WARNING: py:class reference target not found" with numpydoc 1.1.0 numpy/numpydoc#275
• pillow 7.1.2 -> 7.2.0
• sphinx-build removed -E option -- Don’t use a saved environment (the structure caching all cross-references), but rebuild it completely. The default is to only read and parse source files that are new or have changed since the last run.

UPDATE: Tried to pin using docs/requirements.txt and have it installed via .readthedocs.yml before editable install part but didn't work. A later download somehow upgraded it back to 1.1.0.

TODO:

• After this is merged, open a follow-up issue to unpin when Sphinx emits "WARNING: py:class reference target not found" with numpydoc 1.1.0 numpy/numpydoc#275 is resolved.

[pllim]

Now it is astropy/wcs/__init__.py:docstring of astropy.wcs.Wcsprm.get_cdelt::py:class reference target not found: double array[naxis]. Oh gosh, I hope I don't have to go patch CPython.

[pllim]

Will switching from numpydoc to napoleon fix this? Has been on our wish list over at astropy/sphinx-astropy#5 for a while now.

UPDATE: Wow, that opened a different can of worms. Maybe similar to problem at scipy/scipy#3915 .
astropy/coordinates/baseframe.py:docstring of astropy.coordinates.BaseCoordinateFrame:42:Inline strong start-string without end-string

[pllim]

Found some interesting tidbits at https://stackoverflow.com/questions/11417221/sphinx-autodoc-gives-warning-pyclass-reference-target-not-found-type-warning but doesn't seem to help much in this case.

[pllim]

Well, the switch to napoleon didn't work out, but after getting through some hurdle to build doc locally, here is a full list of warnings that RTD will encounter with the current setup on master:

astropy/table/table.py:docstring of astropy.table.TableColumns.keys:: WARNING: py:class reference target not found: a set-like object providing a view on D's keys
astropy/table/table.py:docstring of astropy.table.TableColumns.values:: WARNING: py:class reference target not found: an object providing a view on D's values
astropy/timeseries/__init__.py:docstring of astropy.timeseries.BoxLeastSquaresResults.clear:: WARNING: py:class reference target not found: None.  Remove all items from D.
astropy/timeseries/__init__.py:docstring of astropy.timeseries.BoxLeastSquaresResults.copy:: WARNING: py:class reference target not found: a shallow copy of D
astropy/timeseries/__init__.py:docstring of astropy.timeseries.BoxLeastSquaresResults.items:: WARNING: py:class reference target not found: a set-like object providing a view on D's items
astropy/timeseries/__init__.py:docstring of astropy.timeseries.BoxLeastSquaresResults.keys:: WARNING: py:class reference target not found: a set-like object providing a view on D's keys
astropy/timeseries/__init__.py:docstring of astropy.timeseries.BoxLeastSquaresResults.pop:: WARNING: py:class reference target not found: v, remove specified key and return the corresponding value.
astropy/timeseries/__init__.py:docstring of astropy.timeseries.BoxLeastSquaresResults.update:: WARNING: py:class reference target not found: None.  Update D from dict/iterable E and F.
astropy/timeseries/__init__.py:docstring of astropy.timeseries.BoxLeastSquaresResults.values:: WARNING: py:class reference target not found: an object providing a view on D's values
astropy/wcs/__init__.py:docstring of astropy.wcs.Wcsprm.get_cdelt:: WARNING: py:class reference target not found: double array[naxis]
astropy/wcs/__init__.py:docstring of astropy.wcs.Wcsprm.get_pc:: WARNING: py:class reference target not found: double array[naxis][naxis]
astropy/wcs/__init__.py:docstring of astropy.wcs.Wcsprm.get_ps:: WARNING: py:class reference target not found: list of tuples
astropy/wcs/__init__.py:docstring of astropy.wcs.Wcsprm.get_pv:: WARNING: py:class reference target not found: list of tuples

Aha! numpydoc 1.1.0 broke our RTD job.

[pllim]

The RTD failure is going to bite unrelated PRs, so better get this in sooner than later.

[mwcraig]

This looks good, thanks for tracking down the issue! Should we open an issue as a reminder to change the version restriction when there is a new numpydoc release?

[mhvk]

@pllim - wow, that was good that I saw this, as I was getting quite stuck with similar warnings on my own package. What a mess! I had thought this could be solved with https://github.com/astropy/astropy/blob/master/docs/nitpick-exceptions, but that doesn't work for these errors.

Aside: for myself, I added a --keep-going flag to sphinx-build so that at least I get all errors in one go. Though it means one has to &^$&#( scroll up again to actually find the warnings.

[mhvk]

p.s. I now see our very own Erik pointed to this file in the answer on the stack-overflow question you pointed to... Still not sure why it doesn't seem to work here.

[Cadair]

All the more reason to drop numpydoc in favour of napoleon.

[astrofrog]

Note about the current warnings:

astropy/table/table.py:docstring of astropy.table.TableColumns.keys:: WARNING: py:class reference target not found: a set-like object providing a view on D's keys
astropy/table/table.py:docstring of astropy.table.TableColumns.values:: WARNING: py:class reference target not found: an object providing a view on D's values
astropy/timeseries/__init__.py:docstring of astropy.timeseries.BoxLeastSquaresResults.clear:: WARNING: py:class reference target not found: None.  Remove all items from D.
astropy/timeseries/__init__.py:docstring of astropy.timeseries.BoxLeastSquaresResults.copy:: WARNING: py:class reference target not found: a shallow copy of D
astropy/timeseries/__init__.py:docstring of astropy.timeseries.BoxLeastSquaresResults.items:: WARNING: py:class reference target not found: a set-like object providing a view on D's items
astropy/timeseries/__init__.py:docstring of astropy.timeseries.BoxLeastSquaresResults.keys:: WARNING: py:class reference target not found: a set-like object providing a view on D's keys
astropy/timeseries/__init__.py:docstring of astropy.timeseries.BoxLeastSquaresResults.pop:: WARNING: py:class reference target not found: v, remove specified key and return the corresponding value.
astropy/timeseries/__init__.py:docstring of astropy.timeseries.BoxLeastSquaresResults.update:: WARNING: py:class reference target not found: None.  Update D from dict/iterable E and F.
astropy/timeseries/__init__.py:docstring of astropy.timeseries.BoxLeastSquaresResults.values:: WARNING: py:class reference target not found: an object providing a view on D's values

This kind of issue is quite common and due to the inherited-members directive here:

.. automodapi:: astropy.timeseries
   :inherited-members:

There are three solutions:

• Overload the above methods and write new docstrings
• Add a skip directive to the above automodapi directive to skip the BoxLeastSquaresResults class and add a separate entry that just documents that class, without the inherited members
• Figure out how to get the nitpick exceptions working for this - it's possible that the presence of a quote (') in the warnings is making this not work properly, so we need to figure out how to escape it correctly, maybe by looking at the nitpick code

astropy/wcs/__init__.py:docstring of astropy.wcs.Wcsprm.get_cdelt:: WARNING: py:class reference target not found: double array[naxis]
astropy/wcs/__init__.py:docstring of astropy.wcs.Wcsprm.get_pc:: WARNING: py:class reference target not found: double array[naxis][naxis]

This is a real error - 'double array[naxis]' is not a real type, so this should be ``numpy.ndarray` and the description should mention any dimensionality restrictions.

astropy/wcs/__init__.py:docstring of astropy.wcs.Wcsprm.get_ps:: WARNING: py:class reference target not found: list of tuples
astropy/wcs/__init__.py:docstring of astropy.wcs.Wcsprm.get_pv:: WARNING: py:class reference target not found: list of tuples

Also a real error I think - should just say list and the description should say what should be in the list?

Switching to Napoleon would not fix this - napoleon doesn't really support the full numpydoc spec, see sphinx-doc/sphinx#6861 for an example of an issue I ran into.

[pllim]

Thanks for the inputs, everyone!

Re: three solutions -- I did start on first two and then started doubting the reality and rolled back. I mean, numpydoc 1.1.0 is a minor release, so why are things breaking so badly for us? Why didn't it complain until now? As for the third one, I wasn't ready to mess with @embray 's magic in case this breakage will be patched upstream by numpydoc.

Re: drop numpydoc in favour of napoleon -- It's not a bed of roses. I got thousands of warnings (like 3k-4k).

So how should I proceed? Should I merge this and open a follow-up issue like @mwcraig said?

[mhvk]

My suggestion would be to do what you do right now, for the moment just fix the numpydoc version so tests pass again, and then raise another issue. I did get a sense over at the numpydoc issue that this might get fixed; or perhaps we try to find a fix ourselves so we can nit-pick.
Though ideally fix the "real" errors that @astrofrog pointed out. I think list of tuple may actually work, i.e., writing it as list of tuple*s* may be the problem!

[pllim]

What about now, @mhvk and @astrofrog ? Hope I didn't go too far with the search and replace.

[mhvk]

It seems good enough for now!

[pllim]

Thanks for the review! I'm taking the liberty to merge now so this failure won't be a blocker for other PRs, especially given it's SciPy week.

[pllim]

To close the loop, follow-up issue is at #10529

[embray]

As for the third one, I wasn't ready to mess with @embray 's magic in case this breakage will be patched upstream by numpydoc

Wait, what was my magic? I'm a little confused.

[pllim]

@embray , I think I meant the stuff you posted in https://stackoverflow.com/questions/11417221/sphinx-autodoc-gives-warning-pyclass-reference-target-not-found-type-warning 😅

[astrofrog]

I am backporting this to v4.1.x to fix some docs failures there