Cannot collect tests : pytest.PytestDeprecationWarning

[taldcroft]

Description

I have no idea what's happened to my setup, but suddenly nothing is working. I tried now started entirely from scratch, including removing my ~/.astropy and making a fresh env for testing.

Test collection fails early with:

ERROR  - pytest.PytestDeprecationWarning: direct construction of DocTestModulePlus has been deprecated, please use DocTestModulePlus.from_parent

Steps to Reproduce

astropy) ➜  astropy git:(master)  conda create -n astropy2 python=3.6      
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /Users/aldcroft/miniconda3/envs/astropy2

  added / updated specs:
    - python=3.6


The following NEW packages will be INSTALLED:

  ca-certificates    pkgs/main/osx-64::ca-certificates-2020.1.1-0
  certifi            pkgs/main/osx-64::certifi-2020.4.5.1-py36_0
  libcxx             pkgs/main/osx-64::libcxx-4.0.1-hcfea43d_1
  libcxxabi          pkgs/main/osx-64::libcxxabi-4.0.1-hcfea43d_1
  libedit            pkgs/main/osx-64::libedit-3.1.20181209-hb402a30_0
  libffi             pkgs/main/osx-64::libffi-3.2.1-h0a44026_6
  ncurses            pkgs/main/osx-64::ncurses-6.2-h0a44026_1
  openssl            pkgs/main/osx-64::openssl-1.1.1g-h1de35cc_0
  pip                pkgs/main/osx-64::pip-20.0.2-py36_1
  python             pkgs/main/osx-64::python-3.6.10-hc70fcce_1
  readline           pkgs/main/osx-64::readline-8.0-h1de35cc_0
  setuptools         pkgs/main/osx-64::setuptools-46.1.3-py36_0
  sqlite             pkgs/main/osx-64::sqlite-3.31.1-h5c1f38d_1
  tk                 pkgs/main/osx-64::tk-8.6.8-ha441bb4_0
  wheel              pkgs/main/osx-64::wheel-0.34.2-py36_0
  xz                 pkgs/main/osx-64::xz-5.2.5-h1de35cc_0
  zlib               pkgs/main/osx-64::zlib-1.2.11-h1de35cc_3


Proceed ([y]/n)? y

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate astropy2
#
# To deactivate an active environment, use
#
#     $ conda deactivate

(astropy) ➜  astropy git:(master)  conda activate astropy2

(astropy2) ➜  astropy git:(master)  git clean -fxd         
Removing .pytest_cache/
...
Removing astropy/wcs/src/docstrings.c
Skipping repository astropy_helpers/
Removing build/
Removing docs/__pycache__/

(astropy2) ➜  astropy git:(master)  pip install -e '.[test]'
Obtaining file:///Users/aldcroft/git/astropy
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting numpy>=1.16
  Using cached numpy-1.18.3-cp36-cp36m-macosx_10_9_x86_64.whl (15.2 MB)
Collecting objgraph; extra == "test"
  Using cached objgraph-3.4.1-py2.py3-none-any.whl (17 kB)
Collecting coverage; extra == "test"
  Using cached coverage-5.1-cp36-cp36m-macosx_10_13_x86_64.whl (203 kB)
Collecting skyfield>=1.20; extra == "test"
  Downloading skyfield-1.20.tar.gz (262 kB)
     |████████████████████████████████| 262 kB 1.6 MB/s 
Collecting ipython; extra == "test"
  Downloading ipython-7.14.0-py3-none-any.whl (782 kB)
     |████████████████████████████████| 782 kB 6.1 MB/s 
Collecting pytest-astropy>=0.8; extra == "test"
  Using cached pytest_astropy-0.8.0-py3-none-any.whl (3.3 kB)
Collecting pytest-xdist; extra == "test"
  Using cached pytest_xdist-1.31.0-py2.py3-none-any.whl (36 kB)
Collecting sgp4; extra == "test"
  Downloading sgp4-2.7-cp36-cp36m-macosx_10_6_intel.whl (165 kB)
     |████████████████████████████████| 165 kB 563 bytes/s 
Collecting graphviz
  Downloading graphviz-0.14-py2.py3-none-any.whl (18 kB)
Requirement already satisfied: certifi>=2017.4.17 in /Users/aldcroft/miniconda3/envs/astropy2/lib/python3.6/site-packages (from skyfield>=1.20; extra == "test"->astropy==4.1.dev1534+g7d2a91538.d20200502) (2020.4.5.1)
Processing /Users/aldcroft/Library/Caches/pip/wheels/48/67/ed/21bb8af8724f9889e7f1f0bb77093a9a71b3f9296574a14837/jplephem-2.14-py3-none-any.whl
Collecting pickleshare
  Using cached pickleshare-0.7.5-py2.py3-none-any.whl (6.9 kB)
Collecting prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0
  Using cached prompt_toolkit-3.0.5-py3-none-any.whl (351 kB)
Requirement already satisfied: setuptools>=18.5 in /Users/aldcroft/miniconda3/envs/astropy2/lib/python3.6/site-packages (from ipython; extra == "test"->astropy==4.1.dev1534+g7d2a91538.d20200502) (46.1.3.post20200330)
Collecting pexpect; sys_platform != "win32"
  Using cached pexpect-4.8.0-py2.py3-none-any.whl (59 kB)
Collecting pygments
  Using cached Pygments-2.6.1-py3-none-any.whl (914 kB)
Collecting decorator
  Using cached decorator-4.4.2-py2.py3-none-any.whl (9.2 kB)
Processing /Users/aldcroft/Library/Caches/pip/wheels/b4/cb/f1/d142b3bb45d488612cf3943d8a1db090eb95e6687045ba61d1/backcall-0.1.0-py3-none-any.whl
Collecting jedi>=0.10
  Using cached jedi-0.17.0-py2.py3-none-any.whl (1.1 MB)
Collecting traitlets>=4.2
  Using cached traitlets-4.3.3-py2.py3-none-any.whl (75 kB)
Collecting appnope; sys_platform == "darwin"
  Using cached appnope-0.1.0-py2.py3-none-any.whl (4.0 kB)
Collecting pytest-doctestplus>=0.2.0
  Using cached pytest_doctestplus-0.5.0-py2.py3-none-any.whl (17 kB)
Collecting pytest-arraydiff>=0.1
  Using cached pytest_arraydiff-0.3-py2.py3-none-any.whl (8.2 kB)
Collecting pytest-astropy-header>=0.1.2
  Using cached pytest_astropy_header-0.1.2-py2.py3-none-any.whl (7.4 kB)
Collecting pytest-filter-subpackage>=0.1
  Using cached pytest_filter_subpackage-0.1.1-py2.py3-none-any.whl (5.1 kB)
Collecting hypothesis>=5.1
  Downloading hypothesis-5.10.4-py3-none-any.whl (292 kB)
     |████████████████████████████████| 292 kB 2.6 MB/s 
Collecting pytest-openfiles>=0.3.1
  Using cached pytest_openfiles-0.5.0-py3-none-any.whl (6.5 kB)
Collecting pytest-remotedata>=0.3.1
  Using cached pytest_remotedata-0.3.2-py2.py3-none-any.whl (8.3 kB)
Collecting pytest>=4.6
  Using cached pytest-5.4.1-py3-none-any.whl (246 kB)
Collecting pytest-cov>=2.0
  Using cached pytest_cov-2.8.1-py2.py3-none-any.whl (18 kB)
Collecting pytest-forked
  Using cached pytest_forked-1.1.3-py2.py3-none-any.whl (4.5 kB)
Collecting execnet>=1.1
  Using cached execnet-1.7.1-py2.py3-none-any.whl (39 kB)
Collecting six
  Using cached six-1.14.0-py2.py3-none-any.whl (10 kB)
Collecting wcwidth
  Using cached wcwidth-0.1.9-py2.py3-none-any.whl (19 kB)
Collecting ptyprocess>=0.5
  Using cached ptyprocess-0.6.0-py2.py3-none-any.whl (39 kB)
Collecting parso>=0.7.0
  Using cached parso-0.7.0-py2.py3-none-any.whl (100 kB)
Collecting ipython-genutils
  Using cached ipython_genutils-0.2.0-py2.py3-none-any.whl (26 kB)
Collecting attrs>=19.2.0
  Using cached attrs-19.3.0-py2.py3-none-any.whl (39 kB)
Collecting sortedcontainers<3.0.0,>=2.1.0
  Using cached sortedcontainers-2.1.0-py2.py3-none-any.whl (28 kB)
Processing /Users/aldcroft/Library/Caches/pip/wheels/a1/d9/f2/b5620c01e9b3e858c6877b1045fda5b115cf7df6490f883382/psutil-5.7.0-cp36-cp36m-macosx_10_9_x86_64.whl
Collecting pluggy<1.0,>=0.12
  Using cached pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Collecting packaging
  Using cached packaging-20.3-py2.py3-none-any.whl (37 kB)
Collecting py>=1.5.0
  Using cached py-1.8.1-py2.py3-none-any.whl (83 kB)
Collecting more-itertools>=4.0.0
  Using cached more_itertools-8.2.0-py3-none-any.whl (43 kB)
Collecting importlib-metadata>=0.12; python_version < "3.8"
  Using cached importlib_metadata-1.6.0-py2.py3-none-any.whl (30 kB)
Collecting apipkg>=1.4
  Using cached apipkg-1.5-py2.py3-none-any.whl (4.9 kB)
Collecting pyparsing>=2.0.2
  Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting zipp>=0.5
  Using cached zipp-3.1.0-py3-none-any.whl (4.9 kB)
Building wheels for collected packages: skyfield
  Building wheel for skyfield (setup.py) ... done
  Created wheel for skyfield: filename=skyfield-1.20-py3-none-any.whl size=297372 sha256=33c228a291bfc2ec55964457b418dc3a51fd8ac8ee13ded05d66283ab83d5607
  Stored in directory: /Users/aldcroft/Library/Caches/pip/wheels/e9/9d/45/451b32f65b27ab00717c980a1df6e41938825840b0ac657922
Successfully built skyfield
Installing collected packages: numpy, graphviz, objgraph, coverage, jplephem, sgp4, skyfield, pickleshare, wcwidth, prompt-toolkit, ptyprocess, pexpect, pygments, decorator, backcall, parso, jedi, six, ipython-genutils, traitlets, appnope, ipython, zipp, importlib-metadata, pluggy, pyparsing, packaging, py, attrs, more-itertools, pytest, pytest-doctestplus, pytest-arraydiff, pytest-astropy-header, pytest-filter-subpackage, sortedcontainers, hypothesis, psutil, pytest-openfiles, pytest-remotedata, pytest-cov, pytest-astropy, pytest-forked, apipkg, execnet, pytest-xdist, astropy
  Running setup.py develop for astropy

Successfully installed apipkg-1.5 appnope-0.1.0 astropy attrs-19.3.0 backcall-0.1.0 coverage-5.1 decorator-4.4.2 execnet-1.7.1 graphviz-0.14 hypothesis-5.10.4 importlib-metadata-1.6.0 ipython-7.14.0 ipython-genutils-0.2.0 jedi-0.17.0 jplephem-2.14 more-itertools-8.2.0 numpy-1.18.3 objgraph-3.4.1 packaging-20.3 parso-0.7.0 pexpect-4.8.0 pickleshare-0.7.5 pluggy-0.13.1 prompt-toolkit-3.0.5 psutil-5.7.0 ptyprocess-0.6.0 py-1.8.1 pygments-2.6.1 pyparsing-2.4.7 pytest-5.4.1 pytest-arraydiff-0.3 pytest-astropy-0.8.0 pytest-astropy-header-0.1.2 pytest-cov-2.8.1 pytest-doctestplus-0.5.0 pytest-filter-subpackage-0.1.1 pytest-forked-1.1.3 pytest-openfiles-0.5.0 pytest-remotedata-0.3.2 pytest-xdist-1.31.0 sgp4-2.7 six-1.14.0 skyfield-1.20 sortedcontainers-2.1.0 traitlets-4.3.3 wcwidth-0.1.9 zipp-3.1.0

(astropy2) ➜  astropy git:(master) pytest astropy/io/ascii
==================================================================== test session starts =====================================================================
platform darwin -- Python 3.6.10, pytest-5.4.1, py-1.8.1, pluggy-0.13.1

Running tests with Astropy version 4.1.dev1534+g7d2a91538.d20200502.
Running tests in astropy/io/ascii.

Date: 2020-05-02T15:05:04

Platform: Darwin-19.2.0-x86_64-i386-64bit

Executable: /Users/aldcroft/miniconda3/envs/astropy2/bin/python

Full Python Version: 
3.6.10 |Anaconda, Inc.| (default, Mar 25 2020, 18:53:43) 
[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)]

encodings: sys: utf-8, locale: UTF-8, filesystem: utf-8
byteorder: little
float info: dig: 15, mant_dig: 15

Package versions: 
Numpy: 1.18.3
Scipy: not available
Matplotlib: not available
h5py: not available
Pandas: not available
Cython: not available
Scikit-image: not available
asdf: not available

Using Astropy options: remote_data: none.

rootdir: /Users/aldcroft/git/astropy, inifile: setup.cfg
plugins: arraydiff-0.3, remotedata-0.3.2, xdist-1.31.0, hypothesis-5.10.4, filter-subpackage-0.1.1, openfiles-0.5.0, forked-1.1.3, doctestplus-0.5.0, astropy-header-0.1.2, cov-2.8.1
collected 0 items / 1 error                                                                                                                                  

=========================================================================== ERRORS ===========================================================================
_______________________________________________________________ ERROR collecting test session ________________________________________________________________
../../miniconda3/envs/astropy2/lib/python3.6/site-packages/pluggy/hooks.py:286: in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
../../miniconda3/envs/astropy2/lib/python3.6/site-packages/pluggy/manager.py:93: in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
../../miniconda3/envs/astropy2/lib/python3.6/site-packages/pluggy/manager.py:87: in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
../../miniconda3/envs/astropy2/lib/python3.6/site-packages/pytest_doctestplus/plugin.py:456: in pytest_collect_file
    return self._doctest_module_item_cls(path, parent)
../../miniconda3/envs/astropy2/lib/python3.6/site-packages/_pytest/nodes.py:83: in __call__
    warnings.warn(NODE_USE_FROM_PARENT.format(name=self.__name__), stacklevel=2)
E   pytest.PytestDeprecationWarning: direct construction of DocTestModulePlus has been deprecated, please use DocTestModulePlus.from_parent
================================================================== short test summary info ===================================================================
ERROR  - pytest.PytestDeprecationWarning: direct construction of DocTestModulePlus has been deprecated, please use DocTestModulePlus.from_parent
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
====================================================================== 1 error in 0.52s ======================================================================

System Details

Darwin-19.2.0-x86_64-i386-64bit
Python 3.6.10 |Anaconda, Inc.| (default, Mar 25 2020, 18:53:43)
[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)]
Numpy 1.18.3
astropy 4.1.dev1534+g7d2a91538.d20200502

[i-am-b-soto]

Hey,
Just started playing around with this repo yesterday,
I'm getting the same error with a slightly different approach.

• Forked this repo.
• Cloned it to my local machine
• 'pip install -e .'
• pip install -e .[test]'

-python

import astropy
astropy.test()

Also, I don't know if this is a related issue or not, but when I try to run the tests from setup.py with:
python setup.py test

I get the error:
ModuleNotFoundError: No module named 'extension_helpers'

I'm using:
Python 3.7.6
Ubuntu 1804
gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Numpy 1.18.3
astropy 4.1.dev1534+g7d2a91538.d20200502

[pllim]

@taldcroft , please downgrade to pytest 5.3

[pllim]

@Iamsoto , pip install extension-helpers

[mhvk]

@taldcroft - I'm having the same issue - the other option is to temporarily remove error in the pytest section of setup.cfg.

If we do not have them already, I guess we should be having some cron CI runs on our pytest extensions with pytest-dev - pytest seems to break things quite regularly.

[taldcroft]

Shouldn't the documented pip install command install an acceptable version of pytest?

[bsipocz]

Let's close this as duplicate of #10039. The issue is with the plugin, which has been fixed, but other issues arose with its new release, thus the release has been reverted to avoid confusion about CI failures during the postponed feature freeze frenzy. Now the discussion is ongoing about the right way to proceed in a few other recent issues and PRs.

As a workaround please downgrade and use pytest 5.3 for now.

[bsipocz]

Shouldn't the documented pip install command install an acceptable version of pytest?

The docs certainly can point out the very pinfile tox uses for CI. Duplicating the pinnings in the docs I don't think is practical though.

[bsipocz]

If we do not have them already, I guess we should be having some cron CI runs on our pytest extensions with pytest-dev - pytest seems to break things quite regularly.

We indeed do keep an eye on new versions, both pytest and sphinx releases are reported on slack, and this issue is widely known an talked about a bit, and in fact fixed, but given this week wasn't the calmest; rolling out the fix has been reverted as other issues have came to light due to the fix.

[mhvk]

@bsipocz - absolutely, and no worries!

[taldcroft]

This is all a bit beyond me, but is it possible to have pinned test dependencies that get installed via pip install -e .[test]? It just seems that this basic thing of installing an environment which is known to succeed for astropy tests should be stable and reproducible and not suddenly break when package X puts out a new release. From my perspective in doing astropy development, I really don't care about the particular package versions so long as things are stable.

[bsipocz]

I suppose in this situation the best would indeed be to use tox to handle all the installations and environments. I don't think these temporary limitations are within scope of setup.cfg, but these version issues are taken care of in the tox.ini and upstream in ci-helpers, and the tox environments are known to be working (as CI relies on them, too).

[taldcroft]

To be honest I don't understand what you said at all (in particular the second sentence). I'll say that tox -e test ... is not acceptable for day-to-day development work. It takes well over a minute every time to run a single test, so this makes writing tests impossible for me given how frequently I need to iterate on them.

What I also don't understand is that this issue is labeled a duplicate of #10039, but that issue is also closed but there is a real active problem at this moment, namely a user comes along and precisely follows the testing documentation instructions and then tests cannot be run. So I don't follow all the details but don't understand how the user is supposed to know to downgrade pytest.

[taldcroft]

Maybe a more useful question is - how can I make an interactive working environment that is the same as what tox -e test creates?

I am in a state right now where tox -e test -- -P io.ascii works just fine but pytest astropy/io/ascii fails to even launch with: astropy.utils.exceptions.AstropyWarning: leap-second auto-update failed due to the following exception: IERSStaleWarning('leap-second file is expired.',)
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

This is what I am getting with the environment I created in the original description, but with pytest downgraded to 5.3. I don't get that error before #10210, but that might be a red-herring to some other problem.

[bsipocz]

That issue is closed as this is upstream, a compatibility issue between pytest-doctestplus 0.5 and pytest 5.4. We do close all other unrelated upstream issues and report them in the relevant upstream repos instead, too, I don't see why testing related ones should be different.

To reuse the env: tox creates a .tox directory for the virtualenvs inside. Run tox once to set up everything and then you can manually run the activate script to bring up the same environment and then iterate on the individual tests.

the version limitations otherwise live in this file: https://raw.githubusercontent.com/astropy/ci-helpers/master/pip_pinnings.txt

[pllim]

@taldcroft , it wasn't explicitly documented because it's a temporary problem that is hopefully fixed by now (?). The timing of freeze and breakage coinciding was really unfortunate. On hindsight, maybe there should have been a pinned issue about it.

Also we turn all warnings into error in setup.cfg to avoid technical debt piling up again. We had it off for years and then fixing the 2000+ warnings wasn't very fun. Thank you for your understanding.

[mhvk]

Absolutely need to have error in setup.cfg - I think the main problem really is pytest moving quite fast (and it is getting nicer and nicer, no complaints there), which makes it hard to keep up for plugins. Plus, frankly a bug in pytest for not allowing one to silence deprecation errors outside of the code one is actually testing! (in the init phase). Anyway, definitely going to ensure you don't have to go through all those warnings again!!!!

[taldcroft]

I also completely agree on turning warnings into errors, that was never in question.

I have been just suggesting that the environment installed by pip install -e .[test] include at least some pinning (in particular on pytest which seems notorious for breaking things in ways that are inscrutable) so that local developer testing does not break suddenly for mysterious reasons.

And don't get me wrong, I stand in awe and admiration for the infrastructure team doing all these things I can't even imagine and fighting the constant battle to keep it all together. It's just a tough position where anything less than perfect can end up being a show-stopper.

[mhvk]

Was about to add to this, but really this is best done in a new issue - #10303. If someone explains how to do it there, I'll volunteer for adding it to the documentation.