CoverageWarning raised at end of pytest run during build

[kpinc]

Summary

An CoverageWarning error is raised as pytest completes during the build.
FYI, the OS setup is such that ansible, ansible-lint, etc. is not installed in the OS.

Issue Type

• Bug Report

Ansible and Ansible Lint details

The following venv is identical, but is patched per pull request #1866 to support running ansible-lint without activation of a venv.

$ .tox/py39-core/bin/ansible-lint --version
ansible-lint 5.3.3.dev33 using ansible 2.12.2

• ansible installation method: pip (via tox created venv)
• ansible-lint installation method: pip (via tox created package from source in a tox created venv)

OS / ENVIRONMENT

Debian stable (bullseye, 11.2)
Python 3.9.2

Note that none of the OS ansible packages (ansible, ansible-lint, etc) are installed. There is no ansible in the OS.

STEPS TO REPRODUCE

virtualenv -p venv
git clone https://github.com/ansible-community/ansible-lint.git
cd ansible-lint
../venv/bin/tox

Desired Behaviour

Suppression of the warnings if they are meaningless. Or fix the problem that's creating the warning. Or some explanatory output describing the impact of the problem instead of a "generic" message, if the problem can't be fixed.

Actual Behaviour

Produces the following output for the py39-core build, and similar output for the other py39-* builds. (The 3.10, 3.8, 3.7, and 3.6 builds are skipped on my box.)

py39-core run-test: commands[1] | /home/kop/projects/ansible/ansible-lint-head/.tox/py39-core/bin/python -m pytest --junitxml /home/kop/projects/ansible/ansible-lint-head/.tox/junit.py39-core.xml -m 'not eco' -p pytest_cov --cov ansiblelint --cov /home/kop/projects/ansible/ansible-lint-head/.tox/py39-core/lib/python3.9/site-packages/ansiblelint --cov-report term-missing:skip-covered --no-cov-on-fail
============================= test session starts ==============================
platform linux -- Python 3.9.2, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
cachedir: .tox/py39-core/.pytest_cache
rootdir: /home/kop/projects/ansible/ansible-lint-head, configfile: pytest.ini
plugins: cov-3.0.0, xdist-2.5.0, forked-1.4.0, flaky-3.7.0
gw0 [393] / gw1 [393] / gw2 [393] / gw3 [393]
...........................................................x............ [ 18%]
..................................................x..................... [ 36%]
........................................................................ [ 54%]
........................................................................ [ 73%]
........................................................................ [ 91%]
............................../home/kop/projects/ansible/ansible-lint-head/.tox/py39-core/lib/python3.9/site-packages/coverage/inorout.py:536: CoverageWarning: Module ansiblelint was previously imported, but not measured (module-not-measured)
  self.warn(msg, slug="module-not-measured")
/home/kop/projects/ansible/ansible-lint-head/.tox/py39-core/lib/python3.9/site-packages/coverage/inorout.py:520: CoverageWarning: Module /home/kop/projects/ansible/ansible-lint-head/.tox/py39-core/lib/python3.9/site-packages/ansiblelint was never imported. (module-not-imported)
  self.warn(f"Module {pkg} was never imported.", slug="module-not-imported")
./home/kop/projects/ansible/ansible-lint-head/.tox/py39-core/lib/python3.9/site-packages/coverage/inorout.py:536: CoverageWarning: Module ansiblelint was previously imported, but not measured (module-not-measured)
  self.warn(msg, slug="module-not-measured")
/home/kop/projects/ansible/ansible-lint-head/.tox/py39-core/lib/python3.9/site-packages/coverage/inorout.py:520: CoverageWarning: Module /home/kop/projects/ansible/ansible-lint-head/.tox/py39-core/lib/python3.9/site-packages/ansiblelint was never imported. (module-not-imported)
  self.warn(f"Module {pkg} was never imported.", slug="module-not-imported")
./home/kop/projects/ansible/ansible-lint-head/.tox/py39-core/lib/python3.9/site-packages/coverage/inorout.py:536: CoverageWarning: Module ansiblelint was previously imported, but not measured (module-not-measured)
  self.warn(msg, slug="module-not-measured")
/home/kop/projects/ansible/ansible-lint-head/.tox/py39-core/lib/python3.9/site-packages/coverage/inorout.py:520: CoverageWarning: Module /home/kop/projects/ansible/ansible-lint-head/.tox/py39-core/lib/python3.9/site-packages/ansiblelint was never imported. (module-not-imported)
  self.warn(f"Module {pkg} was never imported.", slug="module-not-imported")
.                                        [100%]/home/kop/projects/ansible/ansible-lint-head/.tox/py39-core/lib/python3.9/site-packages/coverage/inorout.py:536: CoverageWarning: Module ansiblelint was previously imported, but not measured (module-not-measured)
  self.warn(msg, slug="module-not-measured")
/home/kop/projects/ansible/ansible-lint-head/.tox/py39-core/lib/python3.9/site-packages/coverage/inorout.py:520: CoverageWarning: Module /home/kop/projects/ansible/ansible-lint-head/.tox/py39-core/lib/python3.9/site-packages/ansiblelint was never imported. (module-not-imported)
  self.warn(f"Module {pkg} was never imported.", slug="module-not-imported")

- generated xml file: /home/kop/projects/ansible/ansible-lint-head/.tox/junit.py39-core.xml -

----------- coverage: platform linux, python 3.9.2-final-0 -----------

Related: pytest-dev/pytest-cov#148

[ssbarnea]

I suspect this may be related to that line https://github.com/ansible-community/ansible-lint/blob/main/conftest.py#L8 --- as that is loaded before pylint finishes to load. I seem some people suggested use of coveregy.py instead of pytest-cov but I still wonder if there is something else we can do to address this issue.

[kpinc]

On Fri, 11 Feb 2022 11:43:39 -0800 Sorin Sbarnea ***@***.***> wrote: I suspect this may be related to that line https://github.com/ansible-community/ansible-lint/blob/main/conftest.py#L8 --- as that is loaded before pylint finishes to load. I seem some people suggested use of coveregy.py instead of pytest-cov but I still wonder if there is something else we can do to address this issue.

It reminds me of similar feeling issues involving pytest tests of pytest plugins. There's a section on that in the pytest docs. Don't place a lot of faith in this, it is just a feeling. Regards, Karl ***@***.***> Free Software: "You don't pay back, you pay forward." -- Robert A. Heinlein

[ssbarnea]

Based on stackoverflow, the only answer is to use coverage.py, but I will ask on official channel too.

I identified it as being caused by pytest-dev/pytest-cov#148