tox is testing the source code instead of the package?

[galsasi1989]

Hi
I have a project called 'sdk':

sdk/
    sdk/
       __init__.py
       file1.py
       file2.py
    tests
    setup.py
    tox.ini
    pytest.ini

This is the content of the tox.ini:

[tox]
envlist = py{37,38}
requires = tox-wheel
sitepackages = true

[testenv]
wheel = true
setenv =
    PIP_INDEX_URL = https://<artifactory_url>/artifactory/api/pypi/pypi/simple
    PACKAGE_VERSION = {env:PACKAGE_VERSION:}
    PYTHON_VERSION = {envname}
    JENKINS_URL = {env:JENKINS_URL:}
deps =
    pytest >=6.1.2,<7
    moto[s3,sqs] >=2.0.7,<3

whitelist_externals = python

commands =
    python -m pytest --color=yes --show-capture=no -v --junitxml=pytest-results-{envname}-unitests.xml

These are first 4 lines in init.py:

import os
import logging
import structlog
from sdk.file1 import validate  # validate function is existed in file1.py

I was wondering whether tox is testing the package which is created and installed in a new virtual environment or the source code in the file system
I would expect that the package will be tested and not the source code. So in order to be sure, I did a test with the following tox-test.ini:

[tox]
envlist = py{37,38}
sitepackages = true
skipsdist = true

[testenv]
setenv =
    PIP_INDEX_URL = https://<artifactory_url>/artifactory/api/pypi/pypi/simple
    PACKAGE_VERSION = {env:PACKAGE_VERSION:}
    PYTHON_VERSION = {envname}
    JENKINS_URL = {env:JENKINS_URL:}
deps =
    pytest >=6.1.2,<7
    moto[s3,sqs] >=2.0.7,<3
    sdk ==1.19.0

whitelist_externals = python

commands =
    python -m pytest --color=yes --show-capture=no -v --junitxml=pytest-results-{envname}-unitests.xml

In the above tox-test.ini, I am just asking tox to skip on creating the package, but only to create a virtualenv with all packages mentioned in deps and run the tests .

But, before running tox using the command: tox -c tox-test.ini,` I renamed the sdk inner directory to be sdk-back and I created a new sdk directory with the same init.py file as in sdk-backup but file1.py and file2.py are now empty files. This is how my project look:

sdk/
    sdk/
       __init__.py
       file1.py    # empty file, the validate function **does not** exist in this file anymore
       file2.py    # empty file
   sdk-backup/
      __init__.py
     file1.py  # the original file1.py
     file2.py  # the original file2.py
    tests
    setup.py
    tox.ini
    tox-test.ini
    pytest.ini

Now, when I am running tox -c tox-test.ini, I would expect that the tests will be run on the installed package sdk ==1.19.0 but then all the tests fail because the validate function from file1.py does not exist anymore, which is not the case in the sdk ==1.19.0.
As I understand it, the installed package sdk ==1.19.0 is not tested but only the source code

Am I missing something? How can I be sure that only the package which tox creates is tested and the source code is ignored?

[jugmac00]

Hi @galsasi1989,

a couple of notes:

How can I be sure that only the package which tox creates is tested and source code is ignored?

You can avoid this problem by moving your source code into a src directory.

There are a couple of blog posts out there about this topic:
https://web.archive.org/web/20170615032800/https://enotuniq.org/
https://hynek.me/articles/testing-packaging/
https://blog.ganssle.io/articles/2019/08/test-as-installed.html

Also, it is important how you run your tests.

You run python -m pytest - this adds the current directory to the path, see https://jugmac00.github.io/til/what-is-the-difference-between-invoking-pytest-and-python-m-pytest/

Did this answer your question?

Cheers,
Jürgen

[galsasi1989]

Hi @jugmac00

Yes, Thanks for your detailed answer!