Improving our test suite

[joerick]

Our test suite is slow and kinda flakey (I fairly regularly have to restart jobs). How can we improve this?

Timings from GHA:

platform	time
Windows	54min
macOS	40min
Linux	28min

For reference, the slowest tests are:

Windows (54m)

============================== slowest durations ==============================
561.72s call     test/test_0_basic.py::test
327.08s call     test/test_before_test.py::test
223.82s call     test/test_testing.py::test
213.15s call     test/test_testing.py::test_extras_require
212.83s call     test/test_pep518.py::test_pep518
206.89s call     test/test_environment.py::test
157.00s call     test/test_dependency_versions.py::test_dependency_constraints_file[3.x]
129.20s call     test/test_ssl.py::test
107.84s call     test/test_before_build.py::test
107.40s call     test/test_before_build.py::test_cwd
107.32s call     test/test_before_all.py::test
107.04s call     test/test_before_all.py::test_cwd
104.33s call     test/test_cpp_standards.py::test_cpp11
99.82s call     test/test_cpp_standards.py::test_cpp14
73.40s call     test/test_cpp_standards.py::test_cpp17
58.65s call     test/test_build_skip.py::test
34.73s call     test/test_dependency_versions.py::test_dependency_constraints_file[2.7]
33.24s call     test/test_dependency_versions.py::test_pinned_versions[3.6]
31.56s call     test/test_cpp_standards.py::test_cpp17_py27_modern_msvc_workaround
23.89s call     test/test_subdir_package.py::test
23.18s call     test/test_dependency_versions.py::test_pinned_versions[2.7]
18.21s call     test/test_dependency_versions.py::test_pinned_versions[3.5]
16.14s call     test/test_dependency_versions.py::test_pinned_versions[3.8]
11.27s call     test/test_testing.py::test_failing_test

macOS (40m)

============================== slowest durations ===============================
445.38s call     test/test_0_basic.py::test
206.50s call     test/test_before_test.py::test
164.77s call     test/test_testing.py::test
164.69s call     test/test_pep518.py::test_pep518
157.41s call     test/test_testing.py::test_extras_require
121.63s call     test/test_environment.py::test
117.80s call     test/test_cpp_standards.py::test_cpp11
115.62s call     test/test_ssl.py::test
113.13s call     test/test_cpp_standards.py::test_cpp14
112.49s call     test/test_before_all.py::test
108.86s call     test/test_before_build.py::test
108.60s call     test/test_before_all.py::test_cwd
108.45s call     test/test_before_build.py::test_cwd
100.53s call     test/test_dependency_versions.py::test_dependency_constraints_file[3.x]
48.86s call     test/test_cpp_standards.py::test_cpp17
34.25s call     test/test_dependency_versions.py::test_pinned_versions[3.6]
33.06s call     test/test_build_skip.py::test
30.16s call     test/test_dependency_versions.py::test_dependency_constraints_file[2.7]
29.16s call     test/test_dependency_versions.py::test_pinned_versions[2.7]
11.80s call     test/test_testing.py::test_failing_test
11.65s call     test/test_subdir_package.py::test

Linux (28m)

============================== slowest durations ===============================
157.25s call     test/test_before_test.py::test
141.83s call     test/test_testing.py::test
135.68s call     test/test_0_basic.py::test
134.40s call     test/test_testing.py::test_extras_require
124.20s call     test/test_unicode.py::test
102.13s call     test/test_pep518.py::test_pep518
85.44s call     test/test_docker_images.py::test
82.60s call     test/test_manylinuxXXXX_only.py::test[manylinux1]
79.44s call     test/test_manylinuxXXXX_only.py::test[manylinux2014]
60.76s call     test/test_ssl.py::test
56.83s call     test/test_cpp_standards.py::test_cpp11
55.25s call     test/test_cpp_standards.py::test_cpp14
54.45s call     test/test_before_build.py::test
54.45s call     test/test_before_build.py::test_cwd
54.22s call     test/test_manylinuxXXXX_only.py::test[manylinux2010]
54.00s call     test/test_before_all.py::test_cwd
53.85s call     test/test_environment.py::test
53.73s call     test/test_before_all.py::test
42.41s call     test/test_cpp_standards.py::test_cpp17
25.47s call     test/test_build_skip.py::test
16.48s call     test/test_subdir_package.py::test

[joerick]

One idea is more mocking. We run a lot of integration tests. Could we test these by mocking DockerContainer and shell functions, so we don't need to actually do any I/O to check the feature works? I think we'd still require some full integration tests, but maybe 50% could be downgraded to I/O-less ones?

I haven't tried this, so I don't know how it might work in practice.

[joerick]

We could write a fake pip, so that when we call pip wheel xyz, it actually just copies a premade wheel into place? (The trouble with this idea, is that often the integration tests modify the project, and the rely on that modification to test something...)

[henryiii]

As long as we test a real build at least once for every possible configuration. We should try to avoid doing anything expensive more than once, though!

[joerick]

Do you have an idea of many times we to a full test (ie, build something)?

Reading through the tests as they are written now, quite often! At least, I get a lot of comfort from they way the test suite currently checks various subtleties like where the cwd is or how the correct Python is on the PATH. But I suppose these could be recreated as carefully written asserts.

[YannickJadoul]

But our tests should only be slow the first time, no? Because after that, docker images have been downloaded and Python versions have been installed?

[YannickJadoul]

(that's wrt. "doing something expensive"; but I guess it more fits the other threads, actually)

[joerick]

Other idea - speed up the I/O somehow? Could we run our builds on a RAM disk?

[joerick]

Or, parallelism? Could we leverage GHA's parallelism better to e.g. split the tests between a few containers?

[joerick]

A gotcha with this is that the first test is mega-slow due to downloading the docker images/Python distributions. Maybe we should look into actions/cache?

[joerick]

We get 10 parallel builds on GHA, and 10 for Azure as well. Won't help the other CIs though.

That's true. I suppose it might be nice to have a quick result while working on a PR? But we'd still have to wait for the slowest one before merge.

[joerick]

actions/cache is probably quite a good idea, actually. We could cache the Python distribution downloads, we could cache the pip wheel cache too. Maybe we could cache the Docker images?

[Czaki]

I try with a parallel test, but it speedup only on Linux. On MacOS and Windows it fails because pip cannot work concurrent

[henryiii]

I think this was thinking about running 1/n tests on each of n jobs.

[Czaki]

What is the most common reason of faliture? If a network problem then maybe have some cache with all wheels for packages which are used during the test. pip install option --index-url or --extra-index-url allows for pass path to a directory.

Package Index Options:
  -i, --index-url <url>       Base URL of the Python Package Index (default https://pypi.org/simple). This should point to a repository compliant with PEP 503 (the simple repository API) or a local directory laid out in the same format.
  --extra-index-url <url>     Extra URLs of package indexes to use in addition to --index-url. Should follow the same rules as --index-url.
  --no-index                  Ignore package index (only looking at --find-links URLs instead).
  -f, --find-links <url>      If a URL or path to an html file, then parse for links to archives such as sdist (.tar.gz) or wheel (.whl) files. If a local path or file:// URL that's a directory,  then look for archives in the directory listing. Links to VCS
                              project URLs are not supported.

[YannickJadoul]

Does this take a long time to install?

And doesn't pip already cache installed wheels?

[henryiii]

Does the cache get shared across build and tests? Maybe not be that many shared packages, though.

[YannickJadoul]

Probably not. But you could potentially cache that directory? I just mean we might not need -i or -f or so :-)

[henryiii]

Most common failure on Travis is the docker pull. Often because we hit the free pulls limit that docker added a while ago. We have failed on grabbing the nose download before. And circle / docker non Linux systems fail due to broken permissions sometimes.

[Czaki]

Does this take a long time to install?

And doesn't pip already cache installed wheels?

Pip cache, but some of the packages are required in a later test so fail of a download of files in some later in order test may cause test fail after a long time.

[henryiii]

Adding the Github Action for canceling old builds when new ones are pushed might help. #526

[henryiii]

The problem on CircleCI actually looks like something fixable, perhaps?

You are using pip version 9.0.1, however version 20.3.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
DEPRECATION: Python 3.5 reached the end of its life on September 13th, 2020. Please upgrade your Python as Python 3.5 is no longer maintained. pip 21.0 will drop support for Python 3.5 in January 2021. pip 21.0 will remove support for this functionality.
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
delocate 0.8.2 requires bindepend, which is not installed.
delocate 0.8.2 requires machomachomangler, which is not installed.
DEPRECATION: Python 3.5 reached the end of its life on September 13th, 2020. Please upgrade your Python as Python 3.5 is no longer maintained. pip 21.0 will drop support for Python 3.5 in January 2021. pip 21.0 will remove support for this functionality.

Wonder if this is related to today's release of pip 21?

[henryiii]

      install_requires=[
          "machomachomangler; sys_platform == 'win32'",
          "bindepend; sys_platform == 'win32'",
          "wheel",
      ],

pip 9 maybe doesn't support environment markers?

No, it's at least in 8. Why is is complaining about these two missing?

[Czaki]

why this pip is not updated?

[joerick]

Lots of tests will check something simple-ish on all the versions of Python that we support. We could probably reduce that, to 1-2 versions? I thinking of tests like test_before_all, test_before_build, test_before_test. In fact I already did this in test_subdir_package.