(DRAFT) Testing: Improve tests fun factor

[lavocatt]

Introduction

When implementing a test one needs to wait a long time before it can get
executed resulting in long pauses between two debug sessions. This documents aims at diving into the tests of osbuild-composer, taking measurements and, proposing ideas to address the issues of the CI being a bit long to be fun to use.

Analysis of the tests code base

In this section (drafty at the moment) I'll try to come up with a understanding of each test component and what they do.

The different types of tests:

init

This section takes around 6 minutes to complete

rpmbuild

This section takes around 20 minutes to complete

test

This section is taking around 2h to complete

base

max 00:28:05 minutes

regression

max 00:26:22 minutes

ostree (and variants)

max 01:53:36 hours

integration

max 00:30:41 minutes

api

max 00:32:43 minutes

api_v2

max 34:54 minutes

libvirt

max 33:00 minutes

Setup and teardown

setup

As a setup phase, all of the mentioned scripts are needing deploy.sh which installs all the required dependencies for them to run. The time it takes to run it is unknown and making some measurement could possibly help knowing if merging base + regression could make sense.

teardown

Each test implements its own teardown mhechanism.

A global teardown is called at the end of the pipeline and is called cloud_cleaner.sh

measurements

/!\ these measurements are coarse grain made one only one run.

As we can see on the graphs some jobs from the test stage are outliers that are greatly impacting the critical path of the tests. As we can see on the stats down below, the quantiles for the stage test reveals that 75% of the execution time are under 27 minutes and only a few objects are extremely long, up to a max of 01:53:36.

Max INIT Duration  0 days 00:00:06.418000
Max TEST Duration  0 days 01:53:36.967000
Max RPMBUILD Duration  0 days 00:17:30.706000

INIT quantiles ---
0.500   0 days 00:00:06.418000

TEST quantiles ---
0.100   0 days 00:08:27.943400   
0.250   0 days 00:13:24.442000   
0.500   0 days 00:21:50.957000   
0.750   0 days 00:27:48.193000   
0.990   0 days 01:11:58.709839999
0.999   0 days 01:49:27.141284   

RPMBUILD quantiles ---
0.100   0 days 00:12:28.617800   
0.250   0 days 00:12:50.577000   
0.500   0 days 00:13:51.794000   
0.750   0 days 00:14:27.034000   
0.990   0 days 00:17:13.109079999

Top 10 of the longest tasks

7    libvirt: [openstack/centos-stream-8-x86_64]                             test      success 0 days 00:33:00.193000
78   OSTree: [openstack/rhel-8.4-ga-x86_64]                                  test      success 0 days 00:33:14.129000
17   APIV2: [aws, aws/rhel-8.4-ga-x86_64, true]                              test      failed  0 days 00:34:54.572000
77   OSTree: [openstack/rhel-8.5-devel-x86_64]                               test      success 0 days 00:40:35.303000
1    Installer: [openstack/rhel-9.0-beta-nightly-x86_64]                     test      success 0 days 00:42:46.946000
72   OSTree raw image: [openstack/rhel-8.5-devel-x86_64]                     test      success 0 days 00:46:38.142000
2    Installer: [openstack/rhel-8.5-devel-x86_64]                            test      success 0 days 00:53:41.219000
74   New OSTree (RHEL9-BETA): [openstack/rhel-9.0-beta-nightly-x86_64]       test      success 0 days 00:53:41.397000
73   OSTree simplified installer: [openstack/rhel-8.5-x86_64-large]          test      success 0 days 01:11:07.725000
75   New OSTree: [openstack/rhel-8.5-x86_64-large]                           test      success 0 days 01:53:36.967000

Some first ideas

Split Installer and OSTree tests

These tests are the slower, we should see if it is possible to reduce the time needed to execute them.

I think that we need first a way to time and measure what is taking a long time inside these tests.

Remove redundancy

Integration, API, and API_v2 look like each other a bit. There is maybe some optimisation to do over here.

Enable execution on local machine: Decouple the tests from setup and teardown

As opposed to the unit tests we have in go that one can easily execute and debug on his/her local machine. The tests cannot be executed on the local machine prior to CI. Hence the implementation of a test takes a long time and is not friendly to inattention errors.

Some in development ideas

#1869 from @diaasami

Run necessary tests only based on modified files in the PR.

[jrusz]

Just a note, there was already an attempt to make the execution time of ostree tests shorter #1721

[jrusz]

A few more notes from me:

1. It should be possible to execute any of the tests locally (for the ones interacting with clouds you need to set up correct ENV vars)
2. If you want to just debug your changes you can always create a temporary commit on top of your changes which comments out irrelevant lines from .gitlab-ci.yml file making the time between debug sessions significantly shorter

[diaasami]

I am already working on an idea in #1869, which is basically to run necessary tests only based on modified files in the PR.

[lavocatt]

Cool ! I add you idea to the list :)

[diaasami]

I am still working on it since it requires some changes to the GH -> gitlab setup that we have.
Unfortunately the idea helps in less scenarios than I initially thought, that is mainly because most of the tests that take time span multiple components, so it's not possible to have a file <-> test mapping.
Another thing is that there are some distro-specific source files and distro-specific tests (or runners to be accurate), and it makes sense to connect those together however it doesn't seem that the distro-specific source-files change frequently, I do not believe it's worth it.

But even the basic implementation, the following scenarios will improved by this idea:

1. Changing a specific test, only that test will run and none of the others.
2. Changing non-source files; docs/, some of tools/, *.md etc
3. Creating releases and tagging, etc.