Before branching day: integration tests

[danilo-gemoli]

These integration tests cover the tools developed in:

• Before branching day: step 1 #2815
• Before branching day: step 2 #2866
• Before branching day: step 4 #3212

Run them locally with:

$ echo -n {rpm-deps-mirroring-services,release-controller-config-manager,generated-release-gating-jobs} \
  | xargs -d' ' -I{} /bin/bash -c 'cd "${CI_TOOLS_REPO}/cmd/branchingconfigmanagers/{}" && go install'
$ make integration SUITE=br-cuts-*

/cc @jmguzik @droslean

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: danilo-gemoli

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [danilo-gemoli]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[jmguzik]

LGTM but I am not an expert in the field of integration tests in our repo. I am also not sure if we really need them for such tiny tools.
Anyway, @droslean I give time until tomorrow to complain about this commit, otherwise, I'll LGTM :P

[droslean]

/test e2e

[droslean]

@danilo-gemoli Can you elaborate more on why we need all of those .sh scripts? The integration tests should just run the binary and we need at least one small script to compare the output.

So we need the actual and expected data to already exist instead of generating temporary ones.

[danilo-gemoli]

Sure. I've added three integration tests, along with their .sh scripts:

• br-cuts-generated-release-jobs.sh
• br-cuts-release-controller-configs.sh
• br-cuts-rpm-mirroring-services.sh

Each of which setups the whole test, declares a cleanup function, some variables and finally
registers another function (ex.: function branchcuts::rpm_mirroring::run_config_manager()) to run the proper tool.

Each test has its tests cases located in test/integration/branchcuts/$TOOL_UNDER_TEST/$NTH_TEST_CASE, where:

• data contains the files that the tool is expected to bump
• want contains the expected results
• test_config is a simple $KEY=$VALUE text file that holds the configuration the tool needs to be run with (ex.: current_release=4.12, future_release=4.13)

The script hack/lib/branchcuts/test-common.sh holds the code common to all the tests, branchcuts::run_tests_template() being the core function.
That template function loops through every test case, runs a tool, compares the resulting output with
the expected one and finally reports either success or failure.

What follows is an example of execution.
Integration test br-cuts-generated-release-jobs is going to be executed:

• br-cuts-generated-release-jobs.sh runs and then declares:
  • $test_cases_dir: a variable defining where to find the test cases
  • branchcuts::generated_release_jobs::run_config_manager a function that knows how to run the tool generated-release-gating-jobs
• branchcuts::run_tests_template (in test-common.sh) is called and therefore, for every test case, the following happens:
  • test_config is read and parsed
  • some temporary folders are created; those being like the ones generated-release-gating-jobs is expected to read the files from. In this case: ci-operator/config/openshift/release
  • the function branchcuts::generated_release_jobs::run_config_manager declared beforehand is finally called, so the tool executes
  • the resulting output is going to be compared to the expected result. Let's suppose test case test1 being run, the expected output is found in test/integration/branchcuts/generated-release-gating-jobs/test1/want/openshift-release-master__ci-4.11.yaml
  • failure or success is finally reported

Actual and expected data are already there, nothing is generated on the fly.

[droslean]

@danilo-gemoli The only temporary files we should generate are the actual data. The test should have input and expected data, and a script to compare the expected with the actual data.

[danilo-gemoli]

Folders structure (like ci-operator/config/openshift/release) only is automatically generated at every run.
Is that what you're referring to? Should it exists in advance?

[danilo-gemoli]

@droslean I have changed it a little bit in accordance with your advice. Not even the folders structured is generated on the fly anymore. Thanks!

[droslean]

There is no need to separate this file into a folder named test1. We need to have all the input files in one place like we have them in the release repo. And to extend different cases you need to add more configs that include those cases. The tool should run only once and generate the desired output.

[danilo-gemoli]

These test cases are much like the ones you can find into the cluster-init integration tests here: test/integration/cluster-init. I can adhere to the same naming conventions input and expected instead of data and want and renaming test1, test2, ... testN to something more meaningful but, other than that, both have test cases split into several folders, each of which recreates part of the openshift/release repository folders structure.
If you take a look at test/integration/cluster-init.sh you'll notice that cluster-init tool runs more than once.

[openshift-ci]

@danilo-gemoli: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e	a2d8835	link	true	/test e2e

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

[openshift-bot]

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

[openshift-ci]

@openshift-bot: Closed this PR.

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.