Helm Acceptance Tests

This repo contains the source for Helm acceptance tests. The tests are written using Robot Framework.

Note: these tests have only been run against Helm 3 (dev-v3)

Test Summary

Kubernetes Versions

Helm is tested to work against the following versions of Kubernetes:

• 1.15.3
• 1.14.6

Test suite: kubernetes_versions.robot

Shell Completion

Helm's shell completion functionality is tested against the following shells:

• Bash
• Zsh

Test suite: shells.robot

Helm Repositories

Basic functionality of the chart repository subsystem is tested.

Test suite: repos.robot

System requirements

The following tools/commands are expected to be present on the base system prior to running the tests:

• kind
• kubectl
• python3
• pip
• virtualenv

Running the tests

From the root of this repo, run the following:

make acceptance

Alternatively, if you have Docker installed, the system requirements above are not needed, and you can run the following command which will simulate CI:

make github-actions-ci-local

Note: by default, the tests will use helm as found on your PATH. To specify a different helm to test, set and export the ROBOT_HELM_PATH environment variable. For example, if you have helm v2 installed, but want to test helm v3 which is located elsewhere; or if you have helm installed but want to test a different development version of helm.

Selecting which test suites to execute

By default make acceptance will run every test suite (*.robot file) present in the directory specified in the environment variable ROBOT_TEST_ROOT_DIR. You can instead specify which test suites to run by setting and exporting variable ROBOT_RUN_TESTS.

For example, to only run the shells.robot suite:

ROBOT_RUN_TESTS=shells.robot
make acceptance

To specify multiple test suites you can set ROBOT_RUN_TESTS to a comma-separated, or space-separated list. For example:

ROBOT_RUN_TESTS=shells.robot,kubernetes_versions.robot
make acceptance

You can use the list format of ROBOT_RUN_TESTS as a way to specify the order in which the test suites should be run. By default (when ROBOT_RUN_TESTS is not specified), the test suites are run in alphabetical order.

Viewing the results

Robot creates an HTML test report describing test successes/failures.

To view the report, runt the following:

open .acceptance/report.html

Note: by default, the tests will output to the .acceptance/ directory. To modify this location, set the ROBOT_OUTPUT_DIR environment variable.

Kubernetes integration

When testing Helm against multiple Kubernetes versions, new test clusters are created on the fly (using kind), with names in the following format:

helm-acceptance-test-<timestamp>-<kube_version>

If you wish to use an existing kind cluster for one or more versions, you can set an environment variable for a given version.

Here is an example of using an existing kind cluster for Kubernetes version 1.15.0:

export KIND_CLUSTER_1_15_0="helm-ac-keepalive-1.15.0"

A kind cluster can be created manually like so:

kind create cluster \
  --name=helm-ac-keepalive-1.15.0 \
  --image=kindest/node:v1.15.0

Adding a new test case etc.

All files ending in .robot extension in this directory will be executed. Add a new file describing your test, or, alternatively, add to an existing one.

Robot tests themselves are written in (mostly) plain English, but the Python programming language can be used in order to add custom keywords etc.

Notice the lib/ directory - this contains Python libraries that enable us to work with system tools such as kind. The file common.py contains a base class called CommandRunner that you will likely want to leverage when adding support for a new external tool.

The test run is wrapped by acceptance.sh - in this file the environment is validated (i.e. check if required tools present). If any additional Python libraries are required for a new library, it can be appended to ROBOT_PY_REQUIRES.