Merge branch 'master' of github.com:helm/acceptance-testing

* 'master' of github.com:helm/acceptance-testing: (41 commits) New helm global flags --kube-token and --kube-apiserver Add completion tests for release names Add completion tests for plugin completion Test kube-context completion without assuming kubectl Group related tests together Don't use a plugin named 'template' for completion Revert "Support if the help is either stdout or stderr (helm#64)" Update namespace flag tests for zsh Some fixes in preparation for plugin completion Make completion tests work with completion in Go Add completion test on centos Fix helm repo update not working Fix some of the KFAILs Fix the ZFAIL cases Support if the help is either stdout or stderr (helm#64) Print ERROR instead of FAIL Use linux image for docker (helm#60) fix: Helm 3 now supports the -c flag helm plugin 'remove' is now 'uninstall' maint: build helm 3 from master ...
akashshinde · Mar 20, 2020 · e79cfe5 · e79cfe5
2 parents edd1036 + c7f0b5e
commit e79cfe5
Show file tree

Hide file tree

Showing 17 changed files with 972 additions and 151 deletions.
diff --git a/.github/workflows/acceptance-tests-pr.yml b/.github/workflows/acceptance-tests-pr.yml
@@ -0,0 +1,23 @@
+name: acceptance-tests-pr
+
+on:
+  pull_request:
+    branches:
+    - master
+
+jobs:
+  run:
+    runs-on: ubuntu-latest
+    steps:
+    - name: Checkout source code
+      uses: actions/checkout@v1
+
+    - name: Run acceptance tests
+      run: make github-actions-ci
+
+    - uses: actions/upload-artifact@master
+      name: Upload test report
+      with:
+        name: helm-acceptance-testing-report-${{ github.sha }}
+        path: acceptance-testing-reports/${{ github.sha }}/
+      if: always()
diff --git a/.github/workflows/acceptance-tests.yml b/.github/workflows/acceptance-tests.yml
@@ -0,0 +1,22 @@
+name: acceptance-tests
+
+on:
+  push:
+    branches:
+      - master
+jobs:
+  run:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout source code
+        uses: actions/checkout@v1
+
+      - name: Run acceptance tests
+        run: make github-actions-ci
+
+      - uses: actions/upload-artifact@master
+        name: Upload test report
+        with:
+          name: helm-acceptance-testing-report-${{ github.sha }}
+          path: acceptance-testing-reports/${{ github.sha }}/
+        if: always()
diff --git a/.gitignore b/.gitignore
@@ -1,2 +1,5 @@
 .acceptance/
 .idea/
+
+acceptance-testing-reports/
+bin/
diff --git a/Makefile b/Makefile
@@ -3,3 +3,16 @@ SHELL = /bin/bash
 .PHONY: acceptance
 acceptance:
 	@scripts/acceptance.sh
+
+.PHONY: github-actions-ci
+github-actions-ci:
+	@scripts/github-actions-ci.sh
+
+.PHONY: github-actions-ci-local
+github-actions-ci-local:
+	docker run -it --rm \
+	    -v $(shell pwd):/tmp/acceptance-testing \
+	    -w /tmp/acceptance-testing  \
+	    --privileged -v /var/run/docker.sock:/var/run/docker.sock \
+	    --entrypoint=/bin/bash ubuntu:latest \
+	    -c 'set +e; scripts/github-actions-ci.sh; echo "Exited $?. (Ctrl+D to exit shell)"; bash'
diff --git a/README.md b/README.md
@@ -1,11 +1,51 @@
 # Helm Acceptance Tests
 
-*Note: these tests have only been run against Helm 3 ([dev-v3](https://github.com/helm/helm/tree/dev-v3))*
+[![GitHub Actions status](https://github.com/helm/acceptance-testing/workflows/acceptance-tests/badge.svg)](https://github.com/helm/acceptance-testing/actions)
 
 This repo contains the source for Helm acceptance tests.
-
 The tests are written using [Robot Framework](https://robotframework.org/).
 
+*Note: these tests have only been run against Helm 3 ([dev-v3](https://github.com/helm/helm/tree/dev-v3))*
+
+## Test Summary
+
+### Kubernetes Versions
+
+Helm is tested to work against the following versions of Kubernetes:
+
+<!-- 
+TODO
+
+Add support for 1.16+, getting the following error:
+Error: apiVersion "apps/v1beta1" in nginx/templates/deployment.yaml is not available
+[1.16.1](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.16.md)
+
+Also, upgrade to 1.15.4 and 1.14.7
+(see issue on kind: https://github.com/kubernetes-sigs/kind/issues/948)
+
+-->
+
+- [1.15.3](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.15.md)
+- [1.14.6](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.14.md)
+
+Test suite: [kubernetes_versions.robot](./testsuites/kubernetes_versions.robot)
+
+
+### Shell Completion
+
+Helm's shell completion functionality is tested against the following shells:
+
+- Bash
+- Zsh
+
+Test suite: [shells.robot](./testsuites/shells.robot)
+
+### Helm Repositories
+
+Basic functionality of the chart repository subsystem is tested.
+
+Test suite: [repos.robot](./testsuites/repos.robot)
+
 ## System requirements
 
 The following tools/commands are expected to be present on the base system
@@ -25,6 +65,13 @@ From the root of this repo, run the following:
 make acceptance
 ```
 
+Alternatively, if you have Docker installed, 
+the system requirements above are not needed, and you can run the following
+command which will simulate CI:
+```
+make github-actions-ci-local
+```
+
 Note: by default, the tests will use helm as found on your PATH.
 To specify a different helm to test, set and export the `ROBOT_HELM_PATH`
 environment variable.  For example, if you have helm v2 installed, but want
@@ -107,7 +154,6 @@ contains a base class called `CommandRunner` that you will likely want to
 leverage when adding support for a new external tool.
 
 The test run is wrapped by [acceptance.sh](./scripts/acceptance.sh) -
-in this file the environment is validated (i.e. check if required tools present).
-
-sinstalled (including Robot Framework itself). If any additional Python libraries
-are required for a new library, it can be appended to `ROBOT_PY_REQUIRES`.
+in this file the environment is validated (i.e. check if required tools present). 
+If any additional Python libraries are required for a new library, 
+it can be appended to `ROBOT_PY_REQUIRES`.
diff --git a/lib/Helm.py b/lib/Helm.py
@@ -5,15 +5,7 @@
 TEST_CHARTS_ROOT_DIR = os.path.abspath(os.path.dirname(os.path.realpath(__file__)) +'/../testdata/charts')
 
 class Helm(common.CommandRunner):
-    def list_releases(self):
-        cmd = 'helm list'
-        self.run_command(kind_auth_wrap(cmd))
-
     def install_test_chart(self, release_name, test_chart, extra_args):
         chart_path = TEST_CHARTS_ROOT_DIR+'/'+test_chart
         cmd = 'helm install '+release_name+' '+chart_path+' '+extra_args
-        self.run_command(kind_auth_wrap(cmd))
-
-    def delete_release(self, release_name):
-        cmd = 'helm delete '+release_name
-        self.run_command(kind_auth_wrap(cmd))
+        self.run_command(kind_auth_wrap(cmd))
diff --git a/lib/Kind.py b/lib/Kind.py
@@ -9,7 +9,7 @@
 MAX_WAIT_KIND_NODE_SECONDS = 60
 KIND_NODE_INTERVAL_SECONDS = 2
 
-MAX_WAIT_KIND_POD_SECONDS = 60
+MAX_WAIT_KIND_POD_SECONDS = 120
 KIND_POD_INTERVAL_SECONDS = 2
 
 KIND_POD_EXPECTED_NUMBER = 8
@@ -39,6 +39,12 @@ def create_test_cluster_with_kubernetes_version(self, kube_version):
             cmd += ' --image='+DOCKER_HUB_REPO+':v'+kube_version
             self.run_command(cmd)
 
+        # Fix for running kind in docker, switch the port+IP in the kubeconfig
+        if os.path.exists('/.dockerenv'):
+            print('Running in Docker, modifying IP in kubeconfig')
+            fixcmd = 'export KIND_IP=$(docker inspect '+LAST_CLUSTER_NAME+'-control-plane | grep \'IPAddress": "\' | head -1 | awk \'{print $2}\' | tr -d \\",) && sed -i "s/https:\/\/127\.0\.0\.1:.*/https:\/\/${KIND_IP}:6443/" $(kind get kubeconfig-path --name="'+LAST_CLUSTER_NAME+'")'
+            self.run_command(fixcmd)
+
     def delete_test_cluster(self):
         if LAST_CLUSTER_EXISTING:
             print('Not deleting cluster (cluster existed prior to test run)')

diff --git a/lib/Kubectl.py b/lib/Kubectl.py
@@ -2,22 +2,6 @@
 from Kind import kind_auth_wrap
 
 class Kubectl(common.CommandRunner):
-    def get_nodes(self):
-        cmd = 'kubectl get nodes'
-        self.run_command(kind_auth_wrap(cmd))
-
-    def get_pods(self, namespace):
-        cmd = 'kubectl get pods --namespace='+namespace
-        self.run_command(kind_auth_wrap(cmd))
-
-    def get_services(self, namespace):
-        cmd = 'kubectl get services --namespace='+namespace
-        self.run_command(kind_auth_wrap(cmd))
-
-    def get_persistent_volume_claims(self, namespace):
-        cmd = 'kubectl get pvc --namespace='+namespace
-        self.run_command(kind_auth_wrap(cmd))
-
     def service_has_ip(self, namespace, service_name):
         cmd = 'kubectl get services --namespace='+namespace
         cmd += ' | grep '+service_name

diff --git a/lib/Sh.py b/lib/Sh.py
@@ -22,7 +22,10 @@
 class Sh(common.CommandRunner):
     def require_cluster(self, require):
         global needs_cluster
-        needs_cluster = require
+        if require == "True" or require == "true":
+            needs_cluster = True
+        else:
+            needs_cluster = False
 
     def wrap(self, cmd):
         global needs_cluster

diff --git a/scripts/acceptance.sh b/scripts/acceptance.sh
@@ -1,4 +1,18 @@
 #!/bin/bash -e
+#
+# Copyright The Helm Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
 
 # Turn on debug printouts if the user requested a debug level >= $1
 set_shell_debug_level()
@@ -39,18 +53,35 @@ set_shell_debug_level 2
 DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
 cd $DIR/../
 
-# We force the tests to use a directory of our own choosing
-# to make sure that when we wipe it clean, we don't wipe
-# some directory that was important to the user.
-# Don't use .helm as it may collide with a valid directory
-FINAL_DIR_NAME=".helm_acceptance_tests"
-
 # Acceptance test configurables
 ROBOT_PY_REQUIRES="${ROBOT_PY_REQUIRES:-robotframework==3.1.2}"
-ROBOT_OUTPUT_DIR="${ROBOT_OUTPUT_DIR:-${PWD}/.acceptance}"
-ROBOT_HELM_HOME_DIR="${ROBOT_HELM_HOME_DIR:-${ROBOT_OUTPUT_DIR}}/${FINAL_DIR_NAME}"
+export ROBOT_OUTPUT_DIR="${ROBOT_OUTPUT_DIR:-${PWD}/.acceptance}"
 ROBOT_VENV_DIR="${ROBOT_VENV_DIR:-${ROBOT_OUTPUT_DIR}/.venv}"
 
+set_shell_debug_level 3
+echo "=============================================================================="
+echo "Available configuration variables:"
+echo "ROBOT_DEBUG_LEVEL - Choose debug level (0 to 3)."
+echo "                    Current: ${ROBOT_DEBUG_LEVEL}"
+echo "ROBOT_HELM_PATH   - The directory where the helm to test can be found."
+echo "                    Current: ${ROBOT_HELM_PATH:-Helm as found on \$PATH: $(dirname $(which helm))}/helm"
+echo "ROBOT_RUN_TESTS   - Comma-separated list of *.robot files to execute."
+echo "                    Current: ${ROBOT_RUN_TESTS:-unset (all)}"
+echo "ROBOT_OUTPUT_DIR  - The output directory for robot to use."
+echo "                    Current: ${ROBOT_OUTPUT_DIR}"
+echo "ROBOT_VENV_DIR    - The directory to be used for virtualenv."
+echo "                    Current: ${ROBOT_VENV_DIR}"
+echo "ROBOT_PY_REQUIRES - Space-separated list of python packages to install (including the robot framework)."
+echo "                    Current: ${ROBOT_PY_REQUIRES}"
+echo "=============================================================================="
+set_shell_debug_level 2
+
+# Only use the -d flag for mktemp as many other flags don't
+# work on every plateform
+mkdir -p ${ROBOT_OUTPUT_DIR}
+export TMP_DIR="$(mktemp -d ${ROBOT_OUTPUT_DIR}/helm-acceptance.XXXXXX)"
+trap "rm -rf ${TMP_DIR}" EXIT
+
 SUITES_TO_RUN=""
 # Allow to specify which test suites to run in a space-separated or comma-separated list
 for suite in ${ROBOT_RUN_TESTS/,/ }; do
@@ -69,21 +100,9 @@ if [ ! -z "${ROBOT_HELM_PATH}" ]; then
 fi
 export PATH="${ROBOT_VENV_DIR}/bin:${PATH}"
 
-set_shell_debug_level 3
-# A bit of safety before wiping the entire directory
-if [ $(basename ${ROBOT_HELM_HOME_DIR}) == "${FINAL_DIR_NAME}" ]; then
-    rm -rf ${ROBOT_HELM_HOME_DIR}
-else
-    echo "ABORT: should not delete unexpected directory ${ROBOT_HELM_HOME_DIR}"
-    echo "ABORT: error in acceptance-testing code!"
-    echo "Please report a bug at https://github.com/helm/acceptance-testing/issues"
-    exit 1
-fi
-set_shell_debug_level 2
-
-export XDG_CACHE_HOME=${ROBOT_HELM_HOME_DIR}/cache && mkdir -p ${XDG_CACHE_HOME}
-export XDG_CONFIG_HOME=${ROBOT_HELM_HOME_DIR}/config && mkdir -p ${XDG_CONFIG_HOME}
-export XDG_DATA_HOME=${ROBOT_HELM_HOME_DIR}/data && mkdir -p ${XDG_DATA_HOME}
+export XDG_CACHE_HOME=${TMP_DIR}/cache && mkdir -p ${XDG_CACHE_HOME}
+export XDG_CONFIG_HOME=${TMP_DIR}/config && mkdir -p ${XDG_CONFIG_HOME}
+export XDG_DATA_HOME=${TMP_DIR}/data && mkdir -p ${XDG_DATA_HOME}
 
 # We fully support helm v3 and partially support helm v2 at this time.
 # To figure out which version of helm is used, we run 'helm version'
@@ -94,7 +113,7 @@ export XDG_DATA_HOME=${ROBOT_HELM_HOME_DIR}/data && mkdir -p ${XDG_DATA_HOME}
 # contact the cluster, which may not be accessible, and the command
 # will timeout.
 set_shell_debug_level 3
-if helm version -c &> /dev/null; then
+if helm version -c --tls &> /dev/null; then
     echo "===================="
     echo "Running with Helm v2"
     echo "===================="