Skip to content

Latest commit

 

History

History
519 lines (386 loc) · 18.2 KB

TESTING.md

File metadata and controls

519 lines (386 loc) · 18.2 KB
Raw

Testing Bottlerocket

🚧 👷

This section is under active development. We are working on tooling for running Bottlerocket integration tests. While the work is underway, there will be frequent changes to this document.

Unit Tests

It is easy to execute unit tests, you can run them from the root of the repo with cargo make unit-tests. Note that some code in Bottlerocket is conditionally compiled based on variant thus some tests won't be executed. Unless you intend to test the default variant, it is best to pass the relevant variant and architecture like this:

cargo make \
  -e BUILDSYS_VARIANT="aws-ecs-1" \
  -e BUILDSYS_ARCH="x86_64" \
  unit-tests

Integration Tests

Unit tests will only get us so far. Ultimately we want to know if Bottlerocket runs correctly as a complete system. We have created a command line utility and testing system to help us test Bottlerocket holistically.

The test system coordinates:

  • the creation of a cluster (or re-use of an existing cluster),
  • creation of Bottlerocket instances,
  • running tests that target the created cluster and instances,
  • terminating the Bottlerocket instances,
  • terminating the Kubernetes cluster (if desired)

Testsys uses a Kubernetes operator to test bottlerocket. The operator runs in a cluster that is separate from the one where you are testing Bottlerocket nodes. We call this control cluster the testsys cluster. When you launch a Bottlerocket integration test, pods run in the testsys cluster to perform the steps described above.

Setup

EKS

It is possible to run your testsys cluster anywhere so long as it has the necessary authorization and networking. We have plans to make this easy to do in EKS by providing the instructions and role permissions you need. However, some work is still needed on the roles, so check back for those instructions in the future!

Using a Temporary Kind Cluster

For developer workflows, the quickest way to run a testsys cluster is using kind.

Important: only use kind for temporary testsys clusters that you will be using yourself. Do not use kind for long-lived clusters or clusters that you will share with other users.

Here are the steps to set up a testsys cluster using kind.

Create a kind cluster (any name will suffice):

kind create cluster --name testsys

If you want to store the kubeconfig file, set the KUBECONFIG variable to some path (there should be no pre-existing file there). It doesn't really matter where this is, since this is a throwaway cluster and then write the kubeconfig to that path. The environment variable TESTSYS_KUBECONFIG is used by all testsys related cargo make tasks.

export TESTSYS_KUBECONFIG="${HOME}/testsys-kubeconfig.yaml"
kind get kubeconfig --name testsys > $TESTSYS_KUBECONFIG

Install the testsys cluster components:

cargo make setup-test

Testsys containers will need AWS credentials.

Reminder: this is for your developer workflow only, do not share this cluster with other users.

cargo make testsys add secret map  \
 --name "creds" \
 "access-key-id=$(aws configure get aws_access_key_id)" \
 "secret-access-key=$(aws configure get aws_secret_access_key)"

If you have a named profile you can use the following.

PROFILE=<Your desired profile name>
cargo make testsys add secret map  \
 --name "creds" \
 "access-key-id=$(aws configure get aws_access_key_id --profile ${PROFILE})" \
 "secret-access-key=$(aws configure get aws_secret_access_key --profile ${PROFILE})"

If you added a secret, you then need to pass the secret's name to testsys through an environment variable:

export TESTSYS_AWS_SECRET_NAME="awsCredentials=<Name of your secret>"

Conveniences

All testsys commands can be run using cargo make to eliminate the chance of 2 different versions of testsys being used. Testsys requires the controller and the agent images to be of the same testsys version.

cargo make testsys <arguments>

The Bottlerocket components are found in the testsys Kubernetes namespace.

Run

Now that you have the testsys cluster set up, it's time to run a Bottlerocket integration test!

Configuration

There are many arguments that can be configured via environment variables with cargo make; however, it is possible to create a configuration file instead. Check out the example config file for a sample Test.toml file.

For example, the instance type can be specified based on variant requirements:

[aws-k8s]
# Set the default instance type for all `aws-k8s` variants
instance-type = "m5.xlarge"

[aws-k8s-nvidia]
# Override the instance type for `nvidia` `aws-k8s` variants
instance-type = "g5g.2xlarge"

Since aws-k8s-nvidia is a <FAMILY>-<FLAVOR> level configuration it will take precedence over aws-k8s which is <FAMILY> level configuration.

Tables can also be created for custom testing configurations. For a custom test type called foo, the config above can be updated:

[aws-k8s]
# Set the default instance type for all `aws-k8s` variants
instance-type = "m5.xlarge"

[aws-k8s.configuration.foo]
# Set the default instance type for all `aws-k8s` variants when `TESTSYS_TEST=foo` is set
instance-type = "m5.8xlarge"

[aws-k8s-nvidia]
# Override the instance type for `nvidia` `aws-k8s` variants
instance-type = "g5g.2xlarge"

[aws-k8s-nvidia.configuration.foo]
# Override the instance type for `nvidia` `aws-k8s` variants when `TESTSYS_TEST=foo` is set
instance-type = "g5g.8xlarge"

Variants

Different Bottlerocket variants require different implementations in the test system. For example, to ensure that Kubernetes variants are working correctly, we use Sonobuoy to run through the K8s E2E conformance test suite. For ECS, we run a task on Bottlerocket to make sure Bottlerocket is working. We use EC2 and EKS for aws-k8s variants and vSphere for vmware-k8s variants, and so on.

We have attempted use sensible defaults for these behaviors when calling the cargo make test command.

aws-k8s

You need to build Bottlerocket and create an AMI before you can run a test. Change the commands below to the desired aws-k8s variant and AWS region:

Caution: An EKS cluster will be created for you. Because these take a long time to create, the default testsys behavior is to leave this in place so you can re-use it. You will need to delete the EKS cluster manually when you are done using it. (EC2 instances are terminated automatically, but it's worth double-checking to make sure they were terminated.)

cargo make \
  -e BUILDSYS_VARIANT="aws-k8s-1.24" \
  -e BUILDSYS_ARCH="x86_64" \
  build

cargo make \
  -e BUILDSYS_VARIANT="aws-k8s-1.24" \
  -e BUILDSYS_ARCH="x86_64" \
  -e PUBLISH_REGIONS="us-west-2" \
  ami

cargo make \
  -e BUILDSYS_VARIANT="aws-k8s-1.24" \
  -e BUILDSYS_ARCH="x86_64" \
  test
cargo make watch-test

Note: You can provision nodes with karpenter by specifying resource-agent-type = "karpenter" in Test.toml. To follow the generic mapping, use the following configuration:

[aws-k8s.configuration.karpenter]
test-type = "quick"
resource-agent-type = "karpenter"
block-device-mapping = [
    {name = "/dev/xvda", volumeType = "gp3", volumeSize = 4, deleteOnTermination = true},
    {name = "/dev/xvdb", volumeType = "gp3", volumeSize = 20, deleteOnTermination = true},
]

This configuration creates a new test type for all aws-k8s variants called karpenter (the string following .configuration in the table heading).

Before launching nodes with karpenter you will need to add the karpenter role to your cluster's aws-auth config map.

# Change to your clusters name
CLUSTER_NAME=my-cluster
ACCOUNT_ID=your-account-id
REGION=us-west-2
eksctl create iamidentity mapping \
  -r ${REGION} \
  --cluster ${CLUSTER_NAME} \
  --arn arn:aws:iam::${ACCOUNT_ID}:role/KarpenterInstanceNodeRole \
  --username system:node:{{EC2PrivateDNSName}} \
  --group system:bootstrappers \
  --group system:nodes

You can run the test by calling,

cargo make -e TESTSYS_TEST=karpenter test

aws-ecs

You need to build Bottlerocket and create an AMI before you can run a test. The default instance type to be used is m5.large for x86_64 and m6g.large for aarch64, but can be controlled by setting the environment variable TESTSYS_INSTANCE_TYPE. This is useful while testing NVIDIA variants, since they require instance types with support for NVIDIA GPUs. Change the commands below to the desired aws-ecs variant and AWS region:

cargo make \
  -e BUILDSYS_VARIANT="aws-ecs-1" \
  -e BUILDSYS_ARCH="x86_64" \
  build

cargo make \
  -e BUILDSYS_VARIANT="aws-ecs-1" \
  -e BUILDSYS_ARCH="x86_64" \
  -e PUBLISH_REGIONS="us-west-2" \
  ami

cargo make \
  -e BUILDSYS_VARIANT="aws-ecs-1" \
  -e BUILDSYS_ARCH="x86_64" \
  test
cargo make watch-test

Note: For more information on publishing AMIs see publishing.

vmware-k8s

First, an initial management cluster needs to be created using EKS Anywhere. You can then set TESTSYS_MGMT_CLUSTER_KUBECONFIG to the path to the management clusters kubeconfig. You need to build Bottlerocket and a publicly accessible TUF repository to test VMware variants. Either Infra.toml or your environment need to be configured. If using environment variables make sure to set the following environment variables:

  • GOVC_URL
  • GOVC_USERNAME
  • GOVC_PASSWORD
  • GOVC_DATACENTER
  • GOVC_DATASTORE
  • GOVC_NETWORK
  • GOVC_RESOURCE_POOL
  • GOVC_FOLDER

Testsys will use the data center specified in Test.toml first. If no data center is specified in Test.toml, testsys will use the first data center listed in Infra.toml VMware testing also requires a control-plane-endpoint to be set in Test.toml for vSphere K8s cluster creation. Change the commands below to the desired vmware-k8s variant:

First, build the VMware variant you want to test.

cargo make \
  -e BUILDSYS_VARIANT="vmware-k8s-1.23" \
  -e BUILDSYS_ARCH="x86_64" \
  build

Build the TUF repo containing the OVA templates.

cargo make \
  -e BUILDSYS_VARIANT="vmware-k8s-1.23" \
  -e BUILDSYS_ARCH="x86_64" \
  repo

Sync TUF repos containing the VMware variant's metadata and targets. Make sure the TUF repos are accessible via unauthenticated HTTP or HTTPS and match the URLs in Infra.toml.

Now, you can run the test.

cargo make \
  -e BUILDSYS_VARIANT="vmware-k8s-1.23" \
  -e BUILDSYS_ARCH="x86_64" \
  test \
  --mgmt-cluster-kubeconfig ${TESTSYS_MGMT_CLUSTER_KUBECONFIG}

You can monitor the tests with:

cargo make watch-test

metal-k8s

First, an initial baremetal management cluster needs to be created using EKS Anywhere. You can then set TESTSYS_MGMT_CLUSTER_KUBECONFIG to the path to the management clusters kubeconfig. You need to build Bottlerocket and a publicly accessible TUF repository to test metal variants. In addition to the management cluster, you will need to prepare a hardware CSV file containing all machines you want to provision and a cluster config for the cluster. Create a directory in tests/shared/clusters with an identifier for this cluster, i.e cluster1 (tests/shared/clusters/cluster1). In that directory create 2 files, cluster.yaml with the EKS Anywhere cluster config, and hardware.csv. In Test.toml set cluster-names = ["cluster1"] to tell TestSys that we want the cluster config and hardware csv from the directory we just created.

Metal testing also requires and additional manual step for testing. The Bottlerocket build system compresses the metal images with lz4, but EKS Anywhere requires them to be gzipped, so before testing make sure to uncompress the lz4 image and gzip it. Make sure it is downloadable from a URL accessible from the management cluster. The directory used should be added to Test.toml as os-image-dir.

Change the commands below to the desired metal-k8s variant:

First, build the Metal variant you want to test.

cargo make \
  -e BUILDSYS_VARIANT="metal-k8s-1.23" \
  -e BUILDSYS_ARCH="x86_64" \
  build

Build the TUF repo containing the metal images.

cargo make \
  -e BUILDSYS_VARIANT="metal-k8s-1.23" \
  -e BUILDSYS_ARCH="x86_64" \
  repo

Make sure you gzip the metal image and add it to your os-image-dir.

Now, you can run the test.

cargo make \
  -e BUILDSYS_VARIANT="metal-k8s-1.23" \
  -e BUILDSYS_ARCH="x86_64" \
  -e TESTSYS_MGMT_CLUSTER_KUBECONFIG=${TESTSYS_MGMT_CLUSTER_KUBECONFIG}
  test

You can monitor the tests with:

cargo make watch-test

Migration Testing

Migration testing is used to ensure Bottlerocket can update from one version to a new version and back. This involves launching Bottlerocket instances, upgrading them, and downgrading them.

Migration testing launches instances of a starting Bottlerocket version, or a provided initial AMI and migrates instances to the target version. In order to accomplish this a few artifacts need to be created:

  • A publicly accessible TUF repository
  • A previous release of Bottlerocket signed with available keys
  • The AMI ID for the previous release
  • Image artifacts and local TUF repos of said artifacts for current changes

The setup

Prepare Infra.toml

We need the URL of an accessible TUF repo so the Bottlerocket instances know where to retrieve the update metadata and targets. Follow our publishing guide to set up TUF repos. Infra.toml is used by testsys to determine TUF repo locations, so metadata_base_url and targets_base_url need to be set based on the repo that was just created. The examples below also assume that the default repo is being used in Infra.toml, but any repo can be used by setting the PUBLISH_REPO environment variable.

Starting Bottlerocket images

In this example we will use v1.9.0 as our starting Bottlerocket version, but any tag from Bottlerocket will work. The following bash script will checkout the proper branch from git and create the build images and TUF repos for testing.

git checkout "v1.9.0"
cargo make
cargo make ami
cargo make repo

Remember to sync your TUF repos with the new metadata and targets.

Target Bottlerocket images

Now, it's time to create the Bottlerocket artifacts that need to be upgraded to.

Switch to the working git branch that should be built from.

WORKING_BRANCH="develop"
git checkout "${WORKING_BRANCH}"

Next, build Bottlerocket images and repos and sync TUF repos. The architecture and variant can be configured with BUILDSYS_ARCH and BUILDSYS_VARIANT.

cargo make
cargo make ami
cargo make repo

Now, sync your TUF repos with the new metadata and targets.

This completes the setup and it is time to test migrations!

Testing Migrations

The previous steps set up the artifacts necessary to perform migration testing using testsys. Ensure all environment variables are still set and set them if they aren't.

To run the migration test set TESTSYS_TEST=migration in the cargo make test call. This will automatically determine the AMI that should be used by finding the latest released version of bottlerocket and checking the user's AMIs to find the correct starting AMI ID. Remember to set the environment variables for the architecture and variant.

cargo make -e TESTSYS_TEST=migration test

To see the state of the tests as they run use cargo make watch-test.

Testing Workloads

Workload tests are tests designed to run as an orchestrated container. A workload test is defined in Test.toml with a map named workloads.

[aws-nvidia]
workloads = { <WORKLOAD-NAME> = "<WORKLOAD-IMAGE-URI>" }

To run the workload test set TESTSYS_TEST=workload in the cargo make test call.

cargo make -e TESTSYS_TEST=workload test

To see the state of the tests as they run use cargo make watch-test.

For more information can be found in the TestSys workload documentation.

Custom Test Types

Custom tests can be run with TestSys by calling cargo make -e TESTSYS_TEST=<CUSTOM-TEST-NAME> test -f <PATH-TO-TEMPLATED-YAML>.

First, a test agent needs to be constructed. The test-agent-cli provides an interface for creating bash based testing agents. Checkout the runbook for instructions on creating an agent.

Once an agent has been created, the yaml template can be created. Values from Test.toml can be inserted into a yaml manifest so that a single manifest can be used for all variants in a family.

apiVersion: {{api-version}}
kind: Test
metadata:
  # The name of the crd created is dependent on the arch and variant for 
  # the test being run.
  name: {{kube-arch}}-{{kube-variant}}-custom
  namespace: {{namespace}}
spec:
  retries: 5
  agent:
    name: custom-test-agent
    image: example-test-agent-cli:latest
    keepRunning: false
    configuration:
      clusterName: {{cluster-name}}
      instanceType: {{instance-type}}
  resources: []
  dependsOn: []
  # The secrets will automatically be populated from the config file, 
  # no template is needed.
  secrets: {}

After the agent has been build and the yaml file is created, the test can be run using cargo make -e TESTSYS_TEST=<CUSTOM-TEST-NAME> test -f <PATH-TO-YAML-FILE>