Demo for Fault Injection with k6 using xk6-disruptor

Project for demonstrating fault injection with k6 and xk6-disruptor on the OpenTelemetry Demo in Kubernetes.

The scenario is that we have our system under test (SUT) provided by the most excellent OpenTelemetry Astronomy Shop. The entire system will be running within a single Kubernetes cluster on which we can inject failures to test reliability.

Prerequisites

We'll be making use of these tools during the course of this demonstration.

• git - For accessing the sourcecode repositories.
• go - To install xk6 and build a custom k6 binary.
• Docker - For running the examples and other tooling.
• k3d - Wrapper to run a k3s Kubernetes cluster in Docker.
• kubectl - Client for talking to Kubernetes clusters.
• helm - Client for deploying software packages into Kubernetes.

There may be others that I didn't recall as having them installed long ago. My apologies for any issues!

Build our extended k6

k6 is flexible tool allowing for custom capabilities via an extension system using xk6. Using xk6 and extensions, you build a new k6 binary incorporating the desired functionality. xk6-disruptor is one of those extensions which are not included in the core k6 product by default.

🔖 Refer to the xk6 guides for the latest information if you run into problems building.

First, ensure xk6 is installed into your Go toolchain:

go install go.k6.io/xk6/cmd/xk6@latest

Next we'll use xk6 to build a new k6 binary with the disruptor source code pulled directly from GitHub:

xk6 build latest \
  --with github.com/grafana/xk6-disruptor

👉 If you'd like additional extensions to try out, take a look at the Bundle section of the k6 documentation to review extensions and build your command line. Make sure xk6-disruptor is selected!

Once the xk6 command finishes building, verify your custom build by checking the version.

./k6 version

You should see output similar to the following:

$ ./k6 version
k6 v0.46.0 ((devel), go1.21.1, darwin/arm64)
Extensions:
  github.com/grafana/xk6-disruptor v0.3.9, k6/x/disruptor [js]

⚠️ Make sure you use the newly built k6 from your current directory! It's possible that you also have another version of k6 on your system path.

Create a local Kubernetes cluster

We'll use k3d to run a local Kubernetes cluster within Docker. Once installed, use the following command to create a three-node cluster named k6-demo-cluster.

k3d cluster create k6-demo-cluster \
 --api-port 6550 \
 -p "8081:80@loadbalancer" \
 --agents 3

👉 If you've previously created the cluster, you can start the cluster using k3d cluster start k6-demo-cluster if not already running.

Once this is complete, you'll now have a running Kubernetes cluster on which you can use kubectl as well as other tooling like k9s.

$ kubectl get nodes
NAME                             STATUS   ROLES                  AGE   VERSION
k3d-k6-demo-cluster-agent-1      Ready    <none>                 15s   v1.27.4+k3s1
k3d-k6-demo-cluster-server-0     Ready    control-plane,master   18s   v1.27.4+k3s1
k3d-k6-demo-cluster-agent-2      Ready    <none>                 14s   v1.27.4+k3s1
k3d-k6-demo-cluster-agent-0      Ready    <none>                 14s   v1.27.4+k3s1

Deploy the OpenTelemetry Astronomy Shop

The Astronomy Shop provides a realistic microservices environment comprised of multiple services, written in various languages, communicating using various protocols including gRPC, Kafka messaging, and HTTP.

The recommended approach for installing the demo application is using a Helmchart.

🔖 Refer to the Kubernetes installation for the latest information if you run into problems deploying.

If not previously done, you'll need to configure your Helm client to look for deployment packages, i.e. Helmcharts, within the OpenTelemetry chart repository. This should only need to be done once.

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts

Next, we'll create an isolated namespace within our Kubernetes cluster, then install the open-telemetry/opentelemetry-demo chart into it as otel-demo.

kubectl create ns otel-demo
helm install otel-demo open-telemetry/opentelemetry-demo -n otel-demo

Here is an example of the output to expect.

$ helm install otel-demo open-telemetry/opentelemetry-demo -n otel-demo
NAME: otel-demo
LAST DEPLOYED: Tue Sep 19 16:26:07 2023
NAMESPACE: otel-demo
STATUS: deployed
REVISION: 1
NOTES:
=======================================================================================


 ██████╗ ████████╗███████╗██╗         ██████╗ ███████╗███╗   ███╗ ██████╗
██╔═══██╗╚══██╔══╝██╔════╝██║         ██╔══██╗██╔════╝████╗ ████║██╔═══██╗
██║   ██║   ██║   █████╗  ██║         ██║  ██║█████╗  ██╔████╔██║██║   ██║
██║   ██║   ██║   ██╔══╝  ██║         ██║  ██║██╔══╝  ██║╚██╔╝██║██║   ██║
╚██████╔╝   ██║   ███████╗███████╗    ██████╔╝███████╗██║ ╚═╝ ██║╚██████╔╝
 ╚═════╝    ╚═╝   ╚══════╝╚══════╝    ╚═════╝ ╚══════╝╚═╝     ╚═╝ ╚═════╝


- All services are available via the Frontend proxy: http://localhost:8080
  by running these commands:
     kubectl port-forward svc/otel-demo-frontendproxy 8080:8080

  The following services are available at these paths once the proxy is exposed:
  Webstore             http://localhost:8080/
  Grafana              http://localhost:8080/grafana/
  Feature Flags UI     http://localhost:8080/feature/
  Load Generator UI    http://localhost:8080/loadgen/
  Jaeger UI            http://localhost:8080/jaeger/ui/

- OpenTelemetry Collector OTLP/HTTP receiver (required for browser spans to be emitted):
  by running these commands:
     kubectl port-forward svc/otel-demo-otelcol 4318:4318

As noted in the above output from the installation, we'll add some port-forwarding to enable access to the services from outside the Kubernetes cluster, i.e. from our desktop.

👉 This can also be accomplished by creating a Port-Forward on the services from within k9s.

# Enable access to the UI elements from the host machine browser.
kubectl port-forward -n otel-demo svc/otel-demo-frontendproxy 8080:8080

With the forwarding enabled, the following links should now be accessible:

• Web Storefront
• Grafana Dashboards
• Feature Flags UI
• Load Generator UI
• Jaeger UI

# Enable publishing Open Telemetry metrics and traces from our k6 tests.
kubectl port-forward -n otel-demo svc/otel-demo-otelcol 4318:4318

Scenario 0 - Viewing base activity

By default, Locust is running generating some activity against our site. Open Grafana, looking at activity for the Recommendation Service.

Scenario 1 - Create increased load in activity

Using k6, let's create some increased load for the service.

./k6 run tests/01-recommendation-spike.js

Scenario 2 - Cause a little chaos with ServiceDisruptor

Using the objects from xk6-disruptor, we'll inject service disruptions to the recommendation service.

./k6 run tests/02-recommendation-service-disruption.js

Scenario 3 - Cause a little chaos with PodDisruptor

Using the objects from xk6-disruptor, we'll inject pod disruptions to pods of the recommendation service.

./k6 run tests/03-recommendation-pod-disruption.js