PowerfulSeal

PowerfulSeal adds chaos to your Kubernetes clusters, so that you can detect problems in your systems as early as possible. It kills targeted pods and takes VMs up and down.

It follows the Principles of Chaos Engineering, and is inspired by Chaos Monkey. Watch the Seal at KubeCon 2017 Austin.

Embrace the inevitable failure. Embrace The Seal.

Highlights

works with OpenStack, AWS, GCP, Azure, and local machines
speaks Kubernetes natively
interactive and autonomous, policy-driven mode
web interface to interact with PowerfulSeal
metric collection and exposition to Prometheus
minimal setup, easy yaml-based policies
easy to extend

Introduction

PowerfulSeal works in several modes:

Interactive mode is designed to allow you to discover your cluster's components and manually break things to see what happens. It operates on nodes, pods, deployments and namespaces.
Autonomous mode reads a policy file, which can contain any number of pod and node scenarios. Each scenario describes a list of matches, filters and actions to execute on your cluster, and will be executed in a loop.
Label mode allows you to specify which pods to kill with a small number of options by adding seal/ labels to pods. This is a more imperative alternative to autonomous mode.
Demo mode allows you to point the Seal at a cluster and a heapster server and let it try to figure out what to kill based on the resource utilisation.

Modes of operation

Interactive mode

$ seal interactive --help
usage: seal interactive [-h] [--kubeconfig KUBECONFIG]
                        (--openstack | --aws | --gcp | --azure | --no-cloud)
                        [--openstack-cloud-name OPENSTACK_CLOUD_NAME]
                        [--gcp-project GCP_PROJECT]
                        [--azure-resource-group-name AZURE_RESOURCE_GROUP_NAME]
                        [--azure-node-resource-group-name AZURE_NODE_RESOURCE_GROUP_NAME]
                        (-i INVENTORY_FILE | --inventory-kubernetes)
                        [--remote-user REMOTE_USER]
                        [--ssh-allow-missing-host-keys]
                        [--override-ssh-host OVERRIDE_SSH_HOST]
                        [--use-private-ip]
                        [--ssh-path-to-private-key SSH_PATH_TO_PRIVATE_KEY | --ssh-password SSH_PASSWORD]
                        [--ssh-kill-command SSH_KILL_COMMAND]

optional arguments:
  -h, --help            show this help message and exit

Kubernetes settings:
  --kubeconfig KUBECONFIG
                        Location of kube-config file

Cloud settings:
  --openstack           use OpenStack cloud provider
  --aws                 use AWS cloud provider
  --gcp                 use Google cloud provider
  --azure               use Azure cloud provider
  --no-cloud            don't use cloud provider
  --openstack-cloud-name OPENSTACK_CLOUD_NAME
                        optional name of the open stack cloud from your config
                        file to use
  --gcp-project GCP_PROJECT
                        id of the google project, required with --gcp
  --azure-resource-group-name AZURE_RESOURCE_GROUP_NAME
                        optional name of the Azure vm cluster resource group.
                        Used to determine azure-node-resource-group-name.
  --azure-node-resource-group-name AZURE_NODE_RESOURCE_GROUP_NAME
                        name of the Azure vm cluster node resource group

Inventory settings:
  -i INVENTORY_FILE, --inventory-file INVENTORY_FILE
                        the inventory file (in ini format) of groups of hosts
                        to work with
  --inventory-kubernetes
                        reads all kubernetes cluster nodes as inventory

SSH settings:
  --remote-user REMOTE_USER
                        the of the user for the ssh connections
  --ssh-allow-missing-host-keys
                        Allow connection to hosts not present in known_hosts
  --override-ssh-host OVERRIDE_SSH_HOST
                        If you'd like to execute all commands on a different
                        host (for example for minikube) you can override it
                        here
  --use-private-ip      Use the private IP of each node (vs public IP)
  --ssh-path-to-private-key SSH_PATH_TO_PRIVATE_KEY
                        Path to ssh private key
  --ssh-password SSH_PASSWORD
                        ssh password
  --ssh-kill-command SSH_KILL_COMMAND
                        The command to execute on remote host to kill the
                        {container_id}

Here's a sneak peek of what you can do in the interactive mode:

Autonomous mode

Autonomous reads the scenarios to execute from the policy file, and runs them:

The matches are combined together and deduplicated to produce an initial working set
They are run through a series of filters
For all the items remaining after the filters, all actions are executed

$ seal autonomous --help
usage: seal autonomous [-h] [--kubeconfig KUBECONFIG]
                       (--openstack | --aws | --gcp | --azure | --no-cloud)
                       [--openstack-cloud-name OPENSTACK_CLOUD_NAME]
                       [--gcp-project GCP_PROJECT]
                       [--azure-resource-group-name AZURE_RESOURCE_GROUP_NAME]
                       [--azure-node-resource-group-name AZURE_NODE_RESOURCE_GROUP_NAME]
                       (-i INVENTORY_FILE | --inventory-kubernetes)
                       [--remote-user REMOTE_USER]
                       [--ssh-allow-missing-host-keys]
                       [--override-ssh-host OVERRIDE_SSH_HOST]
                       [--use-private-ip]
                       [--ssh-path-to-private-key SSH_PATH_TO_PRIVATE_KEY | --ssh-password SSH_PASSWORD]
                       [--ssh-kill-command SSH_KILL_COMMAND] --policy-file
                       POLICY_FILE
                       [--stdout-collector | --prometheus-collector]
                       [--prometheus-host PROMETHEUS_HOST]
                       [--prometheus-port PROMETHEUS_PORT] [--headless]
                       [--host HOST] [--port PORT]

optional arguments:
  -h, --help            show this help message and exit

Kubernetes settings:
  --kubeconfig KUBECONFIG
                        Location of kube-config file

Cloud settings:
  --openstack           use OpenStack cloud provider
  --aws                 use AWS cloud provider
  --gcp                 use Google cloud provider
  --azure               use Azure cloud provider
  --no-cloud            don't use cloud provider
  --openstack-cloud-name OPENSTACK_CLOUD_NAME
                        optional name of the open stack cloud from your config
                        file to use
  --gcp-project GCP_PROJECT
                        id of the google project, required with --gcp
  --azure-resource-group-name AZURE_RESOURCE_GROUP_NAME
                        optional name of the Azure vm cluster resource group.
                        Used to determine azure-node-resource-group-name.
  --azure-node-resource-group-name AZURE_NODE_RESOURCE_GROUP_NAME
                        name of the Azure vm cluster node resource group

Inventory settings:
  -i INVENTORY_FILE, --inventory-file INVENTORY_FILE
                        the inventory file (in ini format) of groups of hosts
                        to work with
  --inventory-kubernetes
                        reads all kubernetes cluster nodes as inventory

SSH settings:
  --remote-user REMOTE_USER
                        the of the user for the ssh connections
  --ssh-allow-missing-host-keys
                        Allow connection to hosts not present in known_hosts
  --override-ssh-host OVERRIDE_SSH_HOST
                        If you'd like to execute all commands on a different
                        host (for example for minikube) you can override it
                        here
  --use-private-ip      Use the private IP of each node (vs public IP)
  --ssh-path-to-private-key SSH_PATH_TO_PRIVATE_KEY
                        Path to ssh private key
  --ssh-password SSH_PASSWORD
                        ssh password
  --ssh-kill-command SSH_KILL_COMMAND
                        The command to execute on remote host to kill the
                        {container_id}

Policy settings:
  --policy-file POLICY_FILE
                        the policy file to run

Metrics settings:
  --stdout-collector    print metrics collected to stdout
  --prometheus-collector
                        store metrics in Prometheus and expose metrics over a
                        HTTP server

Prometheus settings:
  --prometheus-host PROMETHEUS_HOST
                        Host to expose Prometheus metrics via the HTTP server
                        when using the --prometheus-collector flag
  --prometheus-port PROMETHEUS_PORT
                        Port to expose Prometheus metrics via the HTTP server
                        when using the --prometheus-collector flag

Web UI settings:
  --headless            Doesn't start the UI, just runs the policy
  --host HOST           Specify host for the PowerfulSeal web server
  --port PORT           Specify port for the PowerfulSeal web server

Metrics collection

Autonomous mode also comes with the ability for metrics useful for monitoring to be collected. PowerfulSeal currently has a stdout and Prometheus collector. However, metric collectors are easily extensible so it is easy to add your own. More details can be found here.

Web User Interface

If you're not going to use the UI, use the flag `--headless` to disable it

PowerfulSeal comes with a web interface to help you navigate Autonomous Mode. Features include:

starting/stopping autonomous mode
viewing and filtering logs
changing the configuration (either overwriting the remote policy file or copying the changes to clipboard)
stopping/killing individual nodes and pods

Writing policies

A minimal policy file, doing nothing, looks like this:

config:
  minSecondsBetweenRuns: 77
  maxSecondsBetweenRuns: 100

nodeScenarios: []

podScenarios: []

A more interesting schema, that kills a random pod in default namespace every 1-30 seconds:

config:
  minSecondsBetweenRuns: 1
  maxSecondsBetweenRuns: 30

nodeScenarios: []
podScenarios:
  - name: "delete random pods in default namespace"

    match:
      - namespace:
          name: "default"

    filters:
      - randomSample:
          size: 1

    actions:
      - kill:
          probability: 0.77
          force: true

A full featured example listing most of the available options can be found in the tests.

The schemas are validated against the powerful JSON schema.

Label mode

Label mode is a more imperative alternative to autonomous mode, allowing you to specify which specific per-pod whether a pod should be killed, the days/times it can be killed and the probability of it being killed.

To mark a pod for attack, do kubectl label pods my-app-1 seal/enabled=true, and the Seal will start attacking it, but only during working hours (defaults).

Instructions on how to use label mode can be found in LABELS.md.

$ seal label --help
usage: seal label [-h] --kubeconfig KUBECONFIG
                  (--openstack | --aws | --azure | --no-cloud)
                  [--openstack-cloud-name OPENSTACK_CLOUD_NAME]
                  [--azure-resource-group-name AZURE_RESOURCE_GROUP_NAME]
                  [--azure-node-resource-group-name AZURE_NODE_RESOURCE_GROUP_NAME]
                  (-i INVENTORY_FILE | --inventory-kubernetes)
                  [--remote-user REMOTE_USER] [--ssh-allow-missing-host-keys]
                  [--override-ssh-host OVERRIDE_SSH_HOST]
                  [--ssh-path-to-private-key SSH_PATH_TO_PRIVATE_KEY]
                  [--ssh-password SSH_PASSWORD]
                  [--use-private-ip]
                  [--kubernetes-namespace KUBERNETES_NAMESPACE]
                  [--min-seconds-between-runs MIN_SECONDS_BETWEEN_RUNS]
                  [--max-seconds-between-runs MAX_SECONDS_BETWEEN_RUNS]
                  [--stdout-collector | --prometheus-collector]
                  [--prometheus-host PROMETHEUS_HOST]
                  [--prometheus-port PROMETHEUS_PORT]

optional arguments:
  -h, --help            show this help message and exit

Kubernetes settings:
  --kubeconfig KUBECONFIG
                        Location of kube-config file

Cloud settings:
  --openstack           use OpenStack cloud provider
  --aws                 use AWS cloud provider
  --azure               use Azure cloud provider
  --no-cloud            don't use cloud provider
  --openstack-cloud-name OPENSTACK_CLOUD_NAME
                        optional name of the open stack cloud from your config
                        file to use
  --azure-resource-group-name AZURE_RESOURCE_GROUP_NAME
                        optional name of the Azure VM cluster resource group.
                        Used to determine azure-node-resource-group-name if 
                        that is not provided.  
  --azure-node-resource-group-name AZURE_NODE_RESOURCE_GROUP_NAME
                        name of the Azure VM cluster node resource group
                        Required when using Azure cloud provider.

Inventory settings:
  -i INVENTORY_FILE, --inventory-file INVENTORY_FILE
                        the inventory file of groups of hosts to work with
  --inventory-kubernetes
                        reads all kubernetes cluster nodes as inventory

SSH settings:
  --remote-user REMOTE_USER
                        the of the user for the ssh connections
  --ssh-allow-missing-host-keys
                        Allow connection to hosts not present in known_hosts
  --override-ssh-host OVERRIDE_SSH_HOST
                        If you'd like to execute all commands on a different
                        host (for example for minikube) you can override it
                        here
  --ssh-path-to-private-key SSH_PATH_TO_PRIVATE_KEY
                        Path to ssh private key
  --ssh-password SSH_PASSWORD
                        ssh password
  --use-private-ip      Use the private IP of each node (vs public IP)

Kubernetes options:
  --kubernetes-namespace KUBERNETES_NAMESPACE
                        Namespace to use for label and demo mode (set to blank
                        for all namespaces)

Policy settings:
  --min-seconds-between-runs MIN_SECONDS_BETWEEN_RUNS
                        Minimum number of seconds between runs
  --max-seconds-between-runs MAX_SECONDS_BETWEEN_RUNS
                        Maximum number of seconds between runs

Metrics settings:
  --stdout-collector    print metrics collected to stdout
  --prometheus-collector
                        store metrics in Prometheus and expose metrics over a
                        HTTP server

Prometheus settings:
  --prometheus-host PROMETHEUS_HOST
                        Host to expose Prometheus metrics via the HTTP server
                        when using the --prometheus-collector flag
  --prometheus-port PROMETHEUS_PORT
                        Port to expose Prometheus metrics via the HTTP server
                        when using the --prometheus-collector flag

Demo mode

The main way to use PowerfulSeal is to write a policy file for Autonomous mode which reflects realistic failures in your system. However, PowerfulSeal comes with a demo mode to demonstrate how it can cause chaos on your Kubernetes cluster. Demo mode gets all the pods in the cluster, selects those which are using the most resources, then kills them based on a probability.

Demo mode requires Heapster. To run demo mode, use the demo subcommand along with --heapster-path (path to heapster without a trailing slash, e.g., http://localhost:8001/api/v1/namespaces-kube-system/services/heapster/proxy). You can also optionally specify --aggressiveness (from 1 (weakest) to 5 (strongest)) inclusive, as well as --[min/max]-seconds-between-runs.

$ seal demo --help
usage: seal demo [-h] --kubeconfig KUBECONFIG
                 (--openstack | --aws | --azure | --no-cloud)
                 [--openstack-cloud-name OPENSTACK_CLOUD_NAME]
                 [--azure-resource-group-name AZURE_RESOURCE_GROUP_NAME]
                 [--azure-node-resource-group-name AZURE_NODE_RESOURCE_GROUP_NAME]
                 (-i INVENTORY_FILE | --inventory-kubernetes)
                 [--remote-user REMOTE_USER] [--ssh-allow-missing-host-keys]
                 [--override-ssh-host OVERRIDE_SSH_HOST]
                 [--ssh-path-to-private-key SSH_PATH_TO_PRIVATE_KEY]
                 [--ssh-password SSH_PASSWORD]
                 [--use-private-ip]
                 [--kubernetes-namespace KUBERNETES_NAMESPACE]
                 [--min-seconds-between-runs MIN_SECONDS_BETWEEN_RUNS]
                 [--max-seconds-between-runs MAX_SECONDS_BETWEEN_RUNS]
                 [--stdout-collector | --prometheus-collector]
                 [--prometheus-host PROMETHEUS_HOST]
                 [--prometheus-port PROMETHEUS_PORT] --heapster-path
                 HEAPSTER_PATH [--aggressiveness AGGRESSIVENESS]

optional arguments:
  -h, --help            show this help message and exit

Kubernetes settings:
  --kubeconfig KUBECONFIG
                        Location of kube-config file

Cloud settings:
  --openstack           use OpenStack cloud provider
  --aws                 use AWS cloud provider
  --azure               use Azure cloud provider
  --no-cloud            don't use cloud provider
  --openstack-cloud-name OPENSTACK_CLOUD_NAME
                        optional name of the open stack cloud from your config
                        file to use
  --azure-resource-group-name AZURE_RESOURCE_GROUP_NAME
                        optional name of the Azure VM cluster resource group.
                        Used to determine azure-node-resource-group-name if 
                        that is not provided.  
  --azure-node-resource-group-name AZURE_NODE_RESOURCE_GROUP_NAME
                        name of the Azure VM cluster node resource group
                        Required when using Azure cloud provider.

Inventory settings:
  -i INVENTORY_FILE, --inventory-file INVENTORY_FILE
                        the inventory file of groups of hosts to work with
  --inventory-kubernetes
                        reads all kubernetes cluster nodes as inventory

SSH settings:
  --remote-user REMOTE_USER
                        the of the user for the ssh connections
  --ssh-allow-missing-host-keys
                        Allow connection to hosts not present in known_hosts
  --override-ssh-host OVERRIDE_SSH_HOST
                        If you'd like to execute all commands on a different
                        host (for example for minikube) you can override it
                        here
  --ssh-path-to-private-key SSH_PATH_TO_PRIVATE_KEY
                        Path to ssh private key
  --ssh-password SSH_PASSWORD
                        ssh password
  --use-private-ip      Use the private IP of each node (vs public IP)

Kubernetes options:
  --kubernetes-namespace KUBERNETES_NAMESPACE
                        Namespace to use for label and demo mode (set to blank
                        for all namespaces)

Policy settings:
  --min-seconds-between-runs MIN_SECONDS_BETWEEN_RUNS
                        Minimum number of seconds between runs
  --max-seconds-between-runs MAX_SECONDS_BETWEEN_RUNS
                        Maximum number of seconds between runs

Metrics settings:
  --stdout-collector    print metrics collected to stdout
  --prometheus-collector
                        store metrics in Prometheus and expose metrics over a
                        HTTP server

Prometheus settings:
  --prometheus-host PROMETHEUS_HOST
                        Host to expose Prometheus metrics via the HTTP server
                        when using the --prometheus-collector flag
  --prometheus-port PROMETHEUS_PORT
                        Port to expose Prometheus metrics via the HTTP server
                        when using the --prometheus-collector flag

Heapster settings:
  --heapster-path HEAPSTER_PATH
                        Base path of Heapster without trailing slash
  --aggressiveness AGGRESSIVENESS
                        Aggressiveness of demo mode (default: 3)

Setup

Setup includes:

pointing PowerfulSeal at your Kubernetes cluster by giving it a Kubernetes config file
pointing PowerfulSeal at your cloud by specifying the cloud driver to use and providing credentials
making sure that PowerfulSeal can SSH into the nodes to execute commands on them
writing a set of policies

These interactions are available:

Minikube setup

It is possible to test a subset of Seal's functionality using a minikube setup.

To achieve that, please inspect the Makefile. You will need to override the ssh host, specify the correct username and use minikube's ssh keys.

If you'd like to test out the interactive mode, start with this:

seal \
  -vv \
  interactive \
    --no-cloud \
    --inventory-kubernetes \
    --ssh-allow-missing-host-keys \
    --remote-user docker \
    --ssh-path-to-private-key `minikube ssh-key` \
    --ssh-password `minikube ssh-password` \
    --override-ssh-host `minikube ip`

For label mode, try something like this:

seal \
  -vv \
  label \
    --no-cloud \
    --min-seconds-between-runs 3 \
    --max-seconds-between-runs 10 \
    --inventory-kubernetes \
    --ssh-allow-missing-host-keys \
    --remote-user docker \
    --ssh-path-to-private-key `minikube ssh-key` \
    --ssh-password `minikube ssh-password` \
    --override-ssh-host `minikube ip`

For autonomous mode, this should get you started:

seal \
  -vv \
  autonomous \
    --no-cloud \
    --policy-file ./examples/policy_kill_random_default.yml \
    --inventory-kubernetes \
    --prometheus-collector \
    --prometheus-host 0.0.0.0 \
    --prometheus-port 9999 \
    --ssh-allow-missing-host-keys \
    --remote-user docker \
    --ssh-path-to-private-key `minikube ssh-key` \
    --ssh-password `minikube ssh-password` \
    --override-ssh-host `minikube ip` \
    --host 0.0.0.0 \
    --port 30100

Getting started

PowerfulSeal is available to install through pip:

pip install powerfulseal
powerfulseal --help # or seal --help

To start the web interface, use flags --server --server-host [HOST] --server-port [PORT] when starting PowerfulSeal in autonomous mode and visit the web server at http://HOST:PORT/.

Both Python 2.7 and Python 3 are supported.

Inventory File

PowerfulSeal can use an ansible-style inventory file (in ini format)

[mygroup]
myhost01

[mygroup2]
myhost02

[some_hosts]
myhost01
myhost02

Cloud Provider Requirements

SSH

In all cases, the SSH Keys must be set up for SSH Client access of the nodes.

Azure

The credentials to connect to Azure may be specified in one of two ways:

Supply the full path to an Azure credentials file in the environment variable AZURE_AUTH_LOCATION.
This is the easiest method. The credentials file can be generated via az aks get-credentials -n <cluster name> -g <resouce group> -a -f <desitnation credentials file>
Supply the individual credentials in the environment variables: AZURE_SUBSCRIPTION_ID, AZURE_CLIENT_ID, AZURE_CLIENT_SECRET, AZURE_TENANT_ID

AWS

The credentials to connect to AWS are specified the same as for the AWS CLI

GCP

TBD

OpenStack

TBD

Bare Metal

TBD

Testing

PowerfulSeal uses tox to test with multiple versions on Python. The recommended setup is to install and locally activate the Python versions under tox.ini with pyenv.

Once the required Python versions are set up and can be discovered by tox (e.g., by having them discoverable in your PATH), you can run the tests by running tox.

For testing the web server and more details on testing, see TESTING.md.

Read about the PowerfulSeal

FAQ

Where can I learn more about Chaos Engineering ?

We found these two links to be a good start:

How is it different from Chaos Monkey ?

PowerfulSeal was inspired by Chaos Monkey, but it differs in a couple of important ways.

The Seal does:

speak Kubernetes
offer flexible, easy to write YAML scenarios
provide interactive mode with awesome tab-completion

The Seal doesn't:

need external dependencies (db, Spinnaker), apart from SSH, cloud and Kubernetes API access
need you to setup cron

Can I contribute to The Seal ?

We would love you to. In particular, it would be great to get help with:

get more cloud drivers (currently OpenStack and AWS)
get more awesome filters
~~get an amazing logo~~
make the PowerfulSeal more powerful

Check out our CONTRIBUTING.md file for more information about how to contribute.

Why a Seal ?

It might have been inspired by this comic.

Name		Name	Last commit message	Last commit date
Latest commit History 387 Commits
examples		examples
media		media
powerfulseal		powerfulseal
tests		tests
.gitignore		.gitignore
.travis.yml		.travis.yml
CONTRIBUTING.md		CONTRIBUTING.md
DCO.md		DCO.md
LABELS.md		LABELS.md
LICENSE		LICENSE
MANIFEST.in		MANIFEST.in
METRICS.md		METRICS.md
Makefile		Makefile
README.md		README.md
TESTING.md		TESTING.md
setup.py		setup.py
tox.ini		tox.ini

License

Ruteri/powerfulseal-gcp-integration

Folders and files

Latest commit

History

Repository files navigation