offline-search-index.7b62dee15fb25eece042b95aa4a36e38a377a4ca61f075a29e9904aaf6c6d7b9ee24b45633f2739bbd74cc3cb5cb15e9591103142b4c065031abc00204d7bef5.json

[{"body":"To install the operator follow the installation page for your orchestrator.\n Kubernetes Rancher OpenShift  ","excerpt":"To install the operator follow the installation page for your orchestrator.\n Kubernetes Rancher …","ref":"/docs/reference/cluster-operator/install/","title":"Install"},{"body":"Current Version The current StorageOS version is v2.3.2\nSections Our documentation is arranged into sections, accessible from the navigation bar on the left.\nIntroduction - Quickstart and support information.\nConcepts - Architectural and deep technical information.\nPrerequisites - We require certain prerequisites to be met for the product to function correctly. Please read these carefully and ensure they are implemented.\nPlatforms - Due to differences in the various orchestrators that StorageOS can run under, we list install guides and other platform specific operations here.\nOperations - Platform agnostic operations.\nUse Cases - A set of examples to get up and running with StorageOS quickly.\nReference - Information on our GUI, CLI, and other important information.\n","excerpt":"Current Version The current StorageOS version is v2.3.2\nSections Our documentation is arranged into …","ref":"/docs/introduction/","title":"Introduction"},{"body":" Make sure the prerequisites for StorageOS are satisfied before proceeding.\n  Any Kubernetes managed service such as EKS, AKS, GKE, DO or DockerEE platform can use the following Kubernetes guide to install StorageOS.\n  StorageOS v2 uses CSI only. The native driver has been deprecated.\n  Make sure to add a StorageOS licence after installing.\n   1.19 1.18 1.17 1.16 1.15 1.14 1.13    Install StorageOS on Kubernetes 1.19 The StorageOS Cluster Operator is a Kubernetes native application developed to deploy and configure StorageOS clusters, and assist with maintenance operations. We recommend its use for standard installations.\nThe operator is a Kubernetes controller that watches the StorageOSCluster CRD. Once the controller is ready, a StorageOS cluster definition can be created. The operator will deploy a StorageOS cluster based on the configuration specified in the cluster definition.\n Helm Note: If you want to use Helm to install StorageOS, follow the StorageOS Operator Helm Chart documentation.\nSteps to install StorageOS:  Install StorageOS Operator Create a Secret for default username and password Trigger bootstrap using a CustomResource Apply StorageOS licence  1. Install StorageOS operator Install the StorageOS Cluster Operator using the following yaml manifest.\nkubectl create -f https://github.com/storageos/cluster-operator/releases/download/v2.3.1/storageos-operator.yaml Verify the Cluster Operator Pod Status [root@master03]# kubectl -n storageos-operator get pod NAME READY STATUS RESTARTS AGE storageoscluster-operator-68678798ff-f28zw 1/1 Running 0 3m  The READY 1/1 indicates that storageoscluster resources can be created.\n 2. Create a Secret Before deploying a StorageOS cluster, create a Secret defining the StorageOS API Username and Password in base64 encoding. The API username and password are used to create the default StorageOS admin account which can be used with the StorageOS CLI and to login to the StorageOS GUI. The CSI credentials are used to register the CSI accounts, so Kubernetes and StorageOS communicate over an authenticated API.\napiVersion: v1 kind: Secret metadata: name: \u0026#34;storageos-api\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; labels: app: \u0026#34;storageos\u0026#34; type: \u0026#34;kubernetes.io/storageos\u0026#34; data: # echo -n \u0026#39;\u0026lt;secret\u0026gt;\u0026#39; | base64 apiUsername: c3RvcmFnZW9z apiPassword: c3RvcmFnZW9z # CSI Credentials csiProvisionUsername: c3RvcmFnZW9z csiProvisionPassword: c3RvcmFnZW9z csiControllerPublishUsername: c3RvcmFnZW9z csiControllerPublishPassword: c3RvcmFnZW9z csiNodePublishUsername: c3RvcmFnZW9z csiNodePublishPassword: c3RvcmFnZW9z csiControllerExpandUsername: c3RvcmFnZW9z csiControllerExpandPassword: c3RvcmFnZW9z This example contains a default password, for production installations, use a unique, strong password.\n You can define a base64 value by echo -n \u0026quot;mystring\u0026quot; | base64.\n  Make sure that the encoding of the credentials doesn\u0026rsquo;t have special characters such as \u0026lsquo;\\n\u0026rsquo;. The echo -n ensures that a trailing new line is not appended to the string.\n  If you wish to change the default accounts details post-install please see Managing Users\n 3 Trigger a StorageOS installation  upstream  eks  aks  gke   This is a Cluster Definition example.\napiVersion: \u0026#34;storageos.com/v1\u0026#34; kind: StorageOSCluster metadata: name: \u0026#34;example-storageos\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; spec: # StorageOS Pods are in kube-system by default secretRefName: \u0026#34;storageos-api\u0026#34; # Reference from the Secret created in the previous step secretRefNamespace: \u0026#34;storageos-operator\u0026#34; # Namespace of the Secret k8sDistro: \u0026#34;upstream\u0026#34; images: nodeContainer: \u0026#34;storageos/node:v2.3.2\u0026#34; # StorageOS version kvBackend: address: \u0026#39;storageos-etcd-client.storageos-etcd:2379\u0026#39; # Example address, change for your etcd endpoint # address: \u0026#39;10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379\u0026#39; # You can set ETCD server ips resources: requests: memory: \u0026#34;512Mi\u0026#34; cpu: 1 # nodeSelectorTerms: # - matchExpressions: # - key: \u0026#34;node-role.kubernetes.io/worker\u0026#34; # Compute node label will vary according to your installation # operator: In # values: # - \u0026#34;true\u0026#34; This is a Cluster Definition example.\napiVersion: \u0026#34;storageos.com/v1\u0026#34; kind: StorageOSCluster metadata: name: \u0026#34;example-storageos\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; spec: # StorageOS Pods are in kube-system by default secretRefName: \u0026#34;storageos-api\u0026#34; # Reference from the Secret created in the previous step secretRefNamespace: \u0026#34;storageos-operator\u0026#34; # Namespace of the Secret k8sDistro: \u0026#34;eks\u0026#34; images: nodeContainer: \u0026#34;storageos/node:v2.3.2\u0026#34; # StorageOS version kvBackend: address: \u0026#39;storageos-etcd-client.storageos-etcd:2379\u0026#39; # Example address, change for your etcd endpoint # address: \u0026#39;10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379\u0026#39; # You can set ETCD server ips resources: requests: memory: \u0026#34;512Mi\u0026#34; cpu: 1 # nodeSelectorTerms: # - matchExpressions: # - key: \u0026#34;node-role.kubernetes.io/worker\u0026#34; # Compute node label will vary according to your installation # operator: In # values: # - \u0026#34;true\u0026#34; This is a Cluster Definition example.\napiVersion: \u0026#34;storageos.com/v1\u0026#34; kind: StorageOSCluster metadata: name: \u0026#34;example-storageos\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; spec: # StorageOS Pods are in kube-system by default secretRefName: \u0026#34;storageos-api\u0026#34; # Reference from the Secret created in the previous step secretRefNamespace: \u0026#34;storageos-operator\u0026#34; # Namespace of the Secret k8sDistro: \u0026#34;aks\u0026#34; images: nodeContainer: \u0026#34;storageos/node:v2.3.2\u0026#34; # StorageOS version kvBackend: address: \u0026#39;storageos-etcd-client.storageos-etcd:2379\u0026#39; # Example address, change for your etcd endpoint # address: \u0026#39;10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379\u0026#39; # You can set ETCD server ips resources: requests: memory: \u0026#34;512Mi\u0026#34; cpu: 1 # nodeSelectorTerms: # - matchExpressions: # - key: \u0026#34;node-role.kubernetes.io/worker\u0026#34; # Compute node label will vary according to your installation # operator: In # values: # - \u0026#34;true\u0026#34; This is a Cluster Definition example.\napiVersion: \u0026#34;storageos.com/v1\u0026#34; kind: StorageOSCluster metadata: name: \u0026#34;example-storageos\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; spec: # StorageOS Pods are in kube-system by default secretRefName: \u0026#34;storageos-api\u0026#34; # Reference from the Secret created in the previous step secretRefNamespace: \u0026#34;storageos-operator\u0026#34; # Namespace of the Secret k8sDistro: \u0026#34;gke\u0026#34; images: nodeContainer: \u0026#34;storageos/node:v2.3.2\u0026#34; # StorageOS version kvBackend: address: \u0026#39;storageos-etcd-client.storageos-etcd:2379\u0026#39; # Example address, change for your etcd endpoint # address: \u0026#39;10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379\u0026#39; # You can set ETCD server ips resources: requests: memory: \u0026#34;512Mi\u0026#34; cpu: 1 # nodeSelectorTerms: # - matchExpressions: # - key: \u0026#34;node-role.kubernetes.io/worker\u0026#34; # Compute node label will vary according to your installation # operator: In # values: # - \u0026#34;true\u0026#34;   Additional spec parameters are available on the Cluster Operator configuration page.\n  You can find more examples such as deployments referencing a external etcd kv store for StorageOS in the Cluster Operator examples page.\n Verify StorageOS Installation [root@master03]# kubectl -n kube-system get pods -w NAME READY STATUS RESTARTS AGE storageos-csi-helper-5cf59b5b4-f5nwr 2/2 Running 0 3m storageos-daemonset-75f6c 3/3 Running 0 3m storageos-daemonset-czbqx 3/3 Running 0 3m storageos-daemonset-zv4tq 3/3 Running 0 3m storageos-scheduler-6d67b46f67-5c46j 1/1 Running 0 3m  The above command watches the Pods created by the Cluster Definition example. Note that pods typically take approximately 65 seconds to enter the Running Phase.\n 4. License cluster Newly installed StorageOS clusters must be licensed within 24 hours. Our developer license is free, and supports up to 5TiB of provisioned storage.\nTo obtain a license, follow the instructions on our licensing operations  page.\n Install StorageOS on Kubernetes 1.18 The StorageOS Cluster Operator is a Kubernetes native application developed to deploy and configure StorageOS clusters, and assist with maintenance operations. We recommend its use for standard installations.\nThe operator is a Kubernetes controller that watches the StorageOSCluster CRD. Once the controller is ready, a StorageOS cluster definition can be created. The operator will deploy a StorageOS cluster based on the configuration specified in the cluster definition.\n Helm Note: If you want to use Helm to install StorageOS, follow the StorageOS Operator Helm Chart documentation.\nSteps to install StorageOS:  Install StorageOS Operator Create a Secret for default username and password Trigger bootstrap using a CustomResource Apply StorageOS licence  1. Install StorageOS operator Install the StorageOS Cluster Operator using the following yaml manifest.\nkubectl create -f https://github.com/storageos/cluster-operator/releases/download/v2.3.1/storageos-operator.yaml Verify the Cluster Operator Pod Status [root@master03]# kubectl -n storageos-operator get pod NAME READY STATUS RESTARTS AGE storageoscluster-operator-68678798ff-f28zw 1/1 Running 0 3m  The READY 1/1 indicates that storageoscluster resources can be created.\n 2. Create a Secret Before deploying a StorageOS cluster, create a Secret defining the StorageOS API Username and Password in base64 encoding. The API username and password are used to create the default StorageOS admin account which can be used with the StorageOS CLI and to login to the StorageOS GUI. The CSI credentials are used to register the CSI accounts, so Kubernetes and StorageOS communicate over an authenticated API.\napiVersion: v1 kind: Secret metadata: name: \u0026#34;storageos-api\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; labels: app: \u0026#34;storageos\u0026#34; type: \u0026#34;kubernetes.io/storageos\u0026#34; data: # echo -n \u0026#39;\u0026lt;secret\u0026gt;\u0026#39; | base64 apiUsername: c3RvcmFnZW9z apiPassword: c3RvcmFnZW9z # CSI Credentials csiProvisionUsername: c3RvcmFnZW9z csiProvisionPassword: c3RvcmFnZW9z csiControllerPublishUsername: c3RvcmFnZW9z csiControllerPublishPassword: c3RvcmFnZW9z csiNodePublishUsername: c3RvcmFnZW9z csiNodePublishPassword: c3RvcmFnZW9z csiControllerExpandUsername: c3RvcmFnZW9z csiControllerExpandPassword: c3RvcmFnZW9z This example contains a default password, for production installations, use a unique, strong password.\n You can define a base64 value by echo -n \u0026quot;mystring\u0026quot; | base64.\n  Make sure that the encoding of the credentials doesn\u0026rsquo;t have special characters such as \u0026lsquo;\\n\u0026rsquo;. The echo -n ensures that a trailing new line is not appended to the string.\n  If you wish to change the default accounts details post-install please see Managing Users\n 3 Trigger a StorageOS installation  upstream  eks  aks  gke   This is a Cluster Definition example.\napiVersion: \u0026#34;storageos.com/v1\u0026#34; kind: StorageOSCluster metadata: name: \u0026#34;example-storageos\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; spec: # StorageOS Pods are in kube-system by default secretRefName: \u0026#34;storageos-api\u0026#34; # Reference from the Secret created in the previous step secretRefNamespace: \u0026#34;storageos-operator\u0026#34; # Namespace of the Secret k8sDistro: \u0026#34;upstream\u0026#34; images: nodeContainer: \u0026#34;storageos/node:v2.3.2\u0026#34; # StorageOS version kvBackend: address: \u0026#39;storageos-etcd-client.storageos-etcd:2379\u0026#39; # Example address, change for your etcd endpoint # address: \u0026#39;10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379\u0026#39; # You can set ETCD server ips resources: requests: memory: \u0026#34;512Mi\u0026#34; cpu: 1 # nodeSelectorTerms: # - matchExpressions: # - key: \u0026#34;node-role.kubernetes.io/worker\u0026#34; # Compute node label will vary according to your installation # operator: In # values: # - \u0026#34;true\u0026#34; This is a Cluster Definition example.\napiVersion: \u0026#34;storageos.com/v1\u0026#34; kind: StorageOSCluster metadata: name: \u0026#34;example-storageos\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; spec: # StorageOS Pods are in kube-system by default secretRefName: \u0026#34;storageos-api\u0026#34; # Reference from the Secret created in the previous step secretRefNamespace: \u0026#34;storageos-operator\u0026#34; # Namespace of the Secret k8sDistro: \u0026#34;eks\u0026#34; images: nodeContainer: \u0026#34;storageos/node:v2.3.2\u0026#34; # StorageOS version kvBackend: address: \u0026#39;storageos-etcd-client.storageos-etcd:2379\u0026#39; # Example address, change for your etcd endpoint # address: \u0026#39;10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379\u0026#39; # You can set ETCD server ips resources: requests: memory: \u0026#34;512Mi\u0026#34; cpu: 1 # nodeSelectorTerms: # - matchExpressions: # - key: \u0026#34;node-role.kubernetes.io/worker\u0026#34; # Compute node label will vary according to your installation # operator: In # values: # - \u0026#34;true\u0026#34; This is a Cluster Definition example.\napiVersion: \u0026#34;storageos.com/v1\u0026#34; kind: StorageOSCluster metadata: name: \u0026#34;example-storageos\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; spec: # StorageOS Pods are in kube-system by default secretRefName: \u0026#34;storageos-api\u0026#34; # Reference from the Secret created in the previous step secretRefNamespace: \u0026#34;storageos-operator\u0026#34; # Namespace of the Secret k8sDistro: \u0026#34;aks\u0026#34; images: nodeContainer: \u0026#34;storageos/node:v2.3.2\u0026#34; # StorageOS version kvBackend: address: \u0026#39;storageos-etcd-client.storageos-etcd:2379\u0026#39; # Example address, change for your etcd endpoint # address: \u0026#39;10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379\u0026#39; # You can set ETCD server ips resources: requests: memory: \u0026#34;512Mi\u0026#34; cpu: 1 # nodeSelectorTerms: # - matchExpressions: # - key: \u0026#34;node-role.kubernetes.io/worker\u0026#34; # Compute node label will vary according to your installation # operator: In # values: # - \u0026#34;true\u0026#34; This is a Cluster Definition example.\napiVersion: \u0026#34;storageos.com/v1\u0026#34; kind: StorageOSCluster metadata: name: \u0026#34;example-storageos\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; spec: # StorageOS Pods are in kube-system by default secretRefName: \u0026#34;storageos-api\u0026#34; # Reference from the Secret created in the previous step secretRefNamespace: \u0026#34;storageos-operator\u0026#34; # Namespace of the Secret k8sDistro: \u0026#34;gke\u0026#34; images: nodeContainer: \u0026#34;storageos/node:v2.3.2\u0026#34; # StorageOS version kvBackend: address: \u0026#39;storageos-etcd-client.storageos-etcd:2379\u0026#39; # Example address, change for your etcd endpoint # address: \u0026#39;10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379\u0026#39; # You can set ETCD server ips resources: requests: memory: \u0026#34;512Mi\u0026#34; cpu: 1 # nodeSelectorTerms: # - matchExpressions: # - key: \u0026#34;node-role.kubernetes.io/worker\u0026#34; # Compute node label will vary according to your installation # operator: In # values: # - \u0026#34;true\u0026#34;   Additional spec parameters are available on the Cluster Operator configuration page.\n  You can find more examples such as deployments referencing a external etcd kv store for StorageOS in the Cluster Operator examples page.\n Verify StorageOS Installation [root@master03]# kubectl -n kube-system get pods -w NAME READY STATUS RESTARTS AGE storageos-csi-helper-5cf59b5b4-f5nwr 2/2 Running 0 3m storageos-daemonset-75f6c 3/3 Running 0 3m storageos-daemonset-czbqx 3/3 Running 0 3m storageos-daemonset-zv4tq 3/3 Running 0 3m storageos-scheduler-6d67b46f67-5c46j 1/1 Running 0 3m  The above command watches the Pods created by the Cluster Definition example. Note that pods typically take approximately 65 seconds to enter the Running Phase.\n 4. License cluster Newly installed StorageOS clusters must be licensed within 24 hours. Our developer license is free, and supports up to 5TiB of provisioned storage.\nTo obtain a license, follow the instructions on our licensing operations  page.\n Install StorageOS on Kubernetes 1.17 The StorageOS Cluster Operator is a Kubernetes native application developed to deploy and configure StorageOS clusters, and assist with maintenance operations. We recommend its use for standard installations.\nThe operator is a Kubernetes controller that watches the StorageOSCluster CRD. Once the controller is ready, a StorageOS cluster definition can be created. The operator will deploy a StorageOS cluster based on the configuration specified in the cluster definition.\n Helm Note: If you want to use Helm to install StorageOS, follow the StorageOS Operator Helm Chart documentation.\nSteps to install StorageOS:  Install StorageOS Operator Create a Secret for default username and password Trigger bootstrap using a CustomResource Apply StorageOS licence  1. Install StorageOS operator Install the StorageOS Cluster Operator using the following yaml manifest.\nkubectl create -f https://github.com/storageos/cluster-operator/releases/download/v2.3.1/storageos-operator.yaml Verify the Cluster Operator Pod Status [root@master03]# kubectl -n storageos-operator get pod NAME READY STATUS RESTARTS AGE storageoscluster-operator-68678798ff-f28zw 1/1 Running 0 3m  The READY 1/1 indicates that storageoscluster resources can be created.\n 2. Create a Secret Before deploying a StorageOS cluster, create a Secret defining the StorageOS API Username and Password in base64 encoding. The API username and password are used to create the default StorageOS admin account which can be used with the StorageOS CLI and to login to the StorageOS GUI. The CSI credentials are used to register the CSI accounts, so Kubernetes and StorageOS communicate over an authenticated API.\napiVersion: v1 kind: Secret metadata: name: \u0026#34;storageos-api\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; labels: app: \u0026#34;storageos\u0026#34; type: \u0026#34;kubernetes.io/storageos\u0026#34; data: # echo -n \u0026#39;\u0026lt;secret\u0026gt;\u0026#39; | base64 apiUsername: c3RvcmFnZW9z apiPassword: c3RvcmFnZW9z # CSI Credentials csiProvisionUsername: c3RvcmFnZW9z csiProvisionPassword: c3RvcmFnZW9z csiControllerPublishUsername: c3RvcmFnZW9z csiControllerPublishPassword: c3RvcmFnZW9z csiNodePublishUsername: c3RvcmFnZW9z csiNodePublishPassword: c3RvcmFnZW9z csiControllerExpandUsername: c3RvcmFnZW9z csiControllerExpandPassword: c3RvcmFnZW9z This example contains a default password, for production installations, use a unique, strong password.\n You can define a base64 value by echo -n \u0026quot;mystring\u0026quot; | base64.\n  Make sure that the encoding of the credentials doesn\u0026rsquo;t have special characters such as \u0026lsquo;\\n\u0026rsquo;. The echo -n ensures that a trailing new line is not appended to the string.\n  If you wish to change the default accounts details post-install please see Managing Users\n 3 Trigger a StorageOS installation  upstream  eks  aks  gke   This is a Cluster Definition example.\napiVersion: \u0026#34;storageos.com/v1\u0026#34; kind: StorageOSCluster metadata: name: \u0026#34;example-storageos\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; spec: # StorageOS Pods are in kube-system by default secretRefName: \u0026#34;storageos-api\u0026#34; # Reference from the Secret created in the previous step secretRefNamespace: \u0026#34;storageos-operator\u0026#34; # Namespace of the Secret k8sDistro: \u0026#34;upstream\u0026#34; images: nodeContainer: \u0026#34;storageos/node:v2.3.2\u0026#34; # StorageOS version kvBackend: address: \u0026#39;storageos-etcd-client.storageos-etcd:2379\u0026#39; # Example address, change for your etcd endpoint # address: \u0026#39;10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379\u0026#39; # You can set ETCD server ips resources: requests: memory: \u0026#34;512Mi\u0026#34; cpu: 1 # nodeSelectorTerms: # - matchExpressions: # - key: \u0026#34;node-role.kubernetes.io/worker\u0026#34; # Compute node label will vary according to your installation # operator: In # values: # - \u0026#34;true\u0026#34; This is a Cluster Definition example.\napiVersion: \u0026#34;storageos.com/v1\u0026#34; kind: StorageOSCluster metadata: name: \u0026#34;example-storageos\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; spec: # StorageOS Pods are in kube-system by default secretRefName: \u0026#34;storageos-api\u0026#34; # Reference from the Secret created in the previous step secretRefNamespace: \u0026#34;storageos-operator\u0026#34; # Namespace of the Secret k8sDistro: \u0026#34;eks\u0026#34; images: nodeContainer: \u0026#34;storageos/node:v2.3.2\u0026#34; # StorageOS version kvBackend: address: \u0026#39;storageos-etcd-client.storageos-etcd:2379\u0026#39; # Example address, change for your etcd endpoint # address: \u0026#39;10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379\u0026#39; # You can set ETCD server ips resources: requests: memory: \u0026#34;512Mi\u0026#34; cpu: 1 # nodeSelectorTerms: # - matchExpressions: # - key: \u0026#34;node-role.kubernetes.io/worker\u0026#34; # Compute node label will vary according to your installation # operator: In # values: # - \u0026#34;true\u0026#34; This is a Cluster Definition example.\napiVersion: \u0026#34;storageos.com/v1\u0026#34; kind: StorageOSCluster metadata: name: \u0026#34;example-storageos\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; spec: # StorageOS Pods are in kube-system by default secretRefName: \u0026#34;storageos-api\u0026#34; # Reference from the Secret created in the previous step secretRefNamespace: \u0026#34;storageos-operator\u0026#34; # Namespace of the Secret k8sDistro: \u0026#34;aks\u0026#34; images: nodeContainer: \u0026#34;storageos/node:v2.3.2\u0026#34; # StorageOS version kvBackend: address: \u0026#39;storageos-etcd-client.storageos-etcd:2379\u0026#39; # Example address, change for your etcd endpoint # address: \u0026#39;10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379\u0026#39; # You can set ETCD server ips resources: requests: memory: \u0026#34;512Mi\u0026#34; cpu: 1 # nodeSelectorTerms: # - matchExpressions: # - key: \u0026#34;node-role.kubernetes.io/worker\u0026#34; # Compute node label will vary according to your installation # operator: In # values: # - \u0026#34;true\u0026#34; This is a Cluster Definition example.\napiVersion: \u0026#34;storageos.com/v1\u0026#34; kind: StorageOSCluster metadata: name: \u0026#34;example-storageos\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; spec: # StorageOS Pods are in kube-system by default secretRefName: \u0026#34;storageos-api\u0026#34; # Reference from the Secret created in the previous step secretRefNamespace: \u0026#34;storageos-operator\u0026#34; # Namespace of the Secret k8sDistro: \u0026#34;gke\u0026#34; images: nodeContainer: \u0026#34;storageos/node:v2.3.2\u0026#34; # StorageOS version kvBackend: address: \u0026#39;storageos-etcd-client.storageos-etcd:2379\u0026#39; # Example address, change for your etcd endpoint # address: \u0026#39;10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379\u0026#39; # You can set ETCD server ips resources: requests: memory: \u0026#34;512Mi\u0026#34; cpu: 1 # nodeSelectorTerms: # - matchExpressions: # - key: \u0026#34;node-role.kubernetes.io/worker\u0026#34; # Compute node label will vary according to your installation # operator: In # values: # - \u0026#34;true\u0026#34;   Additional spec parameters are available on the Cluster Operator configuration page.\n  You can find more examples such as deployments referencing a external etcd kv store for StorageOS in the Cluster Operator examples page.\n Verify StorageOS Installation [root@master03]# kubectl -n kube-system get pods -w NAME READY STATUS RESTARTS AGE storageos-csi-helper-5cf59b5b4-f5nwr 2/2 Running 0 3m storageos-daemonset-75f6c 3/3 Running 0 3m storageos-daemonset-czbqx 3/3 Running 0 3m storageos-daemonset-zv4tq 3/3 Running 0 3m storageos-scheduler-6d67b46f67-5c46j 1/1 Running 0 3m  The above command watches the Pods created by the Cluster Definition example. Note that pods typically take approximately 65 seconds to enter the Running Phase.\n 4. License cluster Newly installed StorageOS clusters must be licensed within 24 hours. Our developer license is free, and supports up to 5TiB of provisioned storage.\nTo obtain a license, follow the instructions on our licensing operations  page.\n Install StorageOS on Kubernetes 1.16 The StorageOS Cluster Operator is a Kubernetes native application developed to deploy and configure StorageOS clusters, and assist with maintenance operations. We recommend its use for standard installations.\nThe operator is a Kubernetes controller that watches the StorageOSCluster CRD. Once the controller is ready, a StorageOS cluster definition can be created. The operator will deploy a StorageOS cluster based on the configuration specified in the cluster definition.\n Helm Note: If you want to use Helm to install StorageOS, follow the StorageOS Operator Helm Chart documentation.\nSteps to install StorageOS:  Install StorageOS Operator Create a Secret for default username and password Trigger bootstrap using a CustomResource Apply StorageOS licence  1. Install StorageOS operator Install the StorageOS Cluster Operator using the following yaml manifest.\nkubectl create -f https://github.com/storageos/cluster-operator/releases/download/v2.3.1/storageos-operator.yaml Verify the Cluster Operator Pod Status [root@master03]# kubectl -n storageos-operator get pod NAME READY STATUS RESTARTS AGE storageoscluster-operator-68678798ff-f28zw 1/1 Running 0 3m  The READY 1/1 indicates that storageoscluster resources can be created.\n 2. Create a Secret Before deploying a StorageOS cluster, create a Secret defining the StorageOS API Username and Password in base64 encoding. The API username and password are used to create the default StorageOS admin account which can be used with the StorageOS CLI and to login to the StorageOS GUI. The CSI credentials are used to register the CSI accounts, so Kubernetes and StorageOS communicate over an authenticated API.\napiVersion: v1 kind: Secret metadata: name: \u0026#34;storageos-api\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; labels: app: \u0026#34;storageos\u0026#34; type: \u0026#34;kubernetes.io/storageos\u0026#34; data: # echo -n \u0026#39;\u0026lt;secret\u0026gt;\u0026#39; | base64 apiUsername: c3RvcmFnZW9z apiPassword: c3RvcmFnZW9z # CSI Credentials csiProvisionUsername: c3RvcmFnZW9z csiProvisionPassword: c3RvcmFnZW9z csiControllerPublishUsername: c3RvcmFnZW9z csiControllerPublishPassword: c3RvcmFnZW9z csiNodePublishUsername: c3RvcmFnZW9z csiNodePublishPassword: c3RvcmFnZW9z csiControllerExpandUsername: c3RvcmFnZW9z csiControllerExpandPassword: c3RvcmFnZW9z This example contains a default password, for production installations, use a unique, strong password.\n You can define a base64 value by echo -n \u0026quot;mystring\u0026quot; | base64.\n  Make sure that the encoding of the credentials doesn\u0026rsquo;t have special characters such as \u0026lsquo;\\n\u0026rsquo;. The echo -n ensures that a trailing new line is not appended to the string.\n  If you wish to change the default accounts details post-install please see Managing Users\n 3 Trigger a StorageOS installation  upstream  eks  aks  gke   This is a Cluster Definition example.\napiVersion: \u0026#34;storageos.com/v1\u0026#34; kind: StorageOSCluster metadata: name: \u0026#34;example-storageos\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; spec: # StorageOS Pods are in kube-system by default secretRefName: \u0026#34;storageos-api\u0026#34; # Reference from the Secret created in the previous step secretRefNamespace: \u0026#34;storageos-operator\u0026#34; # Namespace of the Secret k8sDistro: \u0026#34;upstream\u0026#34; images: nodeContainer: \u0026#34;storageos/node:v2.3.2\u0026#34; # StorageOS version kvBackend: address: \u0026#39;storageos-etcd-client.storageos-etcd:2379\u0026#39; # Example address, change for your etcd endpoint # address: \u0026#39;10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379\u0026#39; # You can set ETCD server ips resources: requests: memory: \u0026#34;512Mi\u0026#34; cpu: 1 # nodeSelectorTerms: # - matchExpressions: # - key: \u0026#34;node-role.kubernetes.io/worker\u0026#34; # Compute node label will vary according to your installation # operator: In # values: # - \u0026#34;true\u0026#34; This is a Cluster Definition example.\napiVersion: \u0026#34;storageos.com/v1\u0026#34; kind: StorageOSCluster metadata: name: \u0026#34;example-storageos\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; spec: # StorageOS Pods are in kube-system by default secretRefName: \u0026#34;storageos-api\u0026#34; # Reference from the Secret created in the previous step secretRefNamespace: \u0026#34;storageos-operator\u0026#34; # Namespace of the Secret k8sDistro: \u0026#34;eks\u0026#34; images: nodeContainer: \u0026#34;storageos/node:v2.3.2\u0026#34; # StorageOS version kvBackend: address: \u0026#39;storageos-etcd-client.storageos-etcd:2379\u0026#39; # Example address, change for your etcd endpoint # address: \u0026#39;10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379\u0026#39; # You can set ETCD server ips resources: requests: memory: \u0026#34;512Mi\u0026#34; cpu: 1 # nodeSelectorTerms: # - matchExpressions: # - key: \u0026#34;node-role.kubernetes.io/worker\u0026#34; # Compute node label will vary according to your installation # operator: In # values: # - \u0026#34;true\u0026#34; This is a Cluster Definition example.\napiVersion: \u0026#34;storageos.com/v1\u0026#34; kind: StorageOSCluster metadata: name: \u0026#34;example-storageos\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; spec: # StorageOS Pods are in kube-system by default secretRefName: \u0026#34;storageos-api\u0026#34; # Reference from the Secret created in the previous step secretRefNamespace: \u0026#34;storageos-operator\u0026#34; # Namespace of the Secret k8sDistro: \u0026#34;aks\u0026#34; images: nodeContainer: \u0026#34;storageos/node:v2.3.2\u0026#34; # StorageOS version kvBackend: address: \u0026#39;storageos-etcd-client.storageos-etcd:2379\u0026#39; # Example address, change for your etcd endpoint # address: \u0026#39;10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379\u0026#39; # You can set ETCD server ips resources: requests: memory: \u0026#34;512Mi\u0026#34; cpu: 1 # nodeSelectorTerms: # - matchExpressions: # - key: \u0026#34;node-role.kubernetes.io/worker\u0026#34; # Compute node label will vary according to your installation # operator: In # values: # - \u0026#34;true\u0026#34; This is a Cluster Definition example.\napiVersion: \u0026#34;storageos.com/v1\u0026#34; kind: StorageOSCluster metadata: name: \u0026#34;example-storageos\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; spec: # StorageOS Pods are in kube-system by default secretRefName: \u0026#34;storageos-api\u0026#34; # Reference from the Secret created in the previous step secretRefNamespace: \u0026#34;storageos-operator\u0026#34; # Namespace of the Secret k8sDistro: \u0026#34;gke\u0026#34; images: nodeContainer: \u0026#34;storageos/node:v2.3.2\u0026#34; # StorageOS version kvBackend: address: \u0026#39;storageos-etcd-client.storageos-etcd:2379\u0026#39; # Example address, change for your etcd endpoint # address: \u0026#39;10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379\u0026#39; # You can set ETCD server ips resources: requests: memory: \u0026#34;512Mi\u0026#34; cpu: 1 # nodeSelectorTerms: # - matchExpressions: # - key: \u0026#34;node-role.kubernetes.io/worker\u0026#34; # Compute node label will vary according to your installation # operator: In # values: # - \u0026#34;true\u0026#34;   Additional spec parameters are available on the Cluster Operator configuration page.\n  You can find more examples such as deployments referencing a external etcd kv store for StorageOS in the Cluster Operator examples page.\n Verify StorageOS Installation [root@master03]# kubectl -n kube-system get pods -w NAME READY STATUS RESTARTS AGE storageos-csi-helper-5cf59b5b4-f5nwr 2/2 Running 0 3m storageos-daemonset-75f6c 3/3 Running 0 3m storageos-daemonset-czbqx 3/3 Running 0 3m storageos-daemonset-zv4tq 3/3 Running 0 3m storageos-scheduler-6d67b46f67-5c46j 1/1 Running 0 3m  The above command watches the Pods created by the Cluster Definition example. Note that pods typically take approximately 65 seconds to enter the Running Phase.\n 4. License cluster Newly installed StorageOS clusters must be licensed within 24 hours. Our developer license is free, and supports up to 5TiB of provisioned storage.\nTo obtain a license, follow the instructions on our licensing operations  page.\n Install StorageOS on Kubernetes 1.15 The StorageOS Cluster Operator is a Kubernetes native application developed to deploy and configure StorageOS clusters, and assist with maintenance operations. We recommend its use for standard installations.\nThe operator is a Kubernetes controller that watches the StorageOSCluster CRD. Once the controller is ready, a StorageOS cluster definition can be created. The operator will deploy a StorageOS cluster based on the configuration specified in the cluster definition.\n Helm Note: If you want to use Helm to install StorageOS, follow the StorageOS Operator Helm Chart documentation.\nSteps to install StorageOS:  Install StorageOS Operator Create a Secret for default username and password Trigger bootstrap using a CustomResource Apply StorageOS licence  1. Install StorageOS operator Install the StorageOS Cluster Operator using the following yaml manifest.\nkubectl create -f https://github.com/storageos/cluster-operator/releases/download/v2.3.1/storageos-operator.yaml Verify the Cluster Operator Pod Status [root@master03]# kubectl -n storageos-operator get pod NAME READY STATUS RESTARTS AGE storageoscluster-operator-68678798ff-f28zw 1/1 Running 0 3m  The READY 1/1 indicates that storageoscluster resources can be created.\n 2. Create a Secret Before deploying a StorageOS cluster, create a Secret defining the StorageOS API Username and Password in base64 encoding. The API username and password are used to create the default StorageOS admin account which can be used with the StorageOS CLI and to login to the StorageOS GUI. The CSI credentials are used to register the CSI accounts, so Kubernetes and StorageOS communicate over an authenticated API.\napiVersion: v1 kind: Secret metadata: name: \u0026#34;storageos-api\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; labels: app: \u0026#34;storageos\u0026#34; type: \u0026#34;kubernetes.io/storageos\u0026#34; data: # echo -n \u0026#39;\u0026lt;secret\u0026gt;\u0026#39; | base64 apiUsername: c3RvcmFnZW9z apiPassword: c3RvcmFnZW9z # CSI Credentials csiProvisionUsername: c3RvcmFnZW9z csiProvisionPassword: c3RvcmFnZW9z csiControllerPublishUsername: c3RvcmFnZW9z csiControllerPublishPassword: c3RvcmFnZW9z csiNodePublishUsername: c3RvcmFnZW9z csiNodePublishPassword: c3RvcmFnZW9z csiControllerExpandUsername: c3RvcmFnZW9z csiControllerExpandPassword: c3RvcmFnZW9z This example contains a default password, for production installations, use a unique, strong password.\n You can define a base64 value by echo -n \u0026quot;mystring\u0026quot; | base64.\n  Make sure that the encoding of the credentials doesn\u0026rsquo;t have special characters such as \u0026lsquo;\\n\u0026rsquo;. The echo -n ensures that a trailing new line is not appended to the string.\n  If you wish to change the default accounts details post-install please see Managing Users\n 3 Trigger a StorageOS installation  upstream  eks  aks  gke   This is a Cluster Definition example.\napiVersion: \u0026#34;storageos.com/v1\u0026#34; kind: StorageOSCluster metadata: name: \u0026#34;example-storageos\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; spec: # StorageOS Pods are in kube-system by default secretRefName: \u0026#34;storageos-api\u0026#34; # Reference from the Secret created in the previous step secretRefNamespace: \u0026#34;storageos-operator\u0026#34; # Namespace of the Secret k8sDistro: \u0026#34;upstream\u0026#34; images: nodeContainer: \u0026#34;storageos/node:v2.3.2\u0026#34; # StorageOS version kvBackend: address: \u0026#39;storageos-etcd-client.storageos-etcd:2379\u0026#39; # Example address, change for your etcd endpoint # address: \u0026#39;10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379\u0026#39; # You can set ETCD server ips resources: requests: memory: \u0026#34;512Mi\u0026#34; cpu: 1 # nodeSelectorTerms: # - matchExpressions: # - key: \u0026#34;node-role.kubernetes.io/worker\u0026#34; # Compute node label will vary according to your installation # operator: In # values: # - \u0026#34;true\u0026#34; This is a Cluster Definition example.\napiVersion: \u0026#34;storageos.com/v1\u0026#34; kind: StorageOSCluster metadata: name: \u0026#34;example-storageos\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; spec: # StorageOS Pods are in kube-system by default secretRefName: \u0026#34;storageos-api\u0026#34; # Reference from the Secret created in the previous step secretRefNamespace: \u0026#34;storageos-operator\u0026#34; # Namespace of the Secret k8sDistro: \u0026#34;eks\u0026#34; images: nodeContainer: \u0026#34;storageos/node:v2.3.2\u0026#34; # StorageOS version kvBackend: address: \u0026#39;storageos-etcd-client.storageos-etcd:2379\u0026#39; # Example address, change for your etcd endpoint # address: \u0026#39;10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379\u0026#39; # You can set ETCD server ips resources: requests: memory: \u0026#34;512Mi\u0026#34; cpu: 1 # nodeSelectorTerms: # - matchExpressions: # - key: \u0026#34;node-role.kubernetes.io/worker\u0026#34; # Compute node label will vary according to your installation # operator: In # values: # - \u0026#34;true\u0026#34; This is a Cluster Definition example.\napiVersion: \u0026#34;storageos.com/v1\u0026#34; kind: StorageOSCluster metadata: name: \u0026#34;example-storageos\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; spec: # StorageOS Pods are in kube-system by default secretRefName: \u0026#34;storageos-api\u0026#34; # Reference from the Secret created in the previous step secretRefNamespace: \u0026#34;storageos-operator\u0026#34; # Namespace of the Secret k8sDistro: \u0026#34;aks\u0026#34; images: nodeContainer: \u0026#34;storageos/node:v2.3.2\u0026#34; # StorageOS version kvBackend: address: \u0026#39;storageos-etcd-client.storageos-etcd:2379\u0026#39; # Example address, change for your etcd endpoint # address: \u0026#39;10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379\u0026#39; # You can set ETCD server ips resources: requests: memory: \u0026#34;512Mi\u0026#34; cpu: 1 # nodeSelectorTerms: # - matchExpressions: # - key: \u0026#34;node-role.kubernetes.io/worker\u0026#34; # Compute node label will vary according to your installation # operator: In # values: # - \u0026#34;true\u0026#34; This is a Cluster Definition example.\napiVersion: \u0026#34;storageos.com/v1\u0026#34; kind: StorageOSCluster metadata: name: \u0026#34;example-storageos\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; spec: # StorageOS Pods are in kube-system by default secretRefName: \u0026#34;storageos-api\u0026#34; # Reference from the Secret created in the previous step secretRefNamespace: \u0026#34;storageos-operator\u0026#34; # Namespace of the Secret k8sDistro: \u0026#34;gke\u0026#34; images: nodeContainer: \u0026#34;storageos/node:v2.3.2\u0026#34; # StorageOS version kvBackend: address: \u0026#39;storageos-etcd-client.storageos-etcd:2379\u0026#39; # Example address, change for your etcd endpoint # address: \u0026#39;10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379\u0026#39; # You can set ETCD server ips resources: requests: memory: \u0026#34;512Mi\u0026#34; cpu: 1 # nodeSelectorTerms: # - matchExpressions: # - key: \u0026#34;node-role.kubernetes.io/worker\u0026#34; # Compute node label will vary according to your installation # operator: In # values: # - \u0026#34;true\u0026#34;   Additional spec parameters are available on the Cluster Operator configuration page.\n  You can find more examples such as deployments referencing a external etcd kv store for StorageOS in the Cluster Operator examples page.\n Verify StorageOS Installation [root@master03]# kubectl -n kube-system get pods -w NAME READY STATUS RESTARTS AGE storageos-csi-helper-5cf59b5b4-f5nwr 2/2 Running 0 3m storageos-daemonset-75f6c 3/3 Running 0 3m storageos-daemonset-czbqx 3/3 Running 0 3m storageos-daemonset-zv4tq 3/3 Running 0 3m storageos-scheduler-6d67b46f67-5c46j 1/1 Running 0 3m  The above command watches the Pods created by the Cluster Definition example. Note that pods typically take approximately 65 seconds to enter the Running Phase.\n 4. License cluster Newly installed StorageOS clusters must be licensed within 24 hours. Our developer license is free, and supports up to 5TiB of provisioned storage.\nTo obtain a license, follow the instructions on our licensing operations  page.\n Install StorageOS on Kubernetes 1.14 The StorageOS Cluster Operator is a Kubernetes native application developed to deploy and configure StorageOS clusters, and assist with maintenance operations. We recommend its use for standard installations.\nThe operator is a Kubernetes controller that watches the StorageOSCluster CRD. Once the controller is ready, a StorageOS cluster definition can be created. The operator will deploy a StorageOS cluster based on the configuration specified in the cluster definition.\n Helm Note: If you want to use Helm to install StorageOS, follow the StorageOS Operator Helm Chart documentation.\nSteps to install StorageOS:  Install StorageOS Operator Create a Secret for default username and password Trigger bootstrap using a CustomResource Apply StorageOS licence  1. Install StorageOS operator Install the StorageOS Cluster Operator using the following yaml manifest.\nkubectl create -f https://github.com/storageos/cluster-operator/releases/download/v2.3.1/storageos-operator.yaml Verify the Cluster Operator Pod Status [root@master03]# kubectl -n storageos-operator get pod NAME READY STATUS RESTARTS AGE storageoscluster-operator-68678798ff-f28zw 1/1 Running 0 3m  The READY 1/1 indicates that storageoscluster resources can be created.\n 2. Create a Secret Before deploying a StorageOS cluster, create a Secret defining the StorageOS API Username and Password in base64 encoding. The API username and password are used to create the default StorageOS admin account which can be used with the StorageOS CLI and to login to the StorageOS GUI. The CSI credentials are used to register the CSI accounts, so Kubernetes and StorageOS communicate over an authenticated API.\napiVersion: v1 kind: Secret metadata: name: \u0026#34;storageos-api\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; labels: app: \u0026#34;storageos\u0026#34; type: \u0026#34;kubernetes.io/storageos\u0026#34; data: # echo -n \u0026#39;\u0026lt;secret\u0026gt;\u0026#39; | base64 apiUsername: c3RvcmFnZW9z apiPassword: c3RvcmFnZW9z # CSI Credentials csiProvisionUsername: c3RvcmFnZW9z csiProvisionPassword: c3RvcmFnZW9z csiControllerPublishUsername: c3RvcmFnZW9z csiControllerPublishPassword: c3RvcmFnZW9z csiNodePublishUsername: c3RvcmFnZW9z csiNodePublishPassword: c3RvcmFnZW9z csiControllerExpandUsername: c3RvcmFnZW9z csiControllerExpandPassword: c3RvcmFnZW9z This example contains a default password, for production installations, use a unique, strong password.\n You can define a base64 value by echo -n \u0026quot;mystring\u0026quot; | base64.\n  Make sure that the encoding of the credentials doesn\u0026rsquo;t have special characters such as \u0026lsquo;\\n\u0026rsquo;. The echo -n ensures that a trailing new line is not appended to the string.\n  If you wish to change the default accounts details post-install please see Managing Users\n 3 Trigger a StorageOS installation  upstream  eks  aks  gke   This is a Cluster Definition example.\napiVersion: \u0026#34;storageos.com/v1\u0026#34; kind: StorageOSCluster metadata: name: \u0026#34;example-storageos\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; spec: # StorageOS Pods are in kube-system by default secretRefName: \u0026#34;storageos-api\u0026#34; # Reference from the Secret created in the previous step secretRefNamespace: \u0026#34;storageos-operator\u0026#34; # Namespace of the Secret k8sDistro: \u0026#34;upstream\u0026#34; images: nodeContainer: \u0026#34;storageos/node:v2.3.2\u0026#34; # StorageOS version kvBackend: address: \u0026#39;storageos-etcd-client.storageos-etcd:2379\u0026#39; # Example address, change for your etcd endpoint # address: \u0026#39;10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379\u0026#39; # You can set ETCD server ips resources: requests: memory: \u0026#34;512Mi\u0026#34; cpu: 1 # nodeSelectorTerms: # - matchExpressions: # - key: \u0026#34;node-role.kubernetes.io/worker\u0026#34; # Compute node label will vary according to your installation # operator: In # values: # - \u0026#34;true\u0026#34; This is a Cluster Definition example.\napiVersion: \u0026#34;storageos.com/v1\u0026#34; kind: StorageOSCluster metadata: name: \u0026#34;example-storageos\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; spec: # StorageOS Pods are in kube-system by default secretRefName: \u0026#34;storageos-api\u0026#34; # Reference from the Secret created in the previous step secretRefNamespace: \u0026#34;storageos-operator\u0026#34; # Namespace of the Secret k8sDistro: \u0026#34;eks\u0026#34; images: nodeContainer: \u0026#34;storageos/node:v2.3.2\u0026#34; # StorageOS version kvBackend: address: \u0026#39;storageos-etcd-client.storageos-etcd:2379\u0026#39; # Example address, change for your etcd endpoint # address: \u0026#39;10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379\u0026#39; # You can set ETCD server ips resources: requests: memory: \u0026#34;512Mi\u0026#34; cpu: 1 # nodeSelectorTerms: # - matchExpressions: # - key: \u0026#34;node-role.kubernetes.io/worker\u0026#34; # Compute node label will vary according to your installation # operator: In # values: # - \u0026#34;true\u0026#34; This is a Cluster Definition example.\napiVersion: \u0026#34;storageos.com/v1\u0026#34; kind: StorageOSCluster metadata: name: \u0026#34;example-storageos\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; spec: # StorageOS Pods are in kube-system by default secretRefName: \u0026#34;storageos-api\u0026#34; # Reference from the Secret created in the previous step secretRefNamespace: \u0026#34;storageos-operator\u0026#34; # Namespace of the Secret k8sDistro: \u0026#34;aks\u0026#34; images: nodeContainer: \u0026#34;storageos/node:v2.3.2\u0026#34; # StorageOS version kvBackend: address: \u0026#39;storageos-etcd-client.storageos-etcd:2379\u0026#39; # Example address, change for your etcd endpoint # address: \u0026#39;10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379\u0026#39; # You can set ETCD server ips resources: requests: memory: \u0026#34;512Mi\u0026#34; cpu: 1 # nodeSelectorTerms: # - matchExpressions: # - key: \u0026#34;node-role.kubernetes.io/worker\u0026#34; # Compute node label will vary according to your installation # operator: In # values: # - \u0026#34;true\u0026#34; This is a Cluster Definition example.\napiVersion: \u0026#34;storageos.com/v1\u0026#34; kind: StorageOSCluster metadata: name: \u0026#34;example-storageos\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; spec: # StorageOS Pods are in kube-system by default secretRefName: \u0026#34;storageos-api\u0026#34; # Reference from the Secret created in the previous step secretRefNamespace: \u0026#34;storageos-operator\u0026#34; # Namespace of the Secret k8sDistro: \u0026#34;gke\u0026#34; images: nodeContainer: \u0026#34;storageos/node:v2.3.2\u0026#34; # StorageOS version kvBackend: address: \u0026#39;storageos-etcd-client.storageos-etcd:2379\u0026#39; # Example address, change for your etcd endpoint # address: \u0026#39;10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379\u0026#39; # You can set ETCD server ips resources: requests: memory: \u0026#34;512Mi\u0026#34; cpu: 1 # nodeSelectorTerms: # - matchExpressions: # - key: \u0026#34;node-role.kubernetes.io/worker\u0026#34; # Compute node label will vary according to your installation # operator: In # values: # - \u0026#34;true\u0026#34;   Additional spec parameters are available on the Cluster Operator configuration page.\n  You can find more examples such as deployments referencing a external etcd kv store for StorageOS in the Cluster Operator examples page.\n Verify StorageOS Installation [root@master03]# kubectl -n kube-system get pods -w NAME READY STATUS RESTARTS AGE storageos-csi-helper-5cf59b5b4-f5nwr 2/2 Running 0 3m storageos-daemonset-75f6c 3/3 Running 0 3m storageos-daemonset-czbqx 3/3 Running 0 3m storageos-daemonset-zv4tq 3/3 Running 0 3m storageos-scheduler-6d67b46f67-5c46j 1/1 Running 0 3m  The above command watches the Pods created by the Cluster Definition example. Note that pods typically take approximately 65 seconds to enter the Running Phase.\n 4. License cluster Newly installed StorageOS clusters must be licensed within 24 hours. Our developer license is free, and supports up to 5TiB of provisioned storage.\nTo obtain a license, follow the instructions on our licensing operations  page.\n Install StorageOS on Kubernetes 1.13 The StorageOS Cluster Operator is a Kubernetes native application developed to deploy and configure StorageOS clusters, and assist with maintenance operations. We recommend its use for standard installations.\nThe operator is a Kubernetes controller that watches the StorageOSCluster CRD. Once the controller is ready, a StorageOS cluster definition can be created. The operator will deploy a StorageOS cluster based on the configuration specified in the cluster definition.\n Helm Note: If you want to use Helm to install StorageOS, follow the StorageOS Operator Helm Chart documentation.\nSteps to install StorageOS:  Install StorageOS Operator Create a Secret for default username and password Trigger bootstrap using a CustomResource Apply StorageOS licence  1. Install StorageOS operator Install the StorageOS Cluster Operator using the following yaml manifest.\nkubectl create -f https://github.com/storageos/cluster-operator/releases/download/v2.3.1/storageos-operator.yaml Verify the Cluster Operator Pod Status [root@master03]# kubectl -n storageos-operator get pod NAME READY STATUS RESTARTS AGE storageoscluster-operator-68678798ff-f28zw 1/1 Running 0 3m  The READY 1/1 indicates that storageoscluster resources can be created.\n 2. Create a Secret Before deploying a StorageOS cluster, create a Secret defining the StorageOS API Username and Password in base64 encoding. The API username and password are used to create the default StorageOS admin account which can be used with the StorageOS CLI and to login to the StorageOS GUI. The CSI credentials are used to register the CSI accounts, so Kubernetes and StorageOS communicate over an authenticated API.\napiVersion: v1 kind: Secret metadata: name: \u0026#34;storageos-api\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; labels: app: \u0026#34;storageos\u0026#34; type: \u0026#34;kubernetes.io/storageos\u0026#34; data: # echo -n \u0026#39;\u0026lt;secret\u0026gt;\u0026#39; | base64 apiUsername: c3RvcmFnZW9z apiPassword: c3RvcmFnZW9z # CSI Credentials csiProvisionUsername: c3RvcmFnZW9z csiProvisionPassword: c3RvcmFnZW9z csiControllerPublishUsername: c3RvcmFnZW9z csiControllerPublishPassword: c3RvcmFnZW9z csiNodePublishUsername: c3RvcmFnZW9z csiNodePublishPassword: c3RvcmFnZW9z csiControllerExpandUsername: c3RvcmFnZW9z csiControllerExpandPassword: c3RvcmFnZW9z This example contains a default password, for production installations, use a unique, strong password.\n You can define a base64 value by echo -n \u0026quot;mystring\u0026quot; | base64.\n  Make sure that the encoding of the credentials doesn\u0026rsquo;t have special characters such as \u0026lsquo;\\n\u0026rsquo;. The echo -n ensures that a trailing new line is not appended to the string.\n  If you wish to change the default accounts details post-install please see Managing Users\n 3 Trigger a StorageOS installation  upstream  eks  aks  gke   This is a Cluster Definition example.\napiVersion: \u0026#34;storageos.com/v1\u0026#34; kind: StorageOSCluster metadata: name: \u0026#34;example-storageos\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; spec: # StorageOS Pods are in kube-system by default secretRefName: \u0026#34;storageos-api\u0026#34; # Reference from the Secret created in the previous step secretRefNamespace: \u0026#34;storageos-operator\u0026#34; # Namespace of the Secret k8sDistro: \u0026#34;upstream\u0026#34; images: nodeContainer: \u0026#34;storageos/node:v2.3.2\u0026#34; # StorageOS version kvBackend: address: \u0026#39;storageos-etcd-client.storageos-etcd:2379\u0026#39; # Example address, change for your etcd endpoint # address: \u0026#39;10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379\u0026#39; # You can set ETCD server ips resources: requests: memory: \u0026#34;512Mi\u0026#34; cpu: 1 # nodeSelectorTerms: # - matchExpressions: # - key: \u0026#34;node-role.kubernetes.io/worker\u0026#34; # Compute node label will vary according to your installation # operator: In # values: # - \u0026#34;true\u0026#34; This is a Cluster Definition example.\napiVersion: \u0026#34;storageos.com/v1\u0026#34; kind: StorageOSCluster metadata: name: \u0026#34;example-storageos\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; spec: # StorageOS Pods are in kube-system by default secretRefName: \u0026#34;storageos-api\u0026#34; # Reference from the Secret created in the previous step secretRefNamespace: \u0026#34;storageos-operator\u0026#34; # Namespace of the Secret k8sDistro: \u0026#34;eks\u0026#34; images: nodeContainer: \u0026#34;storageos/node:v2.3.2\u0026#34; # StorageOS version kvBackend: address: \u0026#39;storageos-etcd-client.storageos-etcd:2379\u0026#39; # Example address, change for your etcd endpoint # address: \u0026#39;10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379\u0026#39; # You can set ETCD server ips resources: requests: memory: \u0026#34;512Mi\u0026#34; cpu: 1 # nodeSelectorTerms: # - matchExpressions: # - key: \u0026#34;node-role.kubernetes.io/worker\u0026#34; # Compute node label will vary according to your installation # operator: In # values: # - \u0026#34;true\u0026#34; This is a Cluster Definition example.\napiVersion: \u0026#34;storageos.com/v1\u0026#34; kind: StorageOSCluster metadata: name: \u0026#34;example-storageos\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; spec: # StorageOS Pods are in kube-system by default secretRefName: \u0026#34;storageos-api\u0026#34; # Reference from the Secret created in the previous step secretRefNamespace: \u0026#34;storageos-operator\u0026#34; # Namespace of the Secret k8sDistro: \u0026#34;aks\u0026#34; images: nodeContainer: \u0026#34;storageos/node:v2.3.2\u0026#34; # StorageOS version kvBackend: address: \u0026#39;storageos-etcd-client.storageos-etcd:2379\u0026#39; # Example address, change for your etcd endpoint # address: \u0026#39;10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379\u0026#39; # You can set ETCD server ips resources: requests: memory: \u0026#34;512Mi\u0026#34; cpu: 1 # nodeSelectorTerms: # - matchExpressions: # - key: \u0026#34;node-role.kubernetes.io/worker\u0026#34; # Compute node label will vary according to your installation # operator: In # values: # - \u0026#34;true\u0026#34; This is a Cluster Definition example.\napiVersion: \u0026#34;storageos.com/v1\u0026#34; kind: StorageOSCluster metadata: name: \u0026#34;example-storageos\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; spec: # StorageOS Pods are in kube-system by default secretRefName: \u0026#34;storageos-api\u0026#34; # Reference from the Secret created in the previous step secretRefNamespace: \u0026#34;storageos-operator\u0026#34; # Namespace of the Secret k8sDistro: \u0026#34;gke\u0026#34; images: nodeContainer: \u0026#34;storageos/node:v2.3.2\u0026#34; # StorageOS version kvBackend: address: \u0026#39;storageos-etcd-client.storageos-etcd:2379\u0026#39; # Example address, change for your etcd endpoint # address: \u0026#39;10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379\u0026#39; # You can set ETCD server ips resources: requests: memory: \u0026#34;512Mi\u0026#34; cpu: 1 # nodeSelectorTerms: # - matchExpressions: # - key: \u0026#34;node-role.kubernetes.io/worker\u0026#34; # Compute node label will vary according to your installation # operator: In # values: # - \u0026#34;true\u0026#34;   Additional spec parameters are available on the Cluster Operator configuration page.\n  You can find more examples such as deployments referencing a external etcd kv store for StorageOS in the Cluster Operator examples page.\n Verify StorageOS Installation [root@master03]# kubectl -n kube-system get pods -w NAME READY STATUS RESTARTS AGE storageos-csi-helper-5cf59b5b4-f5nwr 2/2 Running 0 3m storageos-daemonset-75f6c 3/3 Running 0 3m storageos-daemonset-czbqx 3/3 Running 0 3m storageos-daemonset-zv4tq 3/3 Running 0 3m storageos-scheduler-6d67b46f67-5c46j 1/1 Running 0 3m  The above command watches the Pods created by the Cluster Definition example. Note that pods typically take approximately 65 seconds to enter the Running Phase.\n 4. License cluster Newly installed StorageOS clusters must be licensed within 24 hours. Our developer license is free, and supports up to 5TiB of provisioned storage.\nTo obtain a license, follow the instructions on our licensing operations  page.\n  First StorageOS volume If this is your first installation you may wish to follow the StorageOS Volume guide for an example of how to mount a StorageOS volume in a Pod.\n","excerpt":"Make sure the prerequisites for StorageOS are satisfied before proceeding.\n  Any Kubernetes managed …","ref":"/docs/install/kubernetes/","title":"Kubernetes"},{"body":"Over the past several months, we\u0026rsquo;ve been hard at work on StorageOS V2, which contains some significant enhancements over our v1 product. We\u0026rsquo;ve built V2 based on our observations of trends in the industry, as well as our own experience.\nMany of our customers want to run big clusters - in the tens or hundreds of nodes. In these sorts of big environments, the challenges multiply. Not only do we need to scale well, but we also need to be more failure tolerant. Bigger environments typically suffer higher failure rates (more nodes = greater chance of something failing), but are also subject to all sorts of transient conditions such as network partitions.\nThe second trend we\u0026rsquo;ve seen become increasingly common is the desire to run multiple clusters, and consume storage between them in some way - sometimes to implement novel topologies such as a centralised storage cluster with satellites consuming the storage, and sometimes to replicate data between those clusters for HA or DR purposes.\nWe\u0026rsquo;ve built V2 with these architectures and design patterns in mind. Not only does it scale well, but it contains the foundations we need to implement a rich set of multi-cluster functionality.\nUpgraded Control Plane At the heart of the V2 release is an upgraded control plane. We\u0026rsquo;ve changed a lot here. Firstly, our usage of etcd is vastly improved. We\u0026rsquo;ve learnt a lot about the subtleties of distributed consensus in the last year, particularly in noisy or unpredictable environments. Not only is StorageOS V2 much lighter on your etcd cluster, but it\u0026rsquo;s a lot more tolerant of transient failure conditions that are often found in cloud environments, or clusters under heavy load.\nWe spent some time describing and testing our internal state machine using the TLA+ formal verification language. This allows us to have a much higher degree of confidence that our algorithms will behave correctly, particularly under hard-to-test edge cases and failure conditions.\nAdditionally, we\u0026rsquo;ve changed the way volumes behave with respect to centralised scheduling. Each volume group (consisting of a master and 0 or more replicas) now behaves as a strongly consistent unit allowing it to take action independent of the activities of the rest of the cluster. Other state can be distributed around the cluster using eventually consistent mechanisms. This approach inherently scales better and allows StorageOS V2 to effectively manage many more nodes and volumes than before.\nWe\u0026rsquo;ve implemented TLS on all endpoints. Not only does this give you encrypted traffic between nodes in your storage cluster, it also protects all endpoints with strong, public key based authentication. Today\u0026rsquo;s IT environments can\u0026rsquo;t rely on firewalls to keep bad actors out - they must implement security at all layers within the stack - defense in depth. While we recognise that this brings a welcome relief to many security conscious administrators, we also know that managing certificate authorities (CAs) can be an unwelcome source of complexity. For this reason, StorageOS V2 implements an internal CA by default, to manage this complexity for you. If you\u0026rsquo;d prefer to integrate your own CA, we support that too - it\u0026rsquo;s up to you.\nFinally - our logging has undergone a complete transformation in this edition. We know that systems engineers and operators don\u0026rsquo;t just value headline features, but that observability and diagnostics are equally important. All logs are now decorated with rich context to help you understand what is happening within your cluster, and we\u0026rsquo;ll output in json by default, for easy ingestion into log aggregators such as Elasticsearch.\nUpgraded Data Plane Not to be outdone, our data plane contains some significant improvements.\nFirstly, we\u0026rsquo;ve completely re-written our sync algorithm (see Delta Sync, used when seeding or catching up replicas that have been offline or partitioned. Our new algorithm uses a Hash List to sync only changed sections of a volume (similar in some ways to what rsync does). StorageOS maintains these hashes during normal operation, meaning that when resyncing a failed replica, for example after a node reboot, we can very quickly and efficiently catch this replica up, rather than needing to promote and build a new one from scratch. This improves resiliency within your cluster, and prevents using excessive network bandwidth during failover conditions - at a time when it might be needed the most.\nSecondly, a new threading model, with dynamic pool sizing, means that StorageOS is faster, a lot faster. In our tests we observed improvements across the board, with improvements in throughput of up to 135% for some scenarios.\n","excerpt":"Over the past several months, we\u0026rsquo;ve been hard at work on StorageOS V2, which contains some …","ref":"/docs/introduction/overview/","title":"What is StorageOS V2"},{"body":"StorageOS requires certain kernel modules to function, in particular Linux-IO , an open-source implementation of the SCSI target.\nWe require the following modules to be loaded:\n target_core_mod tcm_loop target_core_file configfs target_core_user uio   N.B. Other applications utilising TCMU cannot be run concurrently with StorageOS. Doing so may result in corruption of data. On startup, StorageOS will detect if other applications are using TCMU and fall back to FUSE. However if StorageOS is started first there is no mechanism for StorageOS to fallback to FUSE if another application begins to use TCMU. TCMU can be disabled using the DISABLE_TCMU StorageOSCluster spec parameter.\n Depending on the distribution, the modules are shipped as part of the base kernel package or as part of a kernel extras package which needs to be installed.\nDistribution Specifics The following distributions are supported by default:\n RHEL 7.5 CentOS 7 Debian 9 Ubuntu Azure RancherOS - Note CSI is not supported on RancherOS  Ubuntu 16.04/18.04 requires the installation of additional packages.\n N.B. Ubuntu 16.04/18.04 AWS and Ubuntu 18.04 GCE do not provide the necessary linux-image-extra package - see below for more information\n Ubuntu Package Installation Ubuntu 16.04/18.04 Generic and Ubuntu 16.04 GCE require extra packages:\nUbuntu 16.04:\nsudo apt -y update sudo apt -y install linux-image-extra-$(uname -r) Ubuntu 18.04+:\nsudo apt -y update sudo apt -y install linux-modules-extra-$(uname -r) Ubuntu With AWS Or GCE Kernels Ubuntu 16.04/18.04 AWS and Ubuntu 18.04 GCE do not yet provide the linux-image-extra package. As such you should either use Debian, CentOS or RHEL, or install the non-cloud-provider optimised Ubuntu kernel.\nInstalling the non-cloud-provider optimised Ubuntu kernel is something that should only be done with full understanding of potential ramifications.\nsudo apt -y update sudo apt install -y linux-virtual linux-image-extra-virtual sudo apt purge -y linux*aws # Reboot the machine sudo shutdown -r now Automatic Configuration Once required kernel modules are installed on the system, for convenience we provide a container which will ensure the appropriate modules are loaded and ready for use at runtime. On Docker installations, you will need to run the init container prior to starting StorageOS. Our installation guides for Kubernetes and OpenShift include this step.\n# Load the required kernel modules. The Kubernetes and OpenShift installations include this step. docker run --name enable_lio \\  --privileged \\  --rm \\  --cap-add=SYS_ADMIN \\  -v /lib/modules:/lib/modules \\  -v /sys:/sys:rshared \\  storageos/init:0.2 Manual Configuration For those wishing to manage their own kernel configuration, rather than using the init container, perform the following steps:\n Ensure kernel modules are all loaded per list above Ensure configfs is loaded and mounted at /sys/kernel/config  ","excerpt":"StorageOS requires certain kernel modules to function, in particular Linux-IO , an open-source …","ref":"/docs/prerequisites/systemconfiguration/","title":"System Configuration"},{"body":"StorageOSCluster Resource Configuration The following table lists the configurable spec parameters of the StorageOSCluster custom resource and their default values.\n   Parameter Description Default     csi.deploymentStrategy CSI helper deployment strategy (statefulset or deployment) statefulset   csi.enable Enable CSI setup false   csi.enableControllerPublishCreds Enable CSI controller publish credentials false   csi.enableNodePublishCreds Enable CSI node publish credentials false   csi.enableProvisionCreds Enable CSI provision credentials false   debug Enable debug mode for all the cluster nodes false   disableFencing Disable Pod fencing false   disableScheduler Disable StorageOS scheduler false   disableTCMU Disable TCMU to allow co-existence with other TCMU users. Disabling TCMU degrades performance false   disableTelemetry Disable telemetry reports false   forceTCMU Forces TCMU to be enabled or causes StorageOS to abort startup false   images.csiClusterDriverRegistrarContainer CSI Cluster Driver Registrar Container image quay.io/k8scsi/csi-cluster-driver-registrar:v1.0.1   images.csiExternalAttacherContainer CSI External Attacher Container image quay.io/k8scsi/csi-attacher:v1.0.1   images.csiExternalProvisionerContainer CSI External Provisioner Container image storageos/csi-provisioner:v1.0.1   ìmages.csiLivenessProbeContainer CSI Liveness Probe Container Image quay.io/k8scsi/livenessprobe:v1.0.1   images.csiNodeDriverRegistrarContainer CSI Node Driver Registrar Container image quay.io/k8scsi/csi-node-driver-registrar:v1.0.1   images.hyperkubeContainer Deprecated field - HyperKube Container image Default dependent on Scheduler version   images.initContainer StorageOS init container image storageos/init:1.0.0   images.kubeSchedulerContainer Kube scheduler container image Default dependent on Scheduler version   images.nfsContainer StorageOS nfs container image storageos/nfs:1.0.0   images.nodeContainer StorageOS node container image storageos/node:v2.3.2   ingress.annotations Annotations of the ingress used by the cluster    ingress.enable Enable ingress for the cluster false   ingress.hostname Hostname to be used in cluster ingress storageos.local   ingress.tls Enable TLS for the ingress false   k8sDistro The name of the Kubernetes distribution is use, e.g. rancher or eks    kvBackend.address Comma-separated list of addresses of external key-value store. (1.2.3.4:2379,2.3.4.5:2379)    kvBackend.backend (v2 deprecated) Name of the key-value store to use. Set to etcd for external key-value store. embedded   namespace Namespace where storageos cluster resources are created kube-system   nodeSelectorTerms Set node selector for storageos pod placement    pause Pause the operator for cluster maintenance false   resources Set resource requirements for the containers    secretRefName Reference name of storageos secret    secretRefNamespace Namespace of storageos secret    service.annotations Annotations of the Service used by the cluster    service.externalPort External port of the Service used by the cluster 5705   service.internalPort Internal port of the Service used by the cluster 5705   service.name Name of the Service used by the cluster storageos   service.type Type of the Service used by the cluster ClusterIP   sharedDir Path to be shared with kubelet container when deployed as a pod /var/lib/kubelet/plugins/kubernetes.io~storageos   tlsEtcdSecretRefName Secret containing etcd client certificates    tlsEtcdSecretRefNamespace Namespace of the tlsEtcdSecretRefName    tolerations Set pod tolerations for storageos pod placement     ","excerpt":"StorageOSCluster Resource Configuration The following table lists the configurable spec parameters …","ref":"/docs/reference/cluster-operator/configuration/","title":"Configuration"},{"body":" Make sure the prerequisites for StorageOS are satisfied before proceeding. Including the deployment of an etcd cluster and configuration of CRI-O PID limits.\n  For OpenShift upgrades, refer to the OpenShift platform page.\n  If you have installed OpenShift in AWS ensure that the requisite ports are opened for the worker nodes\u0026rsquo; security group.\n  Make sure to add a StorageOS licence after installing.\n StorageOS v2 supports OpenShift v4. For more information check the OpenShift platform page.\n OperatorHub Manual    OperatorHub   Select the OperatorHub from the Catalog sub menu and search for StorageOS\n Choose between using the RedHat Market Place or the Community Operators installation.\n   Select StorageOS and click install\n  Select the install options\n Make sure the Approval Strategy is set to Manual. So the StorageOS Operator doesn\u0026rsquo;t upgrade versions without explicit approval.\n   Start the approval procedure\n  Follow the approval link\n  Approve the installation\n  The StorageOS Cluster Operator is installed along the required CRDs\n  Create a Secret in the openshift-operators project\n  Use the YAML options to create a secret containing the apiUsername and an apiPassword key. The username and password defined in the secret will be used to authenticate when using the StorageOS CLI and GUI. For the communication between StorageOS and OpenShift, the CSI credentials csiProvisionUsername, csiProvisionPassword, csiControllerPublishUsername, csiControllerPublishPassword, csiNodePublishUsername, csiNodePublishPassword are needed. Take note of which project you created the secret in.\nInput the Secret as YAML for simplicity.\napiVersion:v1kind:Secretmetadata:name:storageos-apinamespace:openshift-operatorstype:\u0026#34;kubernetes.io/storageos\u0026#34;data:# echo -n \u0026#39;\u0026lt;secret\u0026gt;\u0026#39; | base64apiUsername:c3RvcmFnZW9zapiPassword:c3RvcmFnZW9z# CSI CredentialscsiProvisionUsername:c3RvcmFnZW9zcsiProvisionPassword:c3RvcmFnZW9zcsiControllerPublishUsername:c3RvcmFnZW9zcsiControllerPublishPassword:c3RvcmFnZW9zcsiNodePublishUsername:c3RvcmFnZW9zcsiNodePublishPassword:c3RvcmFnZW9zcsiControllerExpandUsername:c3RvcmFnZW9zcsiControllerExpandPassword:c3RvcmFnZW9z  Go to the \u0026ldquo;Installed Operators\u0026rdquo;\n Verify that the StorageOS Cluster Operator is installed\n   Go to the \u0026ldquo;StorageOS Cluster\u0026rdquo; section\n  Create the StorageOS Cluster\n A StorageOS Cluster is defined using a Custom Resource Definition\n   Create the Custom Resource\nThe StorageOS cluster resource describes the StorageOS cluster that will be created. Parameters such as the secretRefName, the secretRefNamespace and the kvBackend.address are mandatory.\nAdditional spec parameters are available on the Cluster Operator configuration page.\napiVersion: \u0026#34;storageos.com/v1\u0026#34; kind: StorageOSCluster metadata: name: storageos namespace: openshift-operators spec: # StorageOS Pods are in kube-system by default secretRefName: \u0026#34;storageos-api\u0026#34; # Reference the Secret created in the previous step secretRefNamespace: \u0026#34;openshift-operators\u0026#34; # Namespace of the Secret created in the previous step k8sDistro: \u0026#34;openshift\u0026#34; kvBackend: address: \u0026#39;storageos-etcd-client.etcd:2379\u0026#39; # Example address, change for your etcd endpoint # address: \u0026#39;10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379\u0026#39; # You can set ETCD server ips resources: requests: memory: \u0026#34;512Mi\u0026#34; cpu: 1 # nodeSelectorTerms: # - matchExpressions: # - key: \u0026#34;node-role.kubernetes.io/worker\u0026#34; # Compute node label will vary according to your installation # operator: In # values: # - \u0026#34;true\u0026#34;   Verify that the StorageOS Cluster Resource enters a running state.\n It can take up to a minute to report the StorageOS Pods ready\n   Check the StorageOS Pods in the kube-system project\n A Status of 3/3 for the Daemonset Pods indicates that StorageOS is bootstrapped successfully.\n   License cluster\n A newly installed StorageOS cluster does not include a licence. A cluster must be licensed within 24 hours of the installation. For more information, check the reference licence page.\n You can apply a Free Developer licence following the operations licensing page, or purchase a licence contacting sales@storageos.com.\n   Manual install The StorageOS Cluster Operator is a Kubernetes native application developed to deploy and configure StorageOS clusters, and assist with maintenance operations. We recommend its use for standard installations.\nThe operator is a Kubernetes controller that watches the StorageOSCluster CRD. Once the controller is ready, a StorageOS cluster definition can be created. The operator will deploy a StorageOS cluster based on the configuration specified in the cluster definition.\n Helm Note: If you want to use Helm to install StorageOS, follow the StorageOS Operator Helm Chart documentation.\nSteps to install StorageOS:  Install StorageOS Operator Create a Secret for default username and password Trigger bootstrap using a CustomResource Apply StorageOS licence  1. Install StorageOS operator Install the StorageOS Cluster Operator using the following yaml manifest.\noc create -f https://github.com/storageos/cluster-operator/releases/download/v2.3.1/storageos-operator.yaml Verify the Cluster Operator Pod Status [root@master03]# oc -n storageos-operator get pod NAME READY STATUS RESTARTS AGE storageoscluster-operator-68678798ff-f28zw 1/1 Running 0 3m  The READY 1/1 indicates that storageoscluster resources can be created.\n 2. Create a Secret Before deploying a StorageOS cluster, create a Secret defining the StorageOS API Username and Password in base64 encoding. The API username and password are used to create the default StorageOS admin account which can be used with the StorageOS CLI and to login to the StorageOS GUI. The CSI credentials are used to register the CSI accounts, so Kubernetes and StorageOS communicate over an authenticated API.\napiVersion: v1 kind: Secret metadata: name: \u0026#34;storageos-api\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; labels: app: \u0026#34;storageos\u0026#34; type: \u0026#34;kubernetes.io/storageos\u0026#34; data: # echo -n \u0026#39;\u0026lt;secret\u0026gt;\u0026#39; | base64 apiUsername: c3RvcmFnZW9z apiPassword: c3RvcmFnZW9z # CSI Credentials csiProvisionUsername: c3RvcmFnZW9z csiProvisionPassword: c3RvcmFnZW9z csiControllerPublishUsername: c3RvcmFnZW9z csiControllerPublishPassword: c3RvcmFnZW9z csiNodePublishUsername: c3RvcmFnZW9z csiNodePublishPassword: c3RvcmFnZW9z csiControllerExpandUsername: c3RvcmFnZW9z csiControllerExpandPassword: c3RvcmFnZW9z This example contains a default password, for production installations, use a unique, strong password.\n You can define a base64 value by echo -n \u0026quot;mystring\u0026quot; | base64.\n  Make sure that the encoding of the credentials doesn\u0026rsquo;t have special characters such as \u0026lsquo;\\n\u0026rsquo;. The echo -n ensures that a trailing new line is not appended to the string.\n  If you wish to change the default accounts details post-install please see Managing Users\n 3 Trigger a StorageOS installation This is a Cluster Definition example.\napiVersion: \u0026#34;storageos.com/v1\u0026#34; kind: StorageOSCluster metadata: name: \u0026#34;example-storageos\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; spec: secretRefName: \u0026#34;storageos-api\u0026#34; # Reference from the Secret created in the previous step secretRefNamespace: \u0026#34;storageos-operator\u0026#34; # Namespace of the Secret k8sDistro: \u0026#34;openshift\u0026#34; images: nodeContainer: \u0026#34;storageos/node:v2.3.2\u0026#34; # StorageOS version kvBackend: address: \u0026#39;storageos-etcd-client.storageos-etcd:2379\u0026#39; # Example address, change for your etcd endpoint # address: \u0026#39;10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379\u0026#39; # You can set ETCD server ips resources: requests: memory: \u0026#34;512Mi\u0026#34; cpu: 1 # nodeSelectorTerms: # - matchExpressions: # - key: \u0026#34;node-role.kubernetes.io/worker\u0026#34; # Compute node label will vary according to your installation # operator: In # values: # - \u0026#34;true\u0026#34;  Additional spec parameters are available on the Cluster Operator configuration page.\n  You can find more examples such as deployments referencing a external etcd kv store for StorageOS in the Cluster Operator examples page.\n Verify StorageOS Installation [root@master03]# oc -n kube-system get pods -w NAME READY STATUS RESTARTS AGE storageos-csi-helper-5cf59b5b4-f5nwr 2/2 Running 0 3m storageos-daemonset-75f6c 3/3 Running 0 3m storageos-daemonset-czbqx 3/3 Running 0 3m storageos-daemonset-zv4tq 3/3 Running 0 3m storageos-scheduler-6d67b46f67-5c46j 1/1 Running 0 3m  The above command watches the Pods created by the Cluster Definition example. Note that pods typically take approximately 65 seconds to enter the Running Phase.\n 4. License cluster Newly installed StorageOS clusters must be licensed within 24 hours. Our developer license is free, and supports up to 5TiB of provisioned storage.\nTo obtain a license, follow the instructions on our licensing operations  page.\n  First StorageOS volume If this is your first installation you may wish to follow the StorageOS Volume guide for an example of how to mount a StorageOS volume in a Pod.\n","excerpt":"Make sure the prerequisites for StorageOS are satisfied before proceeding. Including the deployment …","ref":"/docs/install/openshift/","title":"OpenShift"},{"body":"Port list StorageOS daemons listen on specific ports, which we require to be accessible between all nodes in the cluster:\n   Port Number TCP/UDP Use     5701 TCP gRPC   5703 TCP DirectFS   5704 TCP Dataplane Supervisor   5705 TCP REST API   5711 TCP \u0026amp; UDP Gossip service   25705-25960 TCP RWX Volume Endpoints    StorageOS also uses ephemeral ports to dial-out to these ports on other StorageOS nodes. For this reason, outgoing traffic should be enabled.\nFirewalls and VPS providers Some VPS providers (such as Digital Ocean) ship default firewall rulesets which must be updated to allow StorageOS to run. Some example rules are shown below - modify to taste.\nUFW For distributions using UFW, such as RHEL and derivatives:\nufw default allow outgoing ufw allow 5701:5711/tcp ufw allow 5711/udp ufw allow 25705:25960/tcp Firewalld For distributions that enable firewalld to control iptables such as some installations of OpenShift.\nfirewall-cmd --permanent --new-service=storageos firewall-cmd --permanent --service=storageos --add-port=5700-5800/tcp --add-port=25705-25960/tcp firewall-cmd --add-service=storageos --zone=public --permanent firewall-cmd --reload Iptables For those using plain iptables:\n# Inbound traffic iptables -I INPUT -i lo -m comment --comment \u0026#39;Permit loopback traffic\u0026#39; -j ACCEPT iptables -I INPUT -m state --state ESTABLISHED,RELATED -m comment --comment \u0026#39;Permit established traffic\u0026#39; -j ACCEPT iptables -I INPUT -p tcp --dport 5701:5711 -m comment --comment \u0026#39;StorageOS\u0026#39; -j ACCEPT iptables -I INPUT -p udp --dport 5711 -m comment --comment \u0026#39;StorageOS\u0026#39; -j ACCEPT iptables -I INPUT -p tcp --dport 25705:25960 -m comment --comment \u0026#39;StorageOS\u0026#39; -j ACCEPT # Outbound traffic iptables -I OUTPUT -o lo -m comment --comment \u0026#39;Permit loopback traffic\u0026#39; -j ACCEPT iptables -I OUTPUT -d 0.0.0.0/0 -m comment --comment \u0026#39;Permit outbound traffic\u0026#39; -j ACCEPT Please ensure that the iptables rules you have added above come before any default DROP or REJECT rules.\n","excerpt":"Port list StorageOS daemons listen on specific ports, which we require to be accessible between all …","ref":"/docs/prerequisites/firewalls/","title":"Firewalls"},{"body":"  Start by understanding the architecture.\n  Follow our installation guides for Kubernetes, OpenShift or Rancher.\n  Dive into our Use Case documentation for Workloads deployed in Kubernetes.\n  ","excerpt":"  Start by understanding the architecture.\n  Follow our installation guides for Kubernetes, …","ref":"/docs/introduction/quickstart/","title":"Quickstart"},{"body":"Before deploying a StorageOS cluster, create a Secret to define the StorageOS API Username and Password in base64 encoding.\nkubectl create -f - \u0026lt;\u0026lt;END apiVersion: v1 kind: Secret metadata: name: \u0026#34;storageos-api\u0026#34; namespace: \u0026#34;default\u0026#34; labels: app: \u0026#34;storageos\u0026#34; type: \u0026#34;kubernetes.io/storageos\u0026#34; data: apiUsername: c3RvcmFnZW9z apiPassword: c3RvcmFnZW9z # CSI Credentials csiProvisionUsername: c3RvcmFnZW9z csiProvisionPassword: c3RvcmFnZW9z csiControllerPublishUsername: c3RvcmFnZW9z csiControllerPublishPassword: c3RvcmFnZW9z csiNodePublishUsername: c3RvcmFnZW9z csiNodePublishPassword: c3RvcmFnZW9z csiControllerExpandUsername: c3RvcmFnZW9z csiControllerExpandPassword: c3RvcmFnZW9z END This example contains a default password, for production installations, use a unique, strong password.\n Make sure that the encoding of the credentials doesn\u0026rsquo;t have special characters such as \u0026lsquo;\\n\u0026rsquo;.\n  You can define a base64 value by echo -n \u0026quot;mystring\u0026quot; | base64.\n Create a cluster-config.yaml according to your needs from the examples below.\nkubectl create -f cluster-config.yaml Note that StorageOS will be deployed in spec.namespace (storageos by default), irrespective of what NameSpace the CR is defined in.\n Examples  You can checkout all the parameters configurable in the configuration page.\n All examples must reference the storageos-api Secret.\nspec:secretRefName:\u0026#34;storageos-api\u0026#34;# Reference to the Secret created in the previous stepsecretRefNamespace:\u0026#34;storageos-operator\u0026#34;# Namespace of the SecretCheck out Cluster Definition examples for full CR files.\nInstalling with an external etcd spec:kvBackend:address:\u0026#39;10.43.93.95:2379\u0026#39;# IP of the SVC that exposes ETCD# address: \u0026#39;10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379\u0026#39; # You can specify individual IPs of the etcd serversIf using Etcd with mTLS, you need to specify the secret that hold the certificates with the following parameters:\nspec:# External mTLS secured etcd cluster specific propertiestlsEtcdSecretRefName:\u0026#34;etcd-client-tls\u0026#34;# Secret containing etcd client certificatestlsEtcdSecretRefNamespace:\u0026#34;storageos-etcd\u0026#34;# Namespace of the client certificates secretFollow the etcd operations page to setup the secret with the Etcd client certificate, client key and CA.\nInstalling to a subset of nodes In this case we select nodes that are workers. To make sure that StorageOS doesn\u0026rsquo;t start in Master nodes.\nYou can see the labels in the nodes by kubectl get node --show-labels.\nspec:nodeSelectorTerms:- matchExpressions:- key:\u0026#34;node-role.kubernetes.io/worker\u0026#34;operator:Invalues:- \u0026#34;true\u0026#34;# OpenShift uses \u0026#34;node-role.kubernetes.io/compute=true\u0026#34;# Rancher uses \u0026#34;node-role.kubernetes.io/worker=true\u0026#34;# Kops uses \u0026#34;node-role.kubernetes.io/node=\u0026#34; Different provisioners and Kubernetes distributions use node labels differently to specify master vs workers. Node Taints are not enough to make sure StorageOS doesn\u0026rsquo;t start in a node. The JOIN variable is defined by the operator by selecting all the nodes that match the nodeSelectorTerms.\n Enabling CSI spec:csi:enable:truedeploymentStrategy:deploymentenableProvisionCreds:trueenableControllerPublishCreds:trueenableNodePublishCreds:trueenableControllerExpandCreds:trueThe credentials must be defined in the storageos-api Secret\napiVersion:v1kind:Secretmetadata:name:\u0026#34;storageos-api\u0026#34;namespace:\u0026#34;storageos-operator\u0026#34;labels:app:\u0026#34;storageos\u0026#34;type:\u0026#34;kubernetes.io/storageos\u0026#34;data:# echo -n \u0026#39;\u0026lt;secret\u0026gt;\u0026#39; | base64apiUsername:c3RvcmFnZW9zapiPassword:c3RvcmFnZW9z# Add base64 encoded creds below for CSI credentials.csiProvisionUsername:c3RvcmFnZW9zcsiProvisionPassword:c3RvcmFnZW9zcsiControllerPublishUsername:c3RvcmFnZW9zcsiControllerPublishPassword:c3RvcmFnZW9zcsiNodePublishUsername:c3RvcmFnZW9zcsiNodePublishPassword:c3RvcmFnZW9zcsiControllerExpandUsername:c3RvcmFnZW9zcsiControllerExpandPassword:c3RvcmFnZW9zSpecifying a shared directory for use with kubelet as a container spec:sharedDir:\u0026#39;/var/lib/kubelet/plugins/kubernetes.io~storageos\u0026#39;Defining pod resource requests and reservations spec:resources:requests:memory:\u0026#34;512Mi\u0026#34;# cpu: \u0026#34;1\u0026#34;# limits:# memory: \u0026#34;4Gi\u0026#34;Limiting StorageOS can cause malfunction for IO to StorageOS volumes, therefore we do not currently recommend applying upper limits to resources for StorageOS pods.\nSpecifying custom Tolerations spec:tolerations:- key:\u0026#34;key1\u0026#34;operator:\u0026#34;Equal\u0026#34;value:\u0026#34;value1\u0026#34;effect:\u0026#34;EffectToTolerate\u0026#34;- key:\u0026#34;key2\u0026#34;operator:\u0026#34;Exists\u0026#34;Custom tolerations specified in the StorageOSCluster definition are added to all StorageOS components; the StorageOS daemonset, CSI helper and scheduler.\nIn the above example a toleration key1=value1:EffectToTolerate would be tolerated and key2 would be tolerated regardless of the value and effect. For more information about tolerations please see the Kubernetes documentation.\n","excerpt":"Before deploying a StorageOS cluster, create a Secret to define the StorageOS API Username and …","ref":"/docs/reference/cluster-operator/examples/","title":"Cluster Operator examples"},{"body":" Modern versions of Kubernetes, Docker or other Container Runtimes enable mount propagation by default.\n StorageOS requires mount propagation enabled to present devices as volumes for containers (see linux kernel documentation here).\nCertain versions of docker ship with a systemd manifest with MountFlags set to \u0026lsquo;slave\u0026rsquo;, thus preventing StorageOS from working. This can be removed or set to \u0026lsquo;shared\u0026rsquo; with a systemd drop in:\nmkdir -p /etc/systemd/system/docker.service.d/ cat \u0026lt;\u0026lt;EOF \u0026gt; /etc/systemd/system/docker.service.d/mount_propagation_flags.conf [Service] MountFlags=shared EOF # systemctl daemon-reload # systemctl restart docker.service To confirm behaviour, the following command should run without error.\ndocker run -it --rm -v /mnt:/mnt:shared busybox sh -c /bin/date Orchestrators such as Kubernetes or OpenShift have their own ways of exposing this setting. Kubernetes 1.10 and OpenShift 3.10 have mount propagation enabled by default. Previous versions require that feature gates are enabled on the Kubernetes master\u0026rsquo;s controller-manager and apiserver services and in the kubelet service on each node.\nInstallations of orchestrators using Docker require that mount propagation is enabled for both.\nRefer to our installation pages for the orchestrators to see details on how to check and enable mount propagation where appropriate.\n","excerpt":"Modern versions of Kubernetes, Docker or other Container Runtimes enable mount propagation by …","ref":"/docs/prerequisites/mountpropagation/","title":"Mount Propagation"},{"body":"StorageOS is a certified Rancher application. We offer two installation methods:\n Rancher Catalogue - this is the easiest and requires just a few clicks Manual - allowing more control and visibility  Before proceeding, ensure that you have followed our prerequisites. On Rancher, pay particular attention to the OS version and image used - some platforms require extra mainline kernel modules to be enabled.\n Catalog Manual    Catalog Install StorageOS is a Certified application in the Rancher Catalog. You can install StorageOS using the Rancher application install.\nBefore completing the steps below, you will need an etcd cluster. For evaluations it is sufficient to use our simple test recipe. For production installations, please follow our production recipe. Make a note of the etcd endpoint URL in either case.\n  Select the System project of your cluster\n  Select the Apps tab and click Launch\n  Search for StorageOS and click on the App\nThis will install the StorageOS operator, which manages the StorageOS DaemonSet.\n  Check and ammend installation options\nA generic configuration for StorageOS is preset using the default values in the form. Be sure to check the etcd address and ensure it matches the value you noted at the beginning of this guide.\nThe catalog form exposes several useful parameters - documented below.\nFor further customization, you can opt to set the option to \u0026lsquo;Install StorageOS Cluster\u0026rsquo; to false and install a custom CR. See below for this.\n  Launch the StorageOS cluster\n  Verify the cluster bootstrap has successfully completed\n  License the newly installed cluster\nNewly installed StorageOS clusters must be licensed within 24 hours. Our developer license is free, and supports up to 5TiB of provisioned storage.\nYou will need access to the StorageOS API on port 5705 of any of your nodes. For convenience, it is often easiest to port forward the service using the following kubectl incantation (this will block, so a second terminal window may be advisable):\n$ kubectl port-forward -n kube-system svc/storageos 5705 Now follow the instructions on our licensing operations page to obtain and apply a license.\nInstallation of StorageOS is now complete.\n  Simple Customization - Modify Catalog Form The following options are exposed by the catalog form to allow some simple customization of the StorageOS installation.\n Cluster Operator namespace : The Kubernetes namespace where the StorageOS Cluster Operator and other resources will be created. Container Images : By default images are pulled from DockerHub, you can specify the image URLs when using private registries. Install StorageOS cluster : Controls the automatic deployment of StorageOS after installing the Cluster Operator. If set to false, the Operator will be created, but a Custom Resource will not be applied to the cluster. Launch the operator and proceed to the section Advanced Customization below. Namespace : The Kubernetes namespace where StorageOS will be installed. By default, StorageOS installs into the kube-system namespace, which will add a priority class to ensure high priority resource allocation. Installing StorageOS with the priority class prevents StorageOS from being evicted during periods of resource contention. It is inadvisable to modify this under normal circumstances. Username/Password : Default Username and Password for the admin account to be created at StorageOS bootstrap. A random password will be generated by leaving the field empty or clicking the Generate button. External etcd address(es) : Connection and configuration details for an external Etcd cluster.See our documentation here. Node Selectors and Tolerations : Control placement of StorageOS DaemonSet Pods. StorageOS will only be installed on the selected nodes. Tolerations : Define any tolerations you wish the DaemonSet to observe.  Advanced Customization - Apply Custom CR If Install StorageOS Cluster was set to false, StorageOS will not be bootstrapped automatically. After the StorageOS Operator is installed, you can now create a Custom Resource that describes the StorageOS cluster.\n  Select the System Workloads and Import YAML   Create the Secret and CustomResource This is an example.\n--- apiVersion: v1 kind: Secret metadata: name: \u0026#34;storageos-api\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; labels: app: \u0026#34;storageos\u0026#34; type: \u0026#34;kubernetes.io/storageos\u0026#34; data: # echo -n \u0026#39;\u0026lt;secret\u0026gt;\u0026#39; | base64 apiUsername: c3RvcmFnZW9z apiPassword: c3RvcmFnZW9z # CSI Credentials csiProvisionUsername: c3RvcmFnZW9z csiProvisionPassword: c3RvcmFnZW9z csiControllerPublishUsername: c3RvcmFnZW9z csiControllerPublishPassword: c3RvcmFnZW9z csiNodePublishUsername: c3RvcmFnZW9z csiNodePublishPassword: c3RvcmFnZW9z csiControllerExpandUsername: c3RvcmFnZW9z csiControllerExpandPassword: c3RvcmFnZW9z --- apiVersion: \u0026#34;storageos.com/v1\u0026#34; kind: StorageOSCluster metadata: name: \u0026#34;storageos\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; spec: # StorageOS Pods are in kube-system by default secretRefName: \u0026#34;storageos-api\u0026#34; # Reference from the Secret created in the previous step secretRefNamespace: \u0026#34;storageos-operator\u0026#34; # Namespace of the Secret k8sDistro: \u0026#34;rancher\u0026#34; images: nodeContainer: \u0026#34;storageos/node:v2.3.2\u0026#34; # StorageOS version kvBackend: address: \u0026#39;storageos-etcd-client.etcd:2379\u0026#39; # Example address, change for your etcd endpoint # address: \u0026#39;10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379\u0026#39; # You can set ETCD server ips sharedDir: \u0026#39;/var/lib/kubelet/plugins/kubernetes.io~storageos\u0026#39; # Needed when Kubelet as a container resources: requests: memory: \u0026#34;512Mi\u0026#34; nodeSelectorTerms: - matchExpressions: - key: \u0026#34;node-role.kubernetes.io/worker\u0026#34; # Compute node label will vary according to your installation operator: In values: - \u0026#34;true\u0026#34;  Additional spec parameters are available on the Cluster Operator configuration page.\n  You can find more examples such as deployments referencing a external etcd kv store for StorageOS in the Cluster Operator examples page.\n    Manual Installation The StorageOS Cluster Operator is a Kubernetes native application developed to deploy and configure StorageOS clusters, and assist with maintenance operations. We recommend its use for standard installations.\nThe operator is a Kubernetes controller that watches the StorageOSCluster CRD. Once the controller is ready, a StorageOS cluster definition can be created. The operator will deploy a StorageOS cluster based on the configuration specified in the cluster definition.\n Helm Note: If you want to use Helm to install StorageOS, follow the StorageOS Operator Helm Chart documentation.\nSteps to install StorageOS:  Install StorageOS Operator Create a Secret for default username and password Trigger bootstrap using a CustomResource Apply StorageOS licence  1. Install StorageOS operator Install the StorageOS Cluster Operator using the following yaml manifest.\nkubectl create -f https://github.com/storageos/cluster-operator/releases/download/v2.3.1/storageos-operator.yaml Verify the Cluster Operator Pod Status [root@master03]# kubectl -n storageos-operator get pod NAME READY STATUS RESTARTS AGE storageoscluster-operator-68678798ff-f28zw 1/1 Running 0 3m  The READY 1/1 indicates that storageoscluster resources can be created.\n 2. Create a Secret Before deploying a StorageOS cluster, create a Secret defining the StorageOS API Username and Password in base64 encoding. The API username and password are used to create the default StorageOS admin account which can be used with the StorageOS CLI and to login to the StorageOS GUI. The CSI credentials are used to register the CSI accounts, so Kubernetes and StorageOS communicate over an authenticated API.\napiVersion: v1 kind: Secret metadata: name: \u0026#34;storageos-api\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; labels: app: \u0026#34;storageos\u0026#34; type: \u0026#34;kubernetes.io/storageos\u0026#34; data: # echo -n \u0026#39;\u0026lt;secret\u0026gt;\u0026#39; | base64 apiUsername: c3RvcmFnZW9z apiPassword: c3RvcmFnZW9z # CSI Credentials csiProvisionUsername: c3RvcmFnZW9z csiProvisionPassword: c3RvcmFnZW9z csiControllerPublishUsername: c3RvcmFnZW9z csiControllerPublishPassword: c3RvcmFnZW9z csiNodePublishUsername: c3RvcmFnZW9z csiNodePublishPassword: c3RvcmFnZW9z csiControllerExpandUsername: c3RvcmFnZW9z csiControllerExpandPassword: c3RvcmFnZW9z This example contains a default password, for production installations, use a unique, strong password.\n You can define a base64 value by echo -n \u0026quot;mystring\u0026quot; | base64.\n  Make sure that the encoding of the credentials doesn\u0026rsquo;t have special characters such as \u0026lsquo;\\n\u0026rsquo;. The echo -n ensures that a trailing new line is not appended to the string.\n  If you wish to change the default accounts details post-install please see Managing Users\n 3 Trigger a StorageOS installation This is a Cluster Definition example.\napiVersion: \u0026#34;storageos.com/v1\u0026#34; kind: StorageOSCluster metadata: name: \u0026#34;example-storageos\u0026#34; namespace: \u0026#34;storageos-operator\u0026#34; spec: # StorageOS Pods are in kube-system by default secretRefName: \u0026#34;storageos-api\u0026#34; # Reference from the Secret created in the previous step secretRefNamespace: \u0026#34;storageos-operator\u0026#34; # Namespace of the Secret k8sDistro: \u0026#34;rancher\u0026#34; images: nodeContainer: \u0026#34;storageos/node:v2.3.2\u0026#34; # StorageOS version kvBackend: address: \u0026#39;storageos-etcd-client.storageos-etcd:2379\u0026#39; # Example address, change for your etcd endpoint # address: \u0026#39;10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379\u0026#39; # You can set ETCD server ips resources: requests: memory: \u0026#34;512Mi\u0026#34; cpu: 1 # nodeSelectorTerms: # - matchExpressions: # - key: \u0026#34;node-role.kubernetes.io/worker\u0026#34; # Compute node label will vary according to your installation # operator: In # values: # - \u0026#34;true\u0026#34;  Additional spec parameters are available on the Cluster Operator configuration page.\n  You can find more examples such as deployments referencing a external etcd kv store for StorageOS in the Cluster Operator examples page.\n Verify StorageOS Installation [root@master03]# kubectl -n kube-system get pods -w NAME READY STATUS RESTARTS AGE storageos-csi-helper-5cf59b5b4-f5nwr 2/2 Running 0 3m storageos-daemonset-75f6c 3/3 Running 0 3m storageos-daemonset-czbqx 3/3 Running 0 3m storageos-daemonset-zv4tq 3/3 Running 0 3m storageos-scheduler-6d67b46f67-5c46j 1/1 Running 0 3m  The above command watches the Pods created by the Cluster Definition example. Note that pods typically take approximately 65 seconds to enter the Running Phase.\n 4. License cluster Newly installed StorageOS clusters must be licensed within 24 hours. Our developer license is free, and supports up to 5TiB of provisioned storage.\nTo obtain a license, follow the instructions on our licensing operations  page.\n  First StorageOS volume If this is your first installation you may wish to follow the StorageOS Volume guide for an example of how to mount a StorageOS volume in a Pod.\n","excerpt":"StorageOS is a certified Rancher application. We offer two installation methods:\n Rancher Catalogue …","ref":"/docs/install/rancher/","title":"Rancher"},{"body":"Upgrade StorageOS operator from yaml manifest Upgrade the StorageOS operator using the following yaml manifest.\nkubectl apply -f https://github.com/storageos/cluster-operator/releases/download/v2.3.1/storageos-operator.yaml  When you run the above command StorageOS Operator resources will be updated. Since, the Update Strategy of the StorageOS Operator Deployment is set to rolling update, a new StorageOS Operator Pod will be created. Only when the new Pod enters the Running Phase will the old Pod be deleted. Your StorageOS Cluster will not be affected while the StorageOS Operator is upgrading.\n Upgrade StorageOS Operator using Helm If you have installed the StorageOS Operator using the Helm Chart, then you can upgrade the operator using the following commands.\n$ helm list NAME REVISION STATUS CHART APP VERSION NAMESPACE storageos-v1 4 DEPLOYED storageos-operator-0.2.11 1.3.0 storageos-operator $ helm repo update $ helm upgrade $NAME storageos/storageos-operator  When you run the above command StorageOS Operator resources will be updated. Since, the Update Strategy of the StorageOS Operator Deployment is set to rolling update, a new StorageOS Operator Pod will be created. Only when the new Pod enters the Running Phase will the old Pod be deleted. Your StorageOS Cluster will not be affected while the StorageOS Operator is upgrading.\n ","excerpt":"Upgrade StorageOS operator from yaml manifest Upgrade the StorageOS operator using the following …","ref":"/docs/reference/cluster-operator/upgrade/","title":"Cluster Operator Upgrade"},{"body":"OS  Linux X86_64 Kernels satisfying our module prerequisites 3.x kernels have a limitation of 256 active volumes per node 4.x kernels have a limitation of 4096 active volumes per node We are distribution agnostic as long as our prerequisites are met  Orchestrators  Kubernetes 1.13+ OpenShift 4.0+  ","excerpt":"OS  Linux X86_64 Kernels satisfying our module prerequisites 3.x kernels have a limitation of 256 …","ref":"/docs/introduction/platforms/","title":"Supported Platforms and Orchestrators"},{"body":"","excerpt":"","ref":"/docs/concepts/","title":"Concepts"},{"body":"StorageOS recommends that a PID cgroup limit of 32768 be used for StorageOS pods.\n Most environments fulfill this prerequisite by default. Check the StorageOS init container logs as shown below to ensure this is the case.\n StorageOS pods running in Kubernetes are part of a PID cgroup that may limit the maximum number of PIDs that all containers in the PID cgroup slice can spawn. As the Linux kernel assigns a PID to processes and Light Weight Processes (LWP) a low limit can be easily reached under certain circumstances. The PID limit can be set by the Kubernetes distribution or by the container runtime. Generally the limit is set to the machine wide default limit of 32768 but some environments can set this as low as 1024. A low PID limit may prevent StorageOS from spawning the required threads.\nThe StorageOS init container runs a script that checks for the PID limit of the PID cGroup slice that the StorageOS pod runs in. If the script finds that the limit is less than 32768 it will log a warning. This warning can be viewed using kubectl to check the init container logs.\n$ kubectl -n kube-system logs -l app=storageos,kind=daemonset -c storageos-init WARNING: Effective max.pids limit (1024) less than RECOMMENDED_MAX_PIDS_LIMIT (32768) Setting a Kubernetes PID limit Kubernetes defaults to an unlimited PodPidsLimit, which results in the usage of the machine wide limit; typically 32768.\nFor information on how to configure the Kubernetes PID limit see the Kubernetes documentation here.\nSetting a CRI-O PID limit Certain orchestrators or setups use CRI-O as the container runtime. Openshift 4.x currently has CRI-O set a PID limit of 1024 by default. To configure the default CRI-O limit in Openshift 4.x see the RedHat documentation here. To configure CRI-O more generally see the CRI-O documentation here.\n","excerpt":"StorageOS recommends that a PID cgroup limit of 32768 be used for StorageOS pods.\n Most environments …","ref":"/docs/prerequisites/pidlimits/","title":"PID Limits"},{"body":"Minimum requirements: One machine with the following:\n Minimum two core with 4GB RAM. Linux with a 64-bit architecture. Kubernetes 1.9+ with CSI. Container Runtime Engine: CRI-O, Containerd or Docker 1.10+ with mount propagation enabled. The necessary ports should be open. See the ports and firewall settings. Etcd cluster for StorageOS A mechanism for device presentation.  Recommended:  At least three nodes for replication and high availability. Kubernetes 1.13+. Install the storageos CLI. If using Helm2, make sure the tiller ServiceAccount has enough privileges to create resources such as Namespaces, ClusterRoles, etc. For instance, following this installation procedure. System clocks synchronized using NTP or similar methods. While our distributed consensus algorithm does not require synchronised clocks, it does help to more easily correlate logs across multiple nodes. A PID cgroup limit of 32768.  ","excerpt":"Minimum requirements: One machine with the following:\n Minimum two core with 4GB RAM. Linux with a …","ref":"/docs/prerequisites/","title":"Prerequisites"},{"body":"Kubernetes with StorageOS StorageOS integrates transparently with Kubernetes and different distributions such as OpenShift, Rancher, EKS, AKS, GKE, etc. The user can provide standard PVC definitions and StorageOS will dynamically provision matching volumes. StorageOS presents volumes to containers with standard POSIX mount targets. This enables the Kubelet to mount StorageOS volumes using standard linux device files. Checkout device presentation for more details.\nKubernetes and StorageOS communicate with each other to perform actions such as creation, deletion or mounting of volumes. The CSI (Container Storage Interface) driver is the standard method of communication. Using CSI, Kubernetes and StorageOS communicate over a Unix domain socket.\nIt is recommended to use the CSI StorgeOS installation from Kubernetes 1.13 onwards. Although the former communication procedure, the native driver, is still in use and StorageOS maintains support for it, its use is discouraged as it is a deprecated method, to be removed in newer versions of Kubernetes.\nCSI (Container Storage Interface) Note CSI is the standard method of communication that enables storage drivers for Kubernetes to release on their own schedule. The CSI standard allows storage vendors to upgrade, update, and enhance their drivers without the need to update Kubernetes source code, or follow Kubernetes release cycles.\nCSI was released GA from Kubernetes 1.13. StorageOS v2 only supports the use of CSI as a storage driver. In addition, the StorageOS Cluster Operator handles the configuration of the CSI driver and its complexity by detecting the version of the Kubernetes installed.\nCheck out the status of the CSI release cycle in relation with Kubernetes in the CSI project page.\nCSI communication is fully supported by StorageOS if the cluster is deployed with any supported Linux Distribution.\nKubernetes Upgrades on Managed Services Managed services that support in place upgrades are fully supported. However, upgrading Kubernetes using green/blue deployments is not supported. This is because nodes are replaced rather than being upgraded. Any data stored on the nodes is lost when new nodes replace the previous ones.\nDocker Some managed Kubernetes platforms such as Azure AKS, enable the \u0026lsquo;Live-Restore\u0026rsquo; Docker feature, enabling containers to continue running while Docker is stopped or upgraded. This feature can cause nodes to hang while shutting-down or rebooting, as rather than going through an orderly shutdown, StorageOS (and other processes) are killed before the disks are synced and unmounted. Devices in this inaccessible state will log a warning similar to:\nTransport endpoint not connected  To prevent this behaviour, we advise disabling this feature by setting\n{ ... \u0026quot;live-restore\u0026quot;: false }  in /etc/docker/daemon.json.\nHere\u0026rsquo;s an example Ansible snippet that might be used to achieve this\n... - name: configure /etc/docker/daemon.json lineinfile: path: /etc/docker/daemon.json regexp: '^.*\u0026quot;live-restore\u0026quot;: true,$' line: ' \u0026quot;live-restore\u0026quot;: false,' backrefs: yes notify: restart docker ...   Note: Use at own risk; you may need to adapt the example to work in your environment\n ","excerpt":"Kubernetes with StorageOS StorageOS integrates transparently with Kubernetes and different …","ref":"/docs/install/","title":"Install"},{"body":"While StorageOS is platform agnostic, there are certain configurations or operations that relate to a specific platform.\nStorageOS can be used in any Kubernetes distribution when the prerequisites are met. That includes Kubernetes manged services.\n","excerpt":"While StorageOS is platform agnostic, there are certain configurations or operations that relate to …","ref":"/docs/platforms/","title":"Platforms"},{"body":"Use an external Etcd cluster StorageOS uses the etcd distributed key-value store to store essential cluster metadata and manage distributed configuration state. An embedded etcd instance is included in the StorageOS container, but for production environments and testing of production workloads, we recommend deploying an external etcd cluster. For more details about, and an example of, how to run etcd, see the External etcd Operations page.\nIt is highly recommended to use external etcd for cloud environments and place the etcd cluster on stable nodes. Placing the etcd on nodes that are recycled often might affect the normal operations of StorageOS.\nEtcd low latency IO It is recommended to run etcd on low-latency disks and keep other IO-intensive applications separate from the etcd nodes. Etcd is very sensitive to IO latency. Thus, the effect of disk contention can cause etcd downtime.\nBatch jobs such as backups, builds or application bundling can easily cause a high usage of disks making etcd unstable. It is recommended to run such workloads apart from the etcd servers.\nSetup of storage on the hosts We recommend creating a separate filesystem for StorageOS to mitigate the risk of filling the root filesystem on nodes. This has to be done for each node in the cluster.\nFollow the managing host storage best practices page for more details.\nResource reservations StorageOS resource consumption depends on the workloads and the StorageOS features in use.\nThe recommended minimum memory reservation for the StorageOS Pods is 512MB for non-production environments. However it is recommended to prepare nodes so StorageOS can operate with at least with 1-2GB of memory. StorageOS frees memory when possible.\nFor production environments, we recommend 4GB of Memory and 1 CPU as a minimum and to test StorageOS using realistic workloads and tune resources accordingly.\nStorageOS Pods resource allocation will impact directly on the availability of volumes in case of eviction or resource limit triggered restart. It is recommended to not limit StorageOS Pods.\nStorageOS implements a storage engine, therefore limiting CPU consumption might affect the I/O throughput of your volumes.\nSetting a Kubernetes PID limit StorageOS recommends that a PID cgroup limit of 32768 be set. StorageOS is a multi-threaded application and while most Kubernetes distributions set the PID cgroup limit to 32768, some environments can set a limit as low as 1024. The StorageOS init container will print a log message warning if the PID cgroup limit is too low. See our prerequisites for more information.\nMaintain a sufficient number of nodes for replicas to be created To ensure that a new replica can always be created, an additional node should be available. To guarantee high availability, clusters using Volumes with 1 replica must have at least 3 storage nodes. When using Volumes with 2 replicas, at least 4 storage nodes, 3 replicas, 5 nodes, etc.\nMinimum number of storage nodes = 1 (primary) + N (replicas) + 1\nFor more information, see the section on replication.\nStorageOS API username/password The API grants full access to StorageOS functionality, therefore we recommend that the default administrative password of \u0026lsquo;storageos\u0026rsquo; is reset to something unique and strong.\nYou can change the default parameters by encoding the apiUsername and apiPassword values (in base64) into the storageos-api secret.\nTo generate a unique password, a technique such as the following, which generates a pseudo-random 24 character string, may be used:\n# Generate strong password PASSWORD=$(cat -e /dev/urandom | tr -dc \u0026#39;a-zA-Z0-9-!@#$%^\u0026amp;*()_+~\u0026#39; | fold -w 24 | head -n 1) # Convert password to base64 representation for embedding in a K8S secret BASE64PASSWORD=$(echo -n $PASSWORD | base64) Note that the Kubernetes secret containing a strong password must be created before bootstrapping the cluster. Multiple installation procedures use this Secret to create a StorageOS account when the cluster first starts.\nStorageOS Pod placement StorageOS must run on all nodes that will contribute storage capacity to the cluster or that will host Pods which use StorageOS volumes. For production environments, it is recommended to avoid placing StorageOS Pods on Master nodes.\nStorageOS is deployed with a DaemonSet controller, and therefore tolerates the standard unschedulable (:NoSchedule) action. If that is the only taint placed on master or cordoned nodes StorageOS pods might start on them (see the Kubernetes docs for more details). To avoid scheduling StorageOS pods on master nodes, you can add an arbitrary taint to them for which the StorageOS DaemonSet won\u0026rsquo;t have a toleration.\nDedicated instance groups Cloud environments give users the ability to quickly scale the number of nodes in a cluster in response to their needs. Because of the ephemeral nature of the cloud, StorageOS recommends setting conservative downscaling policies.\nFor production clusters, it recommended to use dedicated instance groups for Stateful applications that allow the user to set different scaling policies and define StorageOS pools based on node selectors to collocate volumes.\nLosing a few nodes at the same time could cause the loss of data even when volume replicas are being used.\nPort blocking StorageOS exposes ports to operate. It is recommended that the ports are not accessible from outside the scope of your cluster.\nStorageOS in Docker EE StorageOS does not support running on Swarm nodes nor on mixed (Kubernetes and Swarm) nodes. StorageOS volumes have to be provisioned and used from Kubernetes nodes.\n","excerpt":"Use an external Etcd cluster StorageOS uses the etcd distributed key-value store to store essential …","ref":"/docs/best-practices/","title":"StorageOS Best Practices"},{"body":"","excerpt":"","ref":"/docs/operations/","title":"Operations"},{"body":"StorageOS can be used to provide permanent storage for other applications running in Kubernetes or other Orchestrators that are derived from Kubernetes such as OpenShift or Rancher. This is useful for running stateful applications, such as databases or CI/CD applications, under the control of Kubernetes as Kubernetes can make scheduling decisions without the application data being lost.\nWhat we have outlined in the cookbooks below are some quick deployments of stateful applications into a Kubernetes cluster. These examples are not production ready but have been provided to give you some insight into how to use StorageOS with stateful applications.\nStatefulSets The examples we have provided use StatefulSets as a way to deploy applications into Kubernetes. The reason for this is that the StatefulSet controller is designed to manage stateful applications and it \u0026ldquo;provides guarantees about the ordering and uniqueness\u0026rdquo; of sets of pods.\nPractically this means that when a StatefulSet scales, pods are created in order from 0-(N-1), if a StatefulSet scales down then pods are deleted in reverse order from (N-1)-0.\nSecondly, it means that the behaviour of the StatefulSet controller differs from that of Deployment and ReplicaSet controllers. For Deployment and ReplicaSet controllers \u0026ldquo;\u0026hellip; at many points in the lifetime of a replica set there will be 2 copies of a pod\u0026rsquo;s processes running concurrently\u0026rdquo;. Having two different pods mount a volume at the same time can cause corruption of data. Currently Kubernetes accessModes only apply restrictions to nodes mounting volumes rather than pods, so it is important that StatefulSets are used with StorageOS volumes so the necessary pod uniqueness guarantees are maintained.\nStatefulSet Manifests The StorageOS specific part of the Kubernetes manifests for these examples lies in the VolumeClaimTemplate that\u0026rsquo;s part of the statefulset definition.\nVolumeClaimTemplate\napiVersion:apps/v1kind:StatefulSetmetadata:name:mssqlspec:selector:matchLabels:app:mssqlenv:prodserviceName:mssqlreplicas:1template:metadata:labels:app:mssqlenv:prodspec:serviceAccountName:mssqlcontainers:- name:fooimage:barvolumeMounts:- name:bazmountPath:/var/opt/barenvFrom:- configMapRef:name:mssqlvolumeClaimTemplates:- metadata:name:bazlabels:env:prodspec:accessModes:[\u0026#34;ReadWriteOnce\u0026#34;]storageClassName:\u0026#34;fast\u0026#34;# StorageOS storageClass resources:requests:storage:5GiIn the StatefulSet definition above the container has a volume mount defined called baz. The definition for this volume is found in the VolumeClaimTemplate where the fast storageClass will be used to dynamically provision storage if the persistent volume does not already exist.\n","excerpt":"StorageOS can be used to provide permanent storage for other applications running in Kubernetes or …","ref":"/docs/usecases/","title":"Use Cases: Install applications on Kubernetes with StorageOS"},{"body":"","excerpt":"","ref":"/docs/reference/bundles/","title":"Bundles"},{"body":"","excerpt":"","ref":"/docs/reference/","title":"Reference"},{"body":"When you need support, please raise a ticket via our Help Desk portal. It is important to select the priority of your ticket in accordance with the severity:\n Severity 1 - Urgent Severity 2 - High Severity 3 - Medium Severity 4 - Low  This helps us route and prioritise the ticket accordingly. You can find definitions of our support severity levels on our website.\nResponses to tickets will be cc\u0026rsquo;d via email.\nFor community support and general enquiries, please join our public Slack channel. \nInformation to include in tickets To help us provide effective support, we request that you provide as much information as possible when contacting us. The list below is a suggested starting point. Additionally, please include anything specific, such as log entries, that may help us debug your issue.\nPlatform  Cloud provider/Bare metal OS distribution and version Kernel version docker version and installation procedure (distro packages or docker install)  StorageOS  Version of StorageOS storageos get nodes storageos get volumes storageos describe volume VOL_ID # in case of issues with a specific volume  Orchestrator related (Kubernetes, OpenShift, etc)  Version and installation method Managed or self managed? kubectl -n kube-system get pod kubectl -n kube-system logs -lapp=storageos -c storageos kubectl -n kube-system get storageclass Specific for your namespaces: kubectl describe pvc PVC_NAME Specific for your namespaces: kubectl describe pod POD_NAME  Environment Changes  Details of any recent changes to your environment such as planned maintenance, node reboots, network failures, etcd outage, etc.. This can help speed up ticket triage and resolution considerably  StorageOS Support Bundle StorageOS provides the ability to generate a support bundle that aggregates cluster information. See Support Bundle for a list of what is included.\nStorageOS engineers might ask for a support bundle to be generated during support cases.\nThe information in the bundle is used only for support purposes, and will be removed once it is no longer needed. If the information is sensitive and can\u0026rsquo;t be given to StorageOS, please make sure that the support engineers have as much information about your environment as possible.\nRefer to the Support Bundle documentation page for details of how to generate a bundle.\n","excerpt":"When you need support, please raise a ticket via our Help Desk portal. It is important to select the …","ref":"/docs/support/","title":"Support"},{"body":"Our self-evaluation guide is a step by step recipe for installing and testing StorageOS. This guide is divided into three sections:\n Installation - install StorageOS with a single command Feature Testing - short walkthrough of some of our features Benchmarking - a recipe to benchmark StorageOS on your infrastructure  For more comprehensive documentation including installation advice for complex setups, operational guides, and use-cases, please see our main documentation site.\nSupport for Self Evaluations Should you have questions or require support, there are several ways to get in touch with us. The fastest way to get in touch is to join our public Slack channel. You can also get in touch via email to info@storageos.com.\nFurthermore you can fill out the form below and we will get in touch.\n   hbspt.forms.create({ portalId: \"3402546\", formId: \"a07fecd3-ce5b-4835-b136-51a94a35632b\", sfdcCampaignId: \"70158000000BAZzAAO\" });   Installation In this document we detail a simple installation suitable for evaluation purposes. The etcd we install uses a 3 node cluster with ephemeral storage, and as such is not suitable for production workloads. However, for evaluation purposes it should be sufficient. For production deployments, please see our main documentation pages.\nA standard StorageOS installation uses the StorageOS operator, which performs most platform-specific configuration for you. The StorageOS operator has been certified by Red Hat and is open source.\nThe basic installation steps are:\n Check prerequisites Install an etcd cluster Install the StorageOS Operator Install StorageOS by applying a CR for the Operator to action  Prerequisites While we do not require custom kernel modules or additional userspace tooling, StorageOS does have a few basic prerequisites that are met by default by most modern distributions:\n At least 1 CPU core, 2GB RAM free TCP ports 5701-5710 and TCP \u0026amp; UDP 5711 open between all nodes in the cluster A 64bit supported operating system - StorageOS can run without additional packages in Debian 9, RancherOS, RHEL7.5,8 and CentOS7,8 and need the package linux-image-extra for Ubuntu. Mainline kernel modules target_core_mod, tcp_loop, target_core_file, target_core_user, configfs, and ui. These are present by default on most modern linux distributions, and can be installed with standard package managers. See our system configuration page for instructions.  Quick Installation For those feeling brave and/or lucky, we have encapsulated the installation instructions in this guide into a short bash script. As with similar instructions elsewhere on the internet, caution is advised here. The script is liberally commented and we recommend that you inspect it in your text editor of choice before running.\n Run the following command where kubectl is installed and with the context set for your Kubernetes cluster\n curl -sL https://storageos.run | bash Note that this script deploys a cluster with etcd on transient storage, suitable for evaluation purposes only. Do not use this cluster for production workloads.\nYour new cluster can be used straight away. You can now jump to the Provision a Volume section below to provision your first volume and license the cluster.\nInstalling StorageOS Step By Step This section describes the installation steps in some more detail.\nInstall etcd Here we install a 3 node etcd cluster using the CoreOS operator to install on ephemeral storage within your Kubernetes cluster. This is suitable for evaluation purposes. For production deployment instructions, please see our production etcd page.\n  Download repo\n$ git clone https://github.com/coreos/etcd-operator.git   Configure NS, Role and RoleBinding\n$ export ROLE_NAME=etcd-operator $ export ROLE_BINDING_NAME=etcd-operator $ export NAMESPACE=etcd   Create Namespace\n$ kubectl create namespace $NAMESPACE   Deploy Operator\n$ ./etcd-operator/example/rbac/create_role.sh $ kubectl -n $NAMESPACE create -f - \u0026lt;\u0026lt;END apiVersion: apps/v1 kind: Deployment metadata: name: etcd-operator spec: selector: matchLabels: app: etcd-operator replicas: 1 template: metadata: labels: app: etcd-operator spec: containers: - name: etcd-operator image: quay.io/coreos/etcd-operator:v0.9.4 command: - etcd-operator env: - name: MY_POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace - name: MY_POD_NAME valueFrom: fieldRef: fieldPath: metadata.name END   Create the EtcdCluster resource\n$ kubectl -n etcd create -f - \u0026lt;\u0026lt;YAML apiVersion: \u0026#34;etcd.database.coreos.com/v1beta2\u0026#34; kind: \u0026#34;EtcdCluster\u0026#34; metadata: name: \u0026#34;storageos-etcd\u0026#34; spec: size: 3 # Set to 1 if insufficient nodes version: \u0026#34;3.4.7\u0026#34; pod: etcdEnv: - name: ETCD_QUOTA_BACKEND_BYTES value: \u0026#34;2147483648\u0026#34; # 2 GB - name: ETCD_AUTO_COMPACTION_RETENTION value: \u0026#34;1000\u0026#34; # Keep 1000 revisions - name: ETCD_AUTO_COMPACTION_MODE value: \u0026#34;revision\u0026#34; # Set the revision mode resources: requests: cpu: 200m memory: 300Mi securityContext: runAsNonRoot: true runAsUser: 9000 fsGroup: 9000 tolerations: - operator: \u0026#34;Exists\u0026#34; # add tolerations to match all taints on the master nodes where etcd will be deployed affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchExpressions: - key: etcd_cluster operator: In values: - storageos-etcd topologyKey: kubernetes.io/hostname YAML   This creates a 3 node etcd cluster and associated service in Kubernetes. We can use the service endpoint when creating the StorageOS cluster.\nInstall StorageOS Operator In order to install the StorageOS operator download the requisite yaml manifests or apply them with kubectl.\n$ kubectl create -f https://github.com/storageos/cluster-operator/releases/download/v2.3.1/storageos-operator.yaml You can verify the operator is running using the following command\n$ kubectl get pods -n storageos-operator  Install StorageOS Once the StorageOS operator has been installed a StorageOS cluster can be generated by creating a StorageOSCluster resource.\nA StorageOSCluster resource describes the state of the StorageOS cluster that is desired and the StorageOS operator will create the desired StorageOS cluster. For examples of StorageOSCluster resources please see our examples page here. For a full list of the configurable spec parameters of the StorageOSCluster resource please see here.\n  Create a secret defining the API username and password\n$kubectlcreate-f- \u0026lt;\u0026lt;ENDapiVersion:v1kind:Secretmetadata:name:\u0026#34;storageos-api\u0026#34;namespace:\u0026#34;storageos-operator\u0026#34;labels:app:\u0026#34;storageos\u0026#34;type:\u0026#34;kubernetes.io/storageos\u0026#34;data:# echo -n \u0026#39;\u0026lt;secret\u0026gt;\u0026#39; | base64apiUsername:c3RvcmFnZW9zapiPassword:c3RvcmFnZW9z# CSI CredentialscsiProvisionUsername:c3RvcmFnZW9zcsiProvisionPassword:c3RvcmFnZW9zcsiControllerPublishUsername:c3RvcmFnZW9zcsiControllerPublishPassword:c3RvcmFnZW9zcsiNodePublishUsername:c3RvcmFnZW9zcsiNodePublishPassword:c3RvcmFnZW9zcsiControllerExpandUsername:c3RvcmFnZW9zcsiControllerExpandPassword:c3RvcmFnZW9zEND  Create a StorageOSCluster resource\n$kubectlcreate-f- \u0026lt;\u0026lt;ENDapiVersion:storageos.com/v1kind:StorageOSClustermetadata:name:example-storageosclusternamespace:\u0026#34;storageos-operator\u0026#34;spec:# StorageOS Pods are in kube-system by defaultsecretRefName:\u0026#34;storageos-api\u0026#34;secretRefNamespace:\u0026#34;storageos-operator\u0026#34;k8sDistro:\u0026#34;upstream\u0026#34;# Set the Kubernetes distribution for your cluster (upstream, eks, aks, gke, rancher, dockeree)# storageClassName: fast # The storage class creates by the StorageOS operator is configurablekvBackend:address:\u0026#34;storageos-etcd-client.etcd.svc.cluster.local:2379\u0026#34;END  Confirm that the cluster has been created and that StorageOS pods are running\n$ kubectl -n kube-system get pods StorageOS pods enter a ready state after a minimum of 45s has passed.\n  Deploy the StorageOS CLI as a container\n$ kubectl -n kube-system run \\ --image storageos/cli:v2.3.0 \\ --restart=Never \\ --env STORAGEOS_ENDPOINTS=storageos:5705 \\ --env STORAGEOS_USERNAME=storageos \\ --env STORAGEOS_PASSWORD=storageos \\ --command cli \\ -- /bin/sh -c \u0026#34;while true; do sleep 999999; done\u0026#34;   Provision a StorageOS Volume Now that we have a working StorageOS cluster, we can provision a volume to test everything is working as expected.\n  Create a PVC\n$ kubectl create -f - \u0026lt;\u0026lt;END apiVersion: v1 kind: PersistentVolumeClaim metadata: name: pvc-1 spec: storageClassName: \u0026#34;fast\u0026#34; accessModes: - ReadWriteOnce resources: requests: storage: 5Gi END   Verify that the volume was created with the CLI\n pvc-1 should be listed in the CLI output\n $ kubectl -n kube-system exec -it cli -- storageos get volumes   Create a pod that consumes the PVC\n$ kubectl create -f - \u0026lt;\u0026lt;END apiVersion: v1 kind: Pod metadata: name: d1 spec: containers: - name: debian image: debian:9-slim command: [\u0026#34;/bin/sleep\u0026#34;] args: [ \u0026#34;3600\u0026#34; ] volumeMounts: - mountPath: /mnt name: v1 volumes: - name: v1 persistentVolumeClaim: claimName: pvc-1 END   Check that the pod starts successfully. If the pod starts successfully then the StorageOS cluster is working correctly\n$ kubectl get pod d1 -w The pod mounts a StorageOS volume under /mnt so any files written there will persist beyond the lifetime of the pod. This can be demonstrated using the following commands.\n  Execute a shell inside the pod and write some data to a file\n$ kubectl exec -it d1 -- bash root@d1:/# echo Hello World! \u0026gt; /mnt/hello root@d1:/# cat /mnt/hello  Hello World! should be printed out.\n   Delete the pod\n$ kubectl delete pod d1   Recreate the pod\n$ kubectl create -f - \u0026lt;\u0026lt;END apiVersion: v1 kind: Pod metadata: name: d1 spec: containers: - name: debian image: debian:9-slim command: [\u0026#34;/bin/sleep\u0026#34;] args: [ \u0026#34;3600\u0026#34; ] volumeMounts: - mountPath: /mnt name: v1 volumes: - name: v1 persistentVolumeClaim: claimName: pvc-1 END   Open a shell inside the pod and check the contents of /mnt/hello\n$ kubectl exec -it d1 -- cat /mnt/hello  Hello World! should be printed out.\n   License cluster Newly installed StorageOS clusters must be licensed within 24 hours. Our developer license is free, and supports up to 5TiB of provisioned storage.\nYou will need access to the StorageOS API on port 5705 of any of your nodes. For convenience, it is often easiest to port forward the service using the following kubectl incantation (this will block, so a second terminal window may be advisable):\n$ kubectl port-forward -n kube-system svc/storageos 5705 Now follow the instructions on our operations licensing page to obtain and apply a license.\nInstallation of StorageOS is now complete.\nStorageOS Features Now that you have a fully functional StorageOS cluster we will explain some of our features that may be of use to you as you complete application and synthetic benchmarks.\nStorageOS features are all enabled/disabled by applying labels to volumes. These labels can be passed to StorageOS via persistent volume claims (PVCs) or can be applied to volumes using the StorageOS CLI or GUI.\nThe following is not an exhaustive feature list but outlines features which are commonly of use during a self-evaluation.\nVolume Replication StorageOS enables synchronous replication of volumes using the storageos.com/replicas label.\nThe volume that is active is referred to as the master volume. The master volume and its replicas are always placed on separate nodes. In fact if a replica cannot be placed on a node without a replica of the same volume, the volume will fail to be created. For example, in a three node StorageOS cluster a volume with 3 replicas cannot be created as the third replica cannot be placed on a node that doesn\u0026rsquo;t already contain a replica of the same volume.\nSee our replication documentation for more information on volume replication.\n  To test volume replication create the following PersistentVolumeClaim\n$ kubectl create -f - \u0026lt;\u0026lt;END apiVersion: v1 kind: PersistentVolumeClaim metadata: name: pvc-replicated labels: storageos.com/replicas: \u0026#34;1\u0026#34; spec: storageClassName: \u0026#34;fast\u0026#34; accessModes: - ReadWriteOnce resources: requests: storage: 5Gi END  Note that volume replication is enabled by setting the storageos.com/replicas label on the volume.\n   Confirm that a replicated volume has been created by using the StorageOS CLI or UI\n$ kubectl -n kube-system exec -it cli -- storageos get volumes   Create a pod that uses the PVC\n$ kubectl create -f - \u0026lt;\u0026lt;END apiVersion: v1 kind: Pod metadata: name: replicated-pod spec: containers: - name: debian image: debian:9-slim command: [\u0026#34;/bin/sleep\u0026#34;] args: [ \u0026#34;3600\u0026#34; ] volumeMounts: - mountPath: /mnt name: v1 volumes: - name: v1 persistentVolumeClaim: claimName: pvc-replicated END   Write data to the volume\n$ kubectl exec -it replicated-pod -- bash root@replicated-pod:/# echo Hello World! \u0026gt; /mnt/hello root@replicated-pod:/# cat /mnt/hello  Hello World! should be printed out.\n   Find the location of the master volume and shutdown the node\nShutting down a node causes all volumes, with online replicas, on the node to be evicted. For replicated volumes this immediately promotes a replica to become the new master.\n$ kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE pvc-replicated Bound pvc-29e2ad6e-8c4e-11e9-8356-027bfbbece86 5Gi RWO fast 1m $ kubectl exec -it -n kube-system cli -- storageos get volumes NAMESPACE NAME SIZE LOCATION ATTACHED ON REPLICAS AGE default pvc-4e796a62-0271-45f9-9908-21d58789a3fe 5.0 GiB kind-worker (online) kind-worker2 1/1 26 seconds ago   Check the location of the master volume and notice that it is on a new node. If the pod that mounted the volume was located on the same node that was shutdown then the pod will need to be recreated.\n$ kubectl exec -it -n kube-system cli -- storageos get volumes NAMESPACE NAME SIZE LOCATION ATTACHED ON REPLICAS AGE default pvc-4e796a62-0271-45f9-9908-21d58789a3fe 5.0 GiB kind-worker2 (online) kind-worker2 1/1 46 seconds ago   Check that the data is still accessible to the pod\n$ kubectl exec -it replicated-pod -- bash root@replicated-pod:/# cat /mnt/hello  Hello World! should be printed out.\n    Benchmarking Benchmarking storage is a complex topic. Considering the many books and papers that have been written about benchmarking, we could write many paragraphs here and only begin to scratch the surface.\nTaking this complexity into account we present recipes for both synthetic benchmarks using FIO, and a sample application benchmark to test PostgreSQL using pgbench. The approaches are complementary - synthetic benchmarks allow us to strictly control the parameters of the IO we put through the system in order to stress various aspects of it. Application benchmarks allow us to get a sense of the performance of the system when running an actual representative workload - which of course is the ultimate arbiter of performance.\nDespite the inherent complexity of benchmarking storage there are a few general considerations to keep in mind.\nConsiderations Local vs. Remote Volumes When a workload is placed on the same node as a volume, there is no network round trip for IO, and performance is consequently improved. When considering the performance of StorageOS it is important to evaluate both local and remote volumes; since for certain workloads we want the high performance of a local attachment, but we also desire the flexibility of knowing that our less performance-sensitive workloads can run from any node in the cluster.\nThe Effect of Synchronous Replication Synchronous replication does not impact the read performance of a volume, but it can have a significant impact on the write performance of the volume, since all writes must be propagated to replicas before being acked to the application. The impact varies in proportion to the inter-node latency. For an inter-node latency of 1ms, we have a max ceiling of 1000 write IOPS, and in reality a little less than that to allow for processing time on the nodes. This is less concerning then it may first appear, since many applications will issue multiple writes in parallel (known as increasing the queue depth).\nSynthetic Benchmarks Prerequisites  Kubernetes cluster with a minimum of 3 nodes and a minimum of 30 Gb available capacity StorageOS CLI running as a pod in the cluster  Synthetic benchmarks using tools such as FIO are a useful way to begin measuring StorageOS performance. While not fully representative of application performance, they allow us to reason about the performance of storage devices without the added complexity of simulating real world workloads, and provide results easily comparable across platforms.\nFIO has a number of parameters that can be adjusted to simulate a variety of workloads and configurations. Parameters that are particularly important are:\n Block Size - the size of the IO units performed. Queue Depth - the amount of concurrent IOs in flight IO Pattern - access can be random across the disk, or sequentially. IO subsystems typically perform better with sequential IO, because of the effect of read ahead caches, and other factors  For this self-evaluation we will run a set of tests based on the excellent DBench, which distills the numerous FIO options into a series of well-crafted scenarios:\n Random Read/Write IOPS and BW - these tests measure the IOPS ceiling (with a 4k block size) and bandwidth ceiling (with a 128K block size) of the volume using a random IO pattern and high queue depth Average Latency - these tests measure the IO latency of the volume under favourable conditions using a random access pattern, low queue depth and low block size Sequential Read/Write - these tests measure the sequential read/write throughput of the volume with a high queue depth and high block size Mixed Random Read/Write IOPS - these tests measure the performance of the volume under a 60/40 read/write workload using a random access pattern and low blocksize  For convenience we present a single script to run the scenarios using local and remote volumes both with and without a replica. Deploy the FIO tests for the four scenarios using the following command:\ncurl -sL https://docs.storageos.com/sh/deploy-synthetic-benchmarks.sh | bash The script will take approximately 20 minutes to complete, and will print the results to STDOUT.\nThe exact results observed will depend on the particular platform and environment, but the following trends should be observable:\n local volumes perform faster than remote volumes read performance is independent of the number of replicas write performance is dependent on the number of replicas  Application Benchmarks While synthetic benchmarks are useful for examining the behaviour of StorageOS with very specific workloads, in order to get a realistic picture of StorageOS performance actual applications should be tested.\nMany applications come with test suites which provide standard workloads. For best results, test using your application of choice with a representative configuration and real world data.\nAs an example of benchmarking an application the following steps lay out how to benchmark a Postgres database backed by a StorageOS volume.\n  Start by cloning the StorageOS use cases repository. Note this is the same repository that we cloned earlier so if you already have a copy just cd storageos-usecases/pgbench.\n$ git clone https://github.com/storageos/use-cases.git storageos-usecases   Move into the Postgres examples folder\n$ cd storageos-usecases/pgbench   Decide which node you want the pgbench pod and volume to be located on. The node needs to be labelled app=postgres\n$ kubectl label node \u0026lt;NODE\u0026gt; app=postgres   Then set the storageos.com/hint.master label in 20-postgres-statefulset.yaml file to match the StorageOS nodeID for the node you have chosen before creating all the files. The StorageOS nodeID can be obtained using the cli and doing a describe node\n$ kubectl create -f .   Confirm that Postgres is up and running\n$ kubectl get pods -w -l app=postgres   Use the StorageOS CLI or the GUI to check the master volume location and the mount location. They should match\n$ kubectl -n kube-system exec -it cli -- storageos get volumes   Exec into the pgbench container and run pgbench\n$ kubectl exec -it pgbench -- bash -c \u0026#39;/opt/cpm/bin/start.sh\u0026#39;    Conclusion After completing these steps you will have benchmark scores for StorageOS. Please keep in mind that benchmarks are only part of the story and that there is no replacement for testing actual production or production like workloads.\nStorageOS invites you to provide feedback on your self-evaluation to the slack channel or by directly emailing us at info@storageos.com]\n","excerpt":"Our self-evaluation guide is a step by step recipe for installing and testing StorageOS. This guide …","ref":"/docs/self-eval/","title":"Self Evaluation Guide"},{"body":"","excerpt":"","ref":"/index.json","title":""},{"body":" \t window.onload = function() { const ui = SwaggerUIBundle({ url: \"/api-swagger-v2.yaml\", dom_id: '#ohpen_swagger_ui', presets: [ SwaggerUIBundle.presets.apis, SwaggerUIStandalonePreset ] }) window.ui = ui } \t","excerpt":" \t window.onload = function() { const ui = SwaggerUIBundle({ url: \"/api-swagger-v2.yaml\", dom_id: …","ref":"/docs/reference/api/","title":"API Reference"},{"body":"$ storageos apply licence --help Usage: storageos apply licence [flags] Examples: $ storageos apply licence --from-file \u0026lt;path-to-licence-file\u0026gt; $ echo \u0026#34;\u0026lt;licence file contents\u0026gt;\u0026#34; | storageos apply licence --from-stdin Flags: --from-file string reads a StorageOS product licence from a specified file path --from-stdin reads a StorageOS product licence from the standard input -h, --help help for licence Global Flags: --endpoints stringArray set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705]) -n, --namespace string specifies the namespace to operate within for commands that require one (default \u0026#34;default\u0026#34;) -o, --output string specifies the output format (one of [json yaml text]) (default \u0026#34;text\u0026#34;) --password string set the StorageOS account password to authenticate with (default \u0026#34;storageos\u0026#34;) --timeout duration set the timeout duration to use for execution of the command (default 5s) --use-ids specify existing StorageOS resources by their unique identifiers instead of by their names --username string set the StorageOS account username to authenticate as (default \u0026#34;storageos\u0026#34;) ","excerpt":"$ storageos apply licence --help Usage: storageos apply licence [flags] Examples: $ storageos apply …","ref":"/docs/reference/cli/apply/","title":"Apply licence"},{"body":"StorageOS is a software-defined storage platform for running stateful applications in Kubernetes.\nFundamentally, StorageOS uses the storage attached to the nodes in the StorageOS cluster to create and present virtual volumes into containers. Space on the host is consumed from the mount point /var/lib/storageos/data, so it is therefore recommended that disk devices are used exclusively for StorageOS, as described in Managing Host Storage \nStorageOS is agnostic to the underlying storage and runs equally well on bare metal, in virtual machines or on cloud providers.\nRead about the cloud native storage principles behind StorageOS.\nStorageOS on Kubernetes StorageOS is architected as a series of containers that fulfil separate, discrete functions.\n  StorageOS Controlplane\nResponsible for monitoring and maintaining the state of volumes and nodes in the cluster\n  StorageOS Dataplane\nResponsible for all I/O path related tasks; reading, writing, compression and caching\n  StorageOS Scheduler\nResponsible for scheduling applications on the same node as applications volume\n  CSI helper\nResponsible for registering StorageOS with Kubernetes as a CSI driver\n  StorageOS Operator\nResponsible for the creation and maintenance of the StorageOS cluster\n  StorageOS is deployed by the StorageOS operator. In Kubernetes, the StorageOS Controlplane and Dataplane are deployed in a single pod managed by a daemonset. This daemonset runs on every node in the cluster that will consume or present storage. The scheduler, CSI helper and Operator run as separate pods and are controlled as deployments.\nStorageOS is designed to feel familiar to Kubernetes and Docker users. Storage is managed through standard StorageClasses and PersistentVolumeClaims, and features are controlled by Kubernetes-style labels and selectors, prefixed with storageos.com/. By default, volumes are cached to improve read performance and compressed to reduce network traffic.\nAny pod may mount a StorageOS virtual volume from any node that is also running StorageOS, regardless of whether the pod and volume are collocated on the same node. Therefore, applications may be started or restarted on any node and access volumes transparently.\n","excerpt":"StorageOS is a software-defined storage platform for running stateful applications in Kubernetes. …","ref":"/docs/concepts/architecture/","title":"Architecture"},{"body":"$ storageos attach Attach a volume to a node Usage: storageos attach [flags] Examples: $ storageos attach --namespace my-namespace-name my-volume my-node Flags: -h, --help help for attach Global Flags: --cache-dir string set the directory used by the StorageOS CLI to cache data that can be used for future commands (default \u0026#34;/root/.cache/storageos\u0026#34;) -c, --config string specifies the config file path (default \u0026#34;/root/.config/storageos/config.yaml\u0026#34;) --endpoints stringArray set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705]) -n, --namespace string specifies the namespace to operate within for commands that require one (default \u0026#34;default\u0026#34;) --no-auth-cache disable the CLI\u0026#39;s caching of authentication sessions -o, --output string specifies the output format (one of [json yaml text]) (default \u0026#34;text\u0026#34;) --password string set the StorageOS account password to authenticate with (default \u0026#34;storageos\u0026#34;) --timeout duration set the timeout duration to use for execution of the command (default 15s) --use-ids specify existing StorageOS resources by their unique identifiers instead of by their names --username string set the StorageOS account username to authenticate as (default \u0026#34;storageos\u0026#34;) ","excerpt":"$ storageos attach Attach a volume to a node Usage: storageos attach [flags] Examples: $ storageos …","ref":"/docs/reference/cli/attach/","title":"Attach"},{"body":"StorageOS is fully compatible with AWS EKS. To install StorageOS on EKS, please follow our Kubernetes installation instructions page.\nAn EKS deployment of Kubernetes uses AWS Linux by default with an optimized kernel. As the requisite kernel modules are not available for StorageOS to use TCMU, FUSE will be used as a fallback. Using FUSE instead of TCMU has performance implications.\nFor more details about the OS Distributions check the System Configuration page.\n","excerpt":"StorageOS is fully compatible with AWS EKS. To install StorageOS on EKS, please follow our …","ref":"/docs/platforms/aws-eks/","title":"AWS EKS"},{"body":"StorageOS is fully compatible with Azure AKS. To install StorageOS on AKS please follow our Kubernetes installation instructions page.\nAn AKS deployment of Kubernetes uses Ubuntu by default with an optimized kernel. All versions of Ubuntu with a kernel version later than4.15.0-1029-azure meet the StorageOS prerequisites.\n","excerpt":"StorageOS is fully compatible with Azure AKS. To install StorageOS on AKS please follow our …","ref":"/docs/platforms/azure-aks/","title":"Azure AKS"},{"body":"Cassandra with StorageOS Cassandra is a popular distributed NoSQL open source database.\nUsing StorageOS persistent volumes with Cassandra means that if a Cassandra pod fails, the cluster is only in a degraded state for as long as it takes Kubernetes to restart the pod. When the pod comes back up the pod data is immediately available. Should Kubernetes schedule the Cassandra pod on a new node, StorageOS allows for the data to be available to the pod, irrespective of whether or not a StorageOS master is located on the same node.\nAs Cassandra has features to allow it to handle replication careful consideration of whether to allow StorageOS or Cassandra to handle replication is required.\nBefore you start, ensure you have StorageOS installed and ready on a Kubernetes cluster. See our guide on how to install StorageOS on Kubernetes for more information.\nDeploying Cassandra on Kubernetes   You can find the latest files in the StorageOS use cases repository\ngit clone https://github.com/storageos/use-cases.git storageos-usecases StatefulSet defintion\napiVersion:apps/v1kind:StatefulSetmetadata:name:cassandraspec:selector:matchLabels:app:cassandraserviceName:cassandrareplicas:3...spec:...volumeMounts:- name:cassandra-datamountPath:/var/lib/cassandra...volumeClaimTemplates:- metadata:name:cassandra-dataspec:accessModes:[\u0026#34;ReadWriteOnce\u0026#34;]storageClassName:\u0026#34;fast\u0026#34;# StorageOS storageClassresources:requests:storage:5GiThis excerpt is from the StatefulSet definition. This file contains the VolumeClaim template that will dynamically provision storage, using the StorageOS storage class. Dynamic provisioning occurs as a volumeMount has been declared with the same name as a VolumeClaimTemplate.\n  Move into the Cassandra examples folder and create the objects\ncd storageos-usecases kubectl create -f ./cassandra   Confirm Cassandra is up and running.\n$ kubectl get pods -w -l app=cassandra NAME READY STATUS RESTARTS AGE cassandra-0 1/1 Running 0 8m32s cassandra-1 1/1 Running 0 7m51s cassandra-2 1/1 Running 0 6m36s   Connect to the Cassandra client pod and connect to the Cassandra server through the service\n$ kubectl exec -it cassandra-0 -- cqlsh cassandra-0.cassandra Connected to K8Demo at cassandra-0.cassandra:9042. [cqlsh 5.0.1 | Cassandra 3.11.3 | CQL spec 3.4.4 | Native protocol v4] Use HELP for help. cqlsh\u0026gt; SELECT cluster_name, listen_address FROM system.local; cluster_name | listen_address --------------+---------------- K8Demo | 100.96.7.124 (1 rows)   ","excerpt":"Cassandra with StorageOS Cassandra is a popular distributed NoSQL open source database.\nUsing …","ref":"/docs/usecases/cassandra/","title":""},{"body":"The storageos command line interface (CLI) is used to manage cluster-wide configuration.\nInstallation # linux/amd64 curl -sSLo storageos https://github.com/storageos/go-cli/releases/download/v2.3.0/storageos_linux_amd64 \u0026amp;\u0026amp; chmod +x storageos \u0026amp;\u0026amp; sudo mv storageos /usr/local/bin/ # MacOS curl -sSLo storageos https://github.com/storageos/go-cli/releases/download/v2.3.0/storageos_darwin_amd64 \u0026amp;\u0026amp; chmod +x storageos \u0026amp;\u0026amp; sudo mv storageos /usr/local/bin/ You will need to provide the correct credentials to connect to the API. The default installation uses the storageos-api Secret to generate the first admin user. By default, it creates a single user with username storageos and password storageos:\nexport STORAGEOS_USERNAME=storageos export STORAGEOS_PASSWORD=storageos export STORAGEOS_ENDPOINTS=10.1.5.249:5705 Run CLI as a container You can run the cli as a container in your Kubernetes cluster. Then exec into it to run commands.\nkubectl -n kube-system run \\  --image storageos/cli:v2.3.0 \\  --restart=Never \\  --env STORAGEOS_ENDPOINTS=storageos:5705 \\  --env STORAGEOS_USERNAME=storageos \\  --env STORAGEOS_PASSWORD=storageos \\  --command cli \\  -- /bin/sh -c \u0026#34;while true; do sleep 100000; done\u0026#34; Usage $ storageos Storage for Cloud Native Applications. By using this product, you are agreeing to the terms of the the StorageOS Ltd. End User Subscription Agreement (EUSA) found at: https://storageos.com/legal/#eusa To be notified about stable releases and latest features, sign up at https://my.storageos.com. Usage: storageos [command] Available Commands: apply Make changes to existing resources attach Attach a volume to a node create Create new resources delete Delete resources in the cluster describe Fetch extended details for resources detach Detach a volume from its current location get Fetch basic details for resources help Help about any command version View version information for the StorageOS CLI Flags: --endpoints stringArray set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705]) -h, --help help for storageos -n, --namespace string specifies the namespace to operate within for commands that require one (default \u0026#34;default\u0026#34;) -o, --output string specifies the output format (one of [json yaml text]) (default \u0026#34;text\u0026#34;) --password string set the StorageOS account password to authenticate with (default \u0026#34;storageos\u0026#34;) --timeout duration set the timeout duration to use for execution of the command (default 5s) --use-ids specify existing StorageOS resources by their unique identifiers instead of by their names --username string set the StorageOS account username to authenticate as (default \u0026#34;storageos\u0026#34;) Additional help topics: storageos env View documentation for configuration settings which can be set in the environment storageos exitcodes View documentation for the exit codes used by the StorageOS CLI Formatting CLI Output StorageOS CLI output can be formatted using the --output option. The strings that are passed to --output are \u0026lsquo;json\u0026rsquo;, \u0026lsquo;yaml\u0026rsquo; or \u0026lsquo;text\u0026rsquo;.\nCheatsheet    Command Subcommand Description     apply licence Apply licence to cluster.   attach  Attach Volume to a node.   create user, volume Create resources such as users or volumes.   delete volume Delete resources.   describe node, volume Show detailed view of resources.   detach volume Detach volume from a node.   get cluster, diagnostics, namespace, node, user, volume List resources.   help  Help   version  Show CLI version.    Source is available on Github.\n","excerpt":"The storageos command line interface (CLI) is used to manage cluster-wide configuration. …","ref":"/docs/reference/cli/","title":"Command line interface"},{"body":"Our cluster operator is a Kubernetes native application developed to deploy and configure StorageOS clusters, and assist with maintenance operations. We recommend its use for standard installations.\nThe operator acts as a Kubernetes controller that watches the StorageOSCluster CR (Custom Resource). Once the controller is ready, a StorageOS cluster definition can be created. The operator will deploy a StorageOS cluster based on the configuration specified in the cluster definition.\nYou can find the source code in the cluster-operator repository.\nInstall the operator following orchestrator specific procedure.\nTo deploy a StorageOS cluster you will need to fulfil the following steps.\n Install Etcd Install operator Create storageos-api Secret Create a CR (Custom Resource) to deploy StorageOS  ","excerpt":"Our cluster operator is a Kubernetes native application developed to deploy and configure StorageOS …","ref":"/docs/reference/cluster-operator/","title":"Cluster Operator"},{"body":"Every StorageOS cluster has a unique ClusterID generated at bootstrap. A StorageOS Licence is specific for a ClusterID.\nHow to obtain the ClusterID You can obtain the ClusterID using either the CLI or GUI.\nGUI You will need access to the StorageOS GUI on port 5705 of any of your nodes. For convenience, it is often easiest to port forward the service using the following kubectl incantation (this will block, so a second terminal window may be advisable):\n$ kubectl port-forward -n kube-system svc/storageos 5705 As an alternative, an Ingress controller may be preferred.\nOnce you have obtained access to the GUI, login using whatever credentials you used to create the cluster and go to the \u0026ldquo;Licence\u0026rdquo; section on the left navigation menu.\nCLI This CLI command can print the cluster ID:\n$ storageos get cluster ID: 704dd165-9580-4da4-a554-0acb96d328cb Licence: expiration: 2021-03-25T13:48:46Z (1 year from now) capacity: 5.0 TiB kind: professional customer name: storageos Created at: 2020-03-25T13:48:33Z (1 hour ago) Updated at: 2020-03-25T13:48:46Z (1 hour ago) Given the Cluster ID, the StorageOS team can generate a licence.\n","excerpt":"Every StorageOS cluster has a unique ClusterID generated at bootstrap. A StorageOS Licence is …","ref":"/docs/operations/cluster-id/","title":"Find ClusterID"},{"body":"StorageOS clusters represent groups of nodes which run a common distributed control plane.\nTypically, a StorageOS cluster maps one-to-one to a Kubernetes (or similar orchestrator) cluster, and we expect our daemonset to run on all worker nodes within the cluster that will consume or present storage.\nClusters use etcd to maintain state and manage distributed consensus between nodes.\n","excerpt":"StorageOS clusters represent groups of nodes which run a common distributed control plane. …","ref":"/docs/concepts/clusters/","title":"Clusters"},{"body":"StorageOS compression is handled on a per volume basis and is disabled by default in v2.2+, as performance is generally increased when compression is disabled due to block alignment. This means that there is a trade off between volume performance and the space the volume occupies on the backend device.\nCompression can be enabled by setting the label storageos.com/nocompress=false on a volume at volume creation time.\nStorageOS utilises the lz4 compression algorithm when writing to the backend store and when compressing replication traffic before it is sent across the network.\nStorageOS detects whether a block can be compressed or not by creating a heuristic that predicts the size of a compressed block. If the heuristic indicates that the compressed block is likely to be larger than the original block then the uncompressed block is stored. Block size increases post compression if the compression dictionary is added to a block that cannot be compressed. By verifying whether blocks can be compressed, disk efficiency is increased and CPU resources are not wasted on attempts to compress uncompressible blocks. StorageOS\u0026rsquo; patented on disk format is used to tell whether individual blocks are compressed without overhead. As such volume compression can be dynamically enabled/disabled even while a volume is in use.\nWhen compression and encryption are both enabled for a volume, blocks are compressed then encrypted.\n","excerpt":"StorageOS compression is handled on a per volume basis and is disabled by default in v2.2+, as …","ref":"/docs/concepts/compression/","title":"Compression"},{"body":"We are always looking to improve our documentation. If you like to help people and can write, read on for the process for submitting your contributions. If your guide is published, you\u0026rsquo;ll receive $250 per article by PayPal.\nContent guidelines A guide contains step by step instructions for how to accomplish a specific task using StorageOS. To be accepted, guides must be:\n Written in English. Relevant, accurate and complete. Technically correct and thoroughly tested. Follow the writing style guide.  Guides should avoid:\n Duplicating an existing guide or other sources, such as blogs or forum posts. Including irrelevant material.  Submission and review You should submit your guide as a pull request to the GitHub repo.\nYour submission will be left open for community review for two weeks. After this, your submission will be reviewed internally for about another week.\nIf accepted, your pull request will be approved and you will have 36 hours to send your submission title and PayPal account information. Non-response will be taken as a go-ahead to publish.\nLegal COPYRIGHT OWNERSHIP. The StorageOS Guides \u0026amp; Tutorials repository is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0) license.\nCREDIT. Nothing contained in this Agreement shall be deeded to require StorageOS to use the Work, or any part thereof, in connection with StorageOS Guides \u0026amp; Tutorials or otherwise. Credit for the Work shall read, “Contributed by writer’s name.”\nPAYMENT. Upon publication of a submission to the StorageOS Guides \u0026amp; Tutorials Repository, the writer will be paid the sum of USD $250.00 as an electronic payment.\n","excerpt":"We are always looking to improve our documentation. If you like to help people and can write, read …","ref":"/docs/reference/contributing/","title":"Contributing to the docs"},{"body":"$ storageos create --help Create new resources Usage: storageos create [command] Available Commands: user Create a new user account volume Provision a new volume Flags: -h, --help help for create Global Flags: --endpoints stringArray set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705]) -n, --namespace string specifies the namespace to operate within for commands that require one (default \u0026#34;default\u0026#34;) -o, --output string specifies the output format (one of [json yaml text]) (default \u0026#34;text\u0026#34;) --password string set the StorageOS account password to authenticate with (default \u0026#34;storageos\u0026#34;) --timeout duration set the timeout duration to use for execution of the command (default 5s) --use-ids specify existing StorageOS resources by their unique identifiers instead of by their names --username string set the StorageOS account username to authenticate as (default \u0026#34;storageos\u0026#34;) Use \u0026#34;storageos create [command] --help\u0026#34; for more information about a command. create volume $ storageos create volume --help Provision a new volume Usage: storageos create volume [flags] Examples: $ storageos create volume --description \u0026#34;This volume contains the data for my app\u0026#34; --fs-type \u0026#34;ext4\u0026#34; --labels env=prod,rack=db-1 --size 10GiB --namespace my-namespace-name my-app $ storageos create volume --replicas 1 --namespace my-namespace-name my-replicated-app Flags: --async perform the operation asynchronously, using the configured timeout duration --cache caches volume data (default true) --compress compress data stored by the volume at rest and during transit (default true) -d, --description string a human-friendly description to give the volume -f, --fs-type string the filesystem to format the new volume with once provisioned (default \u0026#34;ext4\u0026#34;) -h, --help help for volume -l, --labels strings an optional set of labels to assign to the new volume, provided as a comma-separated list of key=value pairs -r, --replicas uint the number of replicated copies of the volume to maintain -s, --size string the capacity to provision the volume with (default \u0026#34;5GiB\u0026#34;) --throttle deprioritises the volume\u0026#39;s traffic by reducing the rate of disk I/O Global Flags: --endpoints stringArray set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705]) -n, --namespace string specifies the namespace to operate within for commands that require one (default \u0026#34;default\u0026#34;) -o, --output string specifies the output format (one of [json yaml text]) (default \u0026#34;text\u0026#34;) --password string set the StorageOS account password to authenticate with (default \u0026#34;storageos\u0026#34;) --timeout duration set the timeout duration to use for execution of the command (default 5s) --use-ids specify existing StorageOS resources by their unique identifiers instead of by their names --username string set the StorageOS account username to authenticate as (default \u0026#34;storageos\u0026#34;) create user $ storageos create user --help Create a new user account Usage: storageos create user [flags] Examples: $ storageos create user --with-username=alice --with-admin=true Flags: -h, --help help for user --with-admin control whether the user is given administrative priviledges --with-groups stringArray the list of policy groups to assign to the user --with-password string the password to assign to the user. If not specified, this will be prompted for. --with-username string the username to assign Global Flags: --endpoints stringArray set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705]) -n, --namespace string specifies the namespace to operate within for commands that require one (default \u0026#34;default\u0026#34;) -o, --output string specifies the output format (one of [json yaml text]) (default \u0026#34;text\u0026#34;) --password string set the StorageOS account password to authenticate with (default \u0026#34;storageos\u0026#34;) --timeout duration set the timeout duration to use for execution of the command (default 5s) --use-ids specify existing StorageOS resources by their unique identifiers instead of by their names --username string set the StorageOS account username to authenticate as (default \u0026#34;storageos\u0026#34;) ","excerpt":"$ storageos create --help Create new resources Usage: storageos create [command] Available Commands: …","ref":"/docs/reference/cli/create/","title":"Create"},{"body":"$ storageos delete -h Delete resources in the cluster Usage: storageos delete [command] Available Commands: volume Delete a volume Flags: -h, --help help for delete Global Flags: --endpoints stringArray set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705]) -n, --namespace string specifies the namespace to operate within for commands that require one (default \u0026#34;default\u0026#34;) -o, --output string specifies the output format (one of [json yaml text]) (default \u0026#34;text\u0026#34;) --password string set the StorageOS account password to authenticate with (default \u0026#34;storageos\u0026#34;) --timeout duration set the timeout duration to use for execution of the command (default 5s) --use-ids specify existing StorageOS resources by their unique identifiers instead of by their names --username string set the StorageOS account username to authenticate as (default \u0026#34;storageos\u0026#34;) Use \u0026#34;storageos delete [command] --help\u0026#34; for more information about a command. delete volume $ storageos delete volume --help Delete a volume Usage: storageos delete volume [volume name] [flags] Examples: $ storageos delete volume my-test-volume my-unneeded-volume $ storagoes delete volume --namespace my-namespace my-old-volume Flags: --async perform the operation asynchronously, using the configured timeout duration --cas string make changes to a resource conditional upon matching a provided version -h, --help help for volume Global Flags: --endpoints stringArray set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705]) -n, --namespace string specifies the namespace to operate within for commands that require one (default \u0026#34;default\u0026#34;) -o, --output string specifies the output format (one of [json yaml text]) (default \u0026#34;text\u0026#34;) --password string set the StorageOS account password to authenticate with (default \u0026#34;storageos\u0026#34;) --timeout duration set the timeout duration to use for execution of the command (default 5s) --use-ids specify existing StorageOS resources by their unique identifiers instead of by their names --username string set the StorageOS account username to authenticate as (default \u0026#34;storageos\u0026#34;) ","excerpt":"$ storageos delete -h Delete resources in the cluster Usage: storageos delete [command] Available …","ref":"/docs/reference/cli/delete/","title":"Delete"},{"body":"$ storageos describe --help Fetch extended details for resources Usage: storageos describe [command] Available Commands: node Retrieve detailed information for nodes in the cluster volume Show detailed information for volumes Flags: -h, --help help for describe Global Flags: --endpoints stringArray set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705]) -n, --namespace string specifies the namespace to operate within for commands that require one (default \u0026#34;default\u0026#34;) -o, --output string specifies the output format (one of [json yaml text]) (default \u0026#34;text\u0026#34;) --password string set the StorageOS account password to authenticate with (default \u0026#34;storageos\u0026#34;) --timeout duration set the timeout duration to use for execution of the command (default 5s) --use-ids specify existing StorageOS resources by their unique identifiers instead of by their names --username string set the StorageOS account username to authenticate as (default \u0026#34;storageos\u0026#34;) Use \u0026#34;storageos describe [command] --help\u0026#34; for more information about a command. describe node $ storageos describe node --help Retrieve detailed information for nodes in the cluster Usage: storageos describe node [node names...] [flags] Aliases: node, nodes Examples: $ storageos describe node my-node-name Flags: -h, --help help for node -l, --selector stringArray filter returned results by a set of comma-separated label selectors Global Flags: --endpoints stringArray set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705]) -n, --namespace string specifies the namespace to operate within for commands that require one (default \u0026#34;default\u0026#34;) -o, --output string specifies the output format (one of [json yaml text]) (default \u0026#34;text\u0026#34;) --password string set the StorageOS account password to authenticate with (default \u0026#34;storageos\u0026#34;) --timeout duration set the timeout duration to use for execution of the command (default 5s) --use-ids specify existing StorageOS resources by their unique identifiers instead of by their names --username string set the StorageOS account username to authenticate as (default \u0026#34;storageos\u0026#34;) describe volume $ storageos describe volume --help Show detailed information for volumes Usage: storageos describe volume [volume names...] [flags] Aliases: volume, volumes Examples: $ storageos describe volumes $ storageos describe volume --namespace my-namespace-name my-volume-name Flags: -h, --help help for volume -l, --selector stringArray filter returned results by a set of comma-separated label selectors Global Flags: --endpoints stringArray set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705]) -n, --namespace string specifies the namespace to operate within for commands that require one (default \u0026#34;default\u0026#34;) -o, --output string specifies the output format (one of [json yaml text]) (default \u0026#34;text\u0026#34;) --password string set the StorageOS account password to authenticate with (default \u0026#34;storageos\u0026#34;) --timeout duration set the timeout duration to use for execution of the command (default 5s) --use-ids specify existing StorageOS resources by their unique identifiers instead of by their names --username string set the StorageOS account username to authenticate as (default \u0026#34;storageos\u0026#34;) ","excerpt":"$ storageos describe --help Fetch extended details for resources Usage: storageos describe [command] …","ref":"/docs/reference/cli/describe/","title":"Describe"},{"body":"$ storageos detach --help Detach a volume from its current location Usage: storageos detach [flags] Examples: $ storageos detach volume --namespace my-namespace-name my-volume Flags: --cas string make changes to a resource conditional upon matching a provided version -h, --help help for detach Global Flags: --endpoints stringArray set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705]) -n, --namespace string specifies the namespace to operate within for commands that require one (default \u0026#34;default\u0026#34;) -o, --output string specifies the output format (one of [json yaml text]) (default \u0026#34;text\u0026#34;) --password string set the StorageOS account password to authenticate with (default \u0026#34;storageos\u0026#34;) --timeout duration set the timeout duration to use for execution of the command (default 5s) --use-ids specify existing StorageOS resources by their unique identifiers instead of by their names --username string set the StorageOS account username to authenticate as (default \u0026#34;storageos\u0026#34;) ","excerpt":"$ storageos detach --help Detach a volume from its current location Usage: storageos detach [flags] …","ref":"/docs/reference/cli/detach/","title":"Detach Volume"},{"body":"StorageOS can generate a cluster diagnostic bundle from the GUI or CLI under user request. The bundle packages the information needed for the engineering team to understand the context of your cluster and begin troubleshooting any issues you may be experiencing.\nGenerate Bundle CLI The StorageOS CLI can generate the bundle.\n$ storageos get diagnostics  The file generated is in the form of diagnostics-${TIME_STAMP}.gz\n GUI Or you can use the StoregeOS GUI.\n Go to section \u0026ldquo;Cluster\u0026rdquo; Press the button \u0026ldquo;DOWNLOAD DIAGNOSTICS\u0026rdquo;.  Data collected in the bundle Most of the data collected in the bundle is regarding the state of the StorageOS cluster, however some other information regarding the infrastructure is also gathered. The information is used to have a clear view of the cluster where StorageOS is running.\nThe bundle incorporates for each node:\n StorageOS Daemonset Pod logs lshw dmesg (kernel logs) StorageOS metadata for the ControlPlane and DataPlane  StorageOS metadata collected  cluster metadata namespaces metadata nodes metadata volumes metadata capacity stats environment variables health  Privacy StorageOS can only obtain the bundle if it is downloaded by the user and given to our engineering team, or uploaded for analysis. The data received by StorageOS is private and never leaves nor will leave StorageOS Inc.\nThe data contained in the cluster diagnostic bundle has the sole purpose of helping customers troubleshoot their issues.\n","excerpt":"StorageOS can generate a cluster diagnostic bundle from the GUI or CLI under user request. The …","ref":"/docs/reference/bundles/diagnostic-bundle/","title":"Diagnostic bundle"},{"body":"To install StorageOS on DockerEE, please follow our Kubernetes installation instructions page.\nDocker EE and the Universal Control Plane can run on different Linux distributions. StorageOS supports RHEL, CentOS, Debian, and selected Ubuntu images. For more details, check out the supported OSs in the prerequisites page.\nStorageOS only supports Kubernetes nodes managed by Docker Enterprise Edition, not those nodes running Swarm. Mixed nodes (those running Kubernetes and Swarm workloads) are not supported. As a consequence, StorageOS volumes can only be provisioned on Kubernetes nodes, and only these nodes should be used for stateful workloads.\n","excerpt":"To install StorageOS on DockerEE, please follow our Kubernetes installation instructions page. …","ref":"/docs/platforms/dockeree/","title":"Docker Enterprise Edition"},{"body":"","excerpt":"","ref":"/docs/","title":"Documentation"},{"body":"ElasticSearch with StorageOS  Elasticsearch is a distributed, RESTful search and analytics engine, most popularly used to aggregate logs, but also to serve as a search backend to a number of different applications.\nUsing StorageOS persistent volumes with ElasticSearch (ES) means that if a pod fails, the cluster is only in a degraded state for as long as it takes Kubernetes to restart the pod. When the pod comes back up, the pod data is immediately available. Should Kubernetes schedule the Elasticsearch pod on a new node, StorageOS allows for the data to be available to the pod, irrespective of whether or not the original StorageOS master volume is located on the same node.\nElasticsearch has features to allow it to handle data replication, and as such careful consideration of whether to allow StorageOS or Elasticsearch to handle replication is required.\nBefore you start, ensure you have StorageOS installed and ready on a Kubernetes cluster. See our guide on how to install StorageOS on Kubernetes for more information.\nDeploying Elasticsearch on Kubernetes Prerequisites Some OS tuning is required, which is done automatically when using our example from the use cases repository.\nElasticsearch requires vm.max_map_count to be increased to a minimum of 262144, which is a system wide setting. One way to achieve this is to run sysctl -w vm.max_map_count=262144 and update /etc/sysctl.conf to ensure it persists over a reboot. See ElasicSearch reference here.\nAdministrators should be aware that this impacts the behaviour of nodes and that there may be collisions with other application settings. Administrators are advised to centrally collate sysctl settings using the tooling of their choice.\nDeployment of the application StatefulSet defintion ---apiVersion:apps/v1kind:StatefulSetmetadata:name:esdata[...]spec:serviceAccountName:elasticsearchcontainers:- name:dataimage:elasticsearch:6.7.0imagePullPolicy:IfNotPresent[...]volumeMounts:- name:datamountPath:/usr/share/elasticsearch/data/data[...]volumeClaimTemplates:- metadata:name:\u0026#34;data\u0026#34;spec:accessModes:[\u0026#34;ReadWriteOnce\u0026#34;]storageClassName:\u0026#34;fast\u0026#34;# \u0026lt;--- default StorageOS storage class nameresources:requests:storage:10Gi# \u0026lt;--- change this to the appropriate valueThis excerpt is from the StatefulSet definition (/elasticsearch/10-es-data.yaml). The file contains the PersistentVolumeClaim template that will dynamically provision the necessary storage, using the StorageOS storage class.\nDynamic provisioning occurs as a volumeMount has been declared with the same name as a VolumeClaimTemplate.\nInstallation Clone the use cases repo You can find the latest files in the StorageOS use cases repostiory in /elasticsearch/\ngit clone https://github.com/storageos/use-cases.git storageos-usecases cd storageos-usecases   Create the kubernetes objects\nThis will install an ES cluster with 3 master, 3 data and 3 coordinator nodes. Combined they will require ~ 14 GiB of available memory in your cluster, however, more may be used as the application is being used\nkubectl apply -f ./elasticsearch/ Once completed, an internal service object will have been created making the cluster available as http://elasticsearch:9200/ which is the default Kibana (when installed via Helm) will be using.\n  Confirm Elasticsearch is up and running\nkubectl get pods -l component=elasticsearch NAME READY STATUS RESTARTS AGE elasticsearch-exporter-d86ffd94-zw45l 1/1 Running 0 5m44s es-coordinator-b7b984dd4-7wlz5 1/1 Running 0 5m44s es-coordinator-b7b984dd4-89w26 1/1 Running 0 5m44s es-coordinator-b7b984dd4-b4t6j 1/1 Running 0 5m44s es-master-78dfd5b49f-9gf5c 1/1 Running 0 5m44s es-master-78dfd5b49f-smsbw 1/1 Running 0 5m44s es-master-78dfd5b49f-z4qpj 1/1 Running 0 5m44s esdata-0 1/1 Running 0 5m44s esdata-1 1/1 Running 0 4m34s esdata-2 1/1 Running 0 3m22s   Connect to ElasticSearch\nTo connect to ES directly, you can use the following port-forward command\nkubectl port-forward svc/elasticsearch 9200 and then access it via http://localhost:9200\n  Kibana (optional) One of the most popular uses of ES is to use it for log aggregation and indexing, Kibana helps us visualize the data in these indices and can be easily used when installed via its Helm chart\n  Install the helm chart.\nhelm install stable/kibana   Once installed, use a port-foward to Kibana instead of directly to ES\nkubectl port-forward --namespace default $(kubectl get pods --namespace default -l \u0026#34;app=kibana\u0026#34; -o jsonpath=\u0026#34;{.items[0].metadata.name}\u0026#34;) 5601 and then access it via http://localhost:5601\n  Monitoring (optional) As part of the example deployment, ES metrics are exposed and can be scraped by Prometheus on port 9108 (see 77-es-exporter.yaml). This is enabled by default, and should work with the default Prometheus install via Helm. If you\u0026rsquo;re using the Prometheus service monitors, you can monitor this installation by creating a monitor for the es-exporter service. For an example of how this is done to monitor StorageOS, please see prometheus-setup.\n","excerpt":"ElasticSearch with StorageOS  Elasticsearch is a distributed, RESTful search and analytics engine, …","ref":"/docs/usecases/elasticsearch/","title":""},{"body":"Etcd is an open-source distributed, strongly consistent key value store that is used by StorageOS to durably persist the StorageOS cluster state. As the backing store for Kubernetes, StorageOS uses etcd for many of the same reasons.\nStorageOS uses etcd as the single source of truth for all StorageOS objects. Whenever a request is made to create, update or delete an object the result is written to etcd before the request is completed. Using etcd as a configuration store allows nodes to retrieve the current cluster state after being offlined, allowing offlined nodes to rejoin the cluster.\n N.B. StorageOS v2.0 does not provide an embedded etcd server as previous versions did. You will need to setup an etcd server for StorageOS to use prior to installation of StorageOS. Please see our etcd prerequisites page for more information on how to install and configure etcd.\n ","excerpt":"Etcd is an open-source distributed, strongly consistent key value store that is used by StorageOS to …","ref":"/docs/concepts/etcd/","title":"Etcd"},{"body":"Check the etcd prerequisites page for a step by step installation of etcd.\nBest practices StorageOS uses etcd as a service, whether it is deployed following the step by step instructions or as a custom installation. It is expected that the user maintains the availability and integrity of the etcd cluster.\nIt is highly recommended to keep the cluster backed up and ensure high availability of its data.\nNetwork low latency It is important to keep the latency between StorageOS nodes and the etcd replicas low. Deploying an etcd cluster in a different data center or region can make StorageOS detect etcd nodes as unavailable due to latency. 10ms latency between StorageOS and etcd would be the maximum threshold for proper functioning of the system.\nDisk low latency Etcd is very sensitive to disk latency. Because of that, it is recommended to run etcd away from other IO-intensive workloads. Operations such as backups, builds or application bundling cause a heavy usage of disks. If such operations run alongside the etcd nodes, they will cause etcd to become unstable and suffer downtime. It is best to run etcd nodes isolated from other IO workloads.\nIOPS requirements As a general rule, for etcd to operate normally on production clusters, we recommend using the size of machine offered by your cloud provider that guarantees a minimum of 500 IOPS. For example, 750 baseline IOPS are guaranteed on a 250GB AWS gp2 EBS instance at time of writing, and block instances on other cloud providers will also specify baseline IOPS figures.\nCloud providers usually provide \u0026ldquo;bursts\u0026rdquo; of IOPS - temporarily higher rates, limited by credits - with larger volumes providing higher burst capacity. If you are relying on burst capacity for etcd, which requires sustained high performance, careful assessment is necessary to ensure sufficient capacity.\nThe rate of etcd operations is affected by the number of nodes, volumes and replicas in the cluster, therefore the figure of 500 is provided as a guideline only. A development cluster with 5 nodes will not have the same etcd traffic as a production cluster with 100 nodes. Adding monitoring to etcd will help to characterise the traffic, and therefore to assess the individual requirements of a cluster and adjust its resources accordingly.\nEtcd advertise urls The etcd startup parameters advertise-client-urls and initial-advertise-peer-urls specify the addresses etcd clients or other etcd members should use to contact the etcd server. The advertised addresses must be reachable from the remote machines - i.e. where StorageOS is running - so it can connect successfully. Do not advertise addresses like localhost or 0.0.0.0 for a production setup since these addresses are unreachable from remote machines.\nMonitoring It is highly recommended to add monitoring to the etcd cluster. Etcd serves Prometheus metrics on the client port http://etcd-url:2379/metrics.\nYou can use StorageOS developed Grafana Dashboards for etcd. When using etcd for production, you can use the etcd-cluster-as-service, while the etcd-cluster-as-pod can be used when using etcd from the operator.\nDefragmentation Etcd uses revisions to store multiple versions of keys. Compaction removes all key revision prior to a certain revision from etcd. Typically the etcd configuration enables the automatic compaction of keys to prevent performance degradation and limit the storage required. Compaction of revisions can create fragmentation that means space on disk is available for use by etcd but is unavailable for use by the file system. In order to reclaim this space, etcd can be defragmented.\nReclaiming space is important because when the etcd database file grows over the \u0026ldquo;DB_BACKEND_BYTES\u0026rdquo; parameter, the cluster triggers an alarm and sets itself read only and only allows reads and deletes. To avoid hitting the db backend bytes limit, compaction and defragmentation are required. How often defragmentation is required depends on the churn of key revisions in etcd.\nThe Grafana Dashboards mentioned above indicate when nodes require defragmentation. Be aware that defragmentation is a blocking operation that is performed per node, hence the etcd node will be locked for the duration of the defragmentation. Defragmentation usually takes a few milliseconds to complete.\nYou can also set cronjobs that execute the following defragmentation script. It will run a defrag when the DB is at 80% full. A defragmentation operation has to be executed per etcd node and it is a blocking operation. It is recommended to not execute the defragmentation on all etcd members at the same time. If using a cronjob, set them up for different times.\nhttps://raw.githubusercontent.com/storageos/deploy/master/k8s/deploy-storageos/etcd-helpers/etcd-ansible-systemd/roles/install_etcd/templates/defrag-etcd.sh.j2 chmod +x defrag-etcd.sh Known CoreOS Etcd Operator issues This topology is only recommended for deployments where isolated nodes cannot be used.\nEtcd is a distributed key-value store database focused on strong consistency. That means that etcd nodes perform operations across the cluster to ensure quorum. If quorum is lost, etcd nodes stop and etcd marks its contents as read-only. This is because it cannot guarantee that new data will be valid. Quorum is fundamental for etcd operations. When running etcd in pods it is therefore important to consider that a loss of quorum could arise from etcd pods being evicted from nodes.\nOperations such as Kubernetes Upgrades with rolling node pools could cause a total failure of the etcd cluster as nodes are discarded in favor of new ones.\nA 3 etcd node cluster can survive losing one node and recover, a 5 node cluster can survive the loss of two nodes. Loss of further nodes will result in quorum being lost.\nThe etcd-operator doesn\u0026rsquo;t support a full stop of the cluster. Stopping the etcd cluster causes the loss of all the etcd keystore and make StorageOS unable to perform metadata changes.\nThe official etcd-operator repository also has a backup deployment operator that can help backup etcd data. A restore of the etcd keyspace from a backup might cause issues due to the disparity between the cluster state and its metadata in a different point in time. If you need to restore from a backup after a failure of etcd, contact the StorageOS support team.\n","excerpt":"Check the etcd prerequisites page for a step by step installation of etcd.\nBest practices StorageOS …","ref":"/docs/operations/etcd/","title":"Etcd"},{"body":"StorageOS requires an etcd cluster in order to function. For more information on why etcd is required please see our etcd concepts page.\nWe do not support using the Kubernetes etcd for StorageOS installations.\nWe provide two methods for installing etcd. For those looking for a quick route to evaluating StorageOS, our first method installs etcd into your Kubernetes cluster using the CoreOS operator. Due to limitations in the CoreOS operator, this installation is not backed by any persistent storage, and therefore is unsuitable for production installations.\nFor production installations, there is currently no satisfactory way of installing a production-grade etcd cluster inside Kubernetes (although the landscape is changing rapidly, watch this space), and our production guidance remains to install etcd on separate machines outside of your Kubernetes cluster. This method is the best way to ensure a stable StorageOS cluster. Please see our etcd operations page for additional information on deployment best practices and concerns.\n Ephemeral pods within Kubernetes (Testing) External Virtual Machines (Production)  Click the tabs below to select the installation method of your choice.\n Testing Production    Testing - Installing Etcd Into Your Kubernetes Cluster This fast and convenient method is useful for quickly creating an etcd cluster in order to evaluate StorageOS. Do not use it for production installations.\nThis method uses the CoreOS etcd-operator to install a 3 node etcd cluster within your Kubernetes cluster, in the storageos-etcd namespace. We then install a Kubernetes service in that same namespace.\nThe official etcd-operator repository also has a backup deployment operator that can help backup etcd data. A restore of the etcd keyspace from a backup might cause issues due to the disparity between the cluster state and its metadata in a different point in time. If you need to restore from a backup after a failure of etcd, contact the StorageOS support team.\nQuick Install For a one command install, the following script uses kubectl to create an etcd cluster in the storageos-etcd namespace. It requires kubectl in the system path, and the context set to the appropriate cluster.\ncurl -s https://docs.storageos.com/sh/deploy-etcd.sh | bash Installation Step by Step For those who would prefer to execute the steps by themselves, they are as follows:\n  Configure Namespace\nexport NAMESPACE=storageos-etcd   Create Namespace\nkubectl create namespace $NAMESPACE   If running in Openshift, an SCC is needed to start Pods\noc adm policy add-scc-to-user anyuid system:serviceaccount:$NAMESPACE:default   Create ClusterRole and ClusterRoleBinding\n$ kubectl -n $NAMESPACE create -f-\u0026lt;\u0026lt;END apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata: name: etcd-operator roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: etcd-operator subjects: - kind: ServiceAccount name: default namespace: $NAMESPACE END $ kubectl -n $NAMESPACE create -f-\u0026lt;\u0026lt;END apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRole metadata: name: etcd-operator rules: - apiGroups: - etcd.database.coreos.com resources: - etcdclusters - etcdbackups - etcdrestores verbs: - \u0026#34;*\u0026#34; - apiGroups: - apiextensions.k8s.io resources: - customresourcedefinitions verbs: - \u0026#34;*\u0026#34; - apiGroups: - \u0026#34;\u0026#34; resources: - pods - services - endpoints - persistentvolumeclaims - events verbs: - \u0026#34;*\u0026#34; - apiGroups: - apps resources: - deployments verbs: - \u0026#34;*\u0026#34; # The following permissions can be removed if not using S3 backup and TLS - apiGroups: - \u0026#34;\u0026#34; resources: - secrets verbs: - get END   Deploy Etcd Operator\n$ kubectl -n $NAMESPACE create -f - \u0026lt;\u0026lt;END apiVersion: apps/v1 kind: Deployment metadata: name: etcd-operator spec: selector: matchLabels: app: etcd-operator replicas: 1 template: metadata: labels: app: etcd-operator spec: containers: - name: etcd-operator image: quay.io/coreos/etcd-operator:v0.9.4 command: - etcd-operator env: - name: MY_POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace - name: MY_POD_NAME valueFrom: fieldRef: fieldPath: metadata.name END Wait for the Etcd Operator Pod to start\nkubectl -n $NAMESPACE get pod -lapp=etcd-operator   Create the EtcdCluster resource\n$kubectl-n$NAMESPACEcreate-f- \u0026lt;\u0026lt;ENDapiVersion:\u0026#34;etcd.database.coreos.com/v1beta2\u0026#34;kind:\u0026#34;EtcdCluster\u0026#34;metadata:name:\u0026#34;storageos-etcd\u0026#34;spec:size:3version:\u0026#34;3.4.7\u0026#34;pod:etcdEnv:- name:ETCD_QUOTA_BACKEND_BYTESvalue:\u0026#34;2147483648\u0026#34;# 2 GB- name:ETCD_AUTO_COMPACTION_RETENTIONvalue:\u0026#34;1000\u0026#34;# Keep 1000 revisions (default)- name:ETCD_AUTO_COMPACTION_MODEvalue:\u0026#34;revision\u0026#34;# Set the revision moderesources:requests:cpu:200mmemory:300MisecurityContext:runAsNonRoot:truerunAsUser:9000fsGroup:9000tolerations:- operator:\u0026#34;Exists\u0026#34;affinity:podAntiAffinity:preferredDuringSchedulingIgnoredDuringExecution:- weight:100podAffinityTerm:labelSelector:matchExpressions:- key:etcd_clusteroperator:Invalues:- storageos-etcdtopologyKey:kubernetes.io/hostnameEND  Installation Verification $ kubectl -n storageos-etcd get pod,svc NAME READY STATUS RESTARTS AGE pod/etcd-operator-55978c4587-8kx7b 1/1 Running 0 2h pod/storageos-etcd-qm9tmrpnlm 1/1 Running 0 2h pod/storageos-etcd-rzhjdz74hp 1/1 Running 0 2h pod/storageos-etcd-wvvv2d9g98 1/1 Running 0 2h NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/storageos-etcd ClusterIP None \u0026lt;none\u0026gt; 2379/TCP,2380/TCP 22h service/storageos-etcd-client ClusterIP 172.30.132.255 \u0026lt;none\u0026gt; 2379/TCP 22h  The URL from the Service storageos-etcd-client.storageos-etcd.svc:2379 will be used later in the StorageOS Cluster CustomResource the kvBackend.address.\n Known etcd-operator issues Etcd is a distributed key-value store database focused on strong consistency. That means that etcd nodes perform operations across the cluster to ensure quorum. If quorum is lost, etcd nodes stop and etcd marks its contents as read-only. This is because it cannot guarantee that new data will be valid. Quorum is fundamental for etcd operations. When running etcd using the CoreOS Operator, it is important to consider that a loss of quorum could arise from etcd pods being evicted from nodes.\nOperations such as Kubernetes Upgrades with rolling node pools could cause a total failure of the etcd cluster as nodes are discarded in favor of new ones.\nA 3 etcd node cluster can survive losing one node and recover, a 5 node cluster can survive the loss of two nodes. Loss of further nodes will result in quorum being lost.\nThe etcd-operator doesn\u0026rsquo;t support a full stop of the cluster. Stopping the etcd cluster causes the loss of all the etcd keystore and make StorageOS unable to perform metadata changes.\n Production - Etcd on External Virtual Machines For production installations, StorageOS strongly recommends running etcd outside of Kubernetes on a minimum of 3 dedicated virtual machines. This topology offers strong guarantees of resilience and uptime. We recommend this architecture in all environments, including those where Kubernetes is being deployed as a managed service.\nStorageOS doesn\u0026rsquo;t require a high performance etcd cluster, as the throughput of metadata to the cluster is low. However, we recommend a careful assessment of IOPS capacity best practices to ensure that etcd operates normally.\nDepending on the level of redundancy you feel comfortable with you can install etcd on the Kubernetes Master nodes. Take extreme care to avoid collisions of the StorageOS etcd installation with the Kubernetes etcd when using the Kubernetes Master nodes. Precautions such as changing the default configuration for the client and peer ports, and ensuring the etcd data directory is modified. The ansible playbook below will default the etcd installation directory to /var/lib/storageos-etcd.\nYou can choose between two installation options.\n Manual Installation Ansible Installation  Installation - Manual This section documents the steps required for manual installation of etcd using standard package management commands and systemd manifests.\n Repeat the following steps on all the nodes that will run etcd as a systemd service.\n   Configure Etcd version and ports\nexport ETCD_VERSION=\u0026#34;3.4.9\u0026#34; export CLIENT_PORT=\u0026#34;2379\u0026#34; export PEERS_PORT=\u0026#34;2380\u0026#34;  If targeting Kubernetes Master nodes, you must change CLIENT_PORT, PEERS_PORT\n   Download Etcd from CoreOS official site\ncurl -L https://github.com/coreos/etcd/releases/download/v${ETCD_VERSION}/etcd-v${ETCD_VERSION}-linux-amd64.tar.gz -o /tmp/etcd-${ETCD_VERSION}-linux-amd64.tar.gz mkdir -p /tmp/etcd-v${ETCD_VERSION}-linux-amd64 tar -xzvf /tmp/etcd-${ETCD_VERSION}-linux-amd64.tar.gz -C /tmp/etcd-v${ETCD_VERSION}-linux-amd64 --strip-components=1 rm /tmp/etcd-${ETCD_VERSION}-linux-amd64.tar.gz   Install Etcd binaries\ncd /tmp/etcd-v${ETCD_VERSION}-linux-amd64 mv etcd /usr/local/sbin/etcd3 mv etcdctl /usr/local/sbin/etcdctl chmod 0755 /usr/local/sbin/etcd3 /usr/local/sbin/etcdctl   Set up persistent Etcd data directory\nmkdir /var/lib/storageos-etcd   Create the systemd environment file\nOn all nodes that will run etcd create a systemd environemnt file /etc/etcd.conf which has the IPs of all the nodes. The NODE_IP will need to change to correspond to the node IP where the environment file resides. NODE1_IP, NODE2_IP and NODE3_IP will remain the same across all three files.\n$ cat \u0026lt;\u0026lt;END \u0026gt; /etc/etcd.conf # NODE_IP is the IP of the node where this file resides. NODE_IP=10.64.10.228 # Node 1 IP NODE1_IP=10.64.10.228 # Node 2 IP NODE2_IP=10.64.14.233 # Node 3 IP NODE3_IP=10.64.12.111 CLIENT_PORT=${CLIENT_PORT} PEERS_PORT=${PEERS_PORT} END # Verify that variables are expanded in the file $ cat /etc/etcd.conf   Create the systemd unit file for etcd3 service\nCreate a systemd unit file /etc/systemd/system/etcd3.service with the following information:\n[Unit] Description=etcd3 Documentation=https://github.com/coreos/etcd Conflicts=etcd2.service [Service] Type=notify Restart=always RestartSec=5s LimitNOFILE=40000 TimeoutStartSec=0 EnvironmentFile=/etc/etcd.conf ExecStart=/usr/local/sbin/etcd3 --name etcd-${NODE_IP} \\  --data-dir /var/lib/storageos-etcd \\  --quota-backend-bytes 8589934592 \\  --auto-compaction-retention 1000 \\  --auto-compaction-mode revision \\  --initial-cluster-state new \\  --initial-cluster-token etcd-token \\  --listen-client-urls http://${NODE_IP}:${CLIENT_PORT},http://127.0.0.1:${CLIENT_PORT} \\  --advertise-client-urls http://${NODE_IP}:${CLIENT_PORT} \\  --listen-peer-urls http://${NODE_IP}:${PEERS_PORT} \\  --initial-advertise-peer-urls http://${NODE_IP}:${PEERS_PORT} \\  --initial-cluster etcd-${NODE1_IP}=http://${NODE1_IP}:${PEERS_PORT},etcd-${NODE2_IP}=http://${NODE2_IP}:${PEERS_PORT},etcd-${NODE3_IP}=http://${NODE3_IP}:${PEERS_PORT} [Install] WantedBy=multi-user.target  $NODE_IP is the IP address of the machine you are installing etcd on.`\n  Note that setting the advertise-client-urls incorrectly will cause any client connection to fail. StorageOS will fail to communicate to Etcd.\n  If enabling TLS, it is recomended to generate your own CA certificate and key. You will need to distribute the keys and certificates for the client auth on all etcd nodes. Moreover, the ExecStart value should look as below:\n ExecStart=/usr/local/sbin/etcd3 --name etcd-${NODE_IP} \\  --data-dir /var/lib/storageos-etcd \\  --quota-backend-bytes 8589934592 \\  --auto-compaction-retention 1000 \\  --auto-compaction-mode revision \\  --peer-auto-tls \\  --client-cert-auth --trusted-ca-file=/path/to/client-cert.pem \\  --cert-file=/path/to/ca.pem \\  --key-file=/path/to/client-key.pem \\  --initial-cluster-state new \\  --initial-cluster-token etcd-token \\  --listen-client-urls https://${NODE_IP}:${CLIENT_PORT} \\  --advertise-client-urls https://${NODE_IP}:${CLIENT_PORT} \\  --listen-peer-urls https://${NODE_IP}:${PEERS_PORT} \\  --initial-advertise-peer-urls https://${NODE_IP}:${PEERS_PORT} \\  --initial-cluster etcd-${NODE1_IP}=https://${NODE1_IP}:${PEERS_PORT},etcd-${NODE2_IP}=https://${NODE2_IP}:${PEERS_PORT},etcd-${NODE3_IP}=https://${NODE3_IP}:${PEERS_PORT}   Reload and start the etc3 systemd service\n$ systemctl daemon-reload $ systemctl enable etcd3.service $ systemctl start etcd3.service   Installation Verification\n The etcdctl binary is installed at /usr/local/bin on the nodes.\n $ ssh $NODE # Any node running the new etcd $ ETCDCTL_API=3 etcdctl --endpoints=http://127.0.0.1:${CLIENT_PORT} member list # $NODE_IP - the IP of the node 66946cff1224bb5, started, etcd-b94bqkb9rf, http://172.28.0.1:2380, http://172.28.0.1:2379 17e7256953f9319b, started, etcd-gjr25s4sdr, http://172.28.0.2:2380, http://172.28.0.2:2379 8b698843a4658823, started, etcd-rqdf9thx5p, http://172.28.0.3:2380, http://172.28.0.3:2379  Read the etcd operations page for our etcd recommendations.\n   Installation - Ansible For a repeatable and automated installation, use of a configuration management tool such as ansible is recommended. StorageOS provides an ansible playbook to help you deploy etcd on standalone virtual machines.\n  Clone StorageOS deployment repository\ngit clone https://github.com/storageos/deploy.git cd k8s/deploy-storageos/etcd-helpers/etcd-ansible-systemd   Edit the inventory file\n The inventory file targets the nodes that will run etcd. The file hosts is an example of such an inventory file.\n $ cat hosts [nodes] centos-1 ip=\u0026#34;10.64.10.228\u0026#34; fqdn=\u0026#34;ip-10-64-10-228.eu-west-2.compute.internal\u0026#34; centos-2 ip=\u0026#34;10.64.14.233\u0026#34; fqdn=\u0026#34;ip-10-64-14-233.eu-west-2.compute.internal\u0026#34; centos-3 ip=\u0026#34;10.64.12.111\u0026#34; fqdn=\u0026#34;ip-10-64-12-111.eu-west-2.compute.internal\u0026#34; # Edit the inventory file $ vi hosts # Or your own inventory file  The ip or fqdn are used to expose the advertise-client-urls of Etcd. Failing to provide valid ip/fqdn will cause any client connection to fail. StorageOS will fail to communicate to Etcd.\n   Edit the etcd configuration\n If targeting Kubernetes Master nodes, you must change etcd_port_client, etcd_port_peers\n $ cat group_vars/all etcd_version: \u0026#34;3.4.9\u0026#34; etcd_port_client: \u0026#34;2379\u0026#34; etcd_port_peers: \u0026#34;2380\u0026#34; etcd_quota_bytes: 8589934592 # 8 GB etcd_auto_compaction_mode: \u0026#34;revision\u0026#34; etcd_auto_compaction_retention: \u0026#34;1000\u0026#34; members: \u0026#34;{{ groups[\u0026#39;nodes\u0026#39;] }}\u0026#34; installation_dir: \u0026#34;/var/lib/storageos-etcd\u0026#34; advertise_format: \u0026#39;fqdn\u0026#39; # fqdn || ip backup_file: \u0026#34;/tmp/backup.db\u0026#34; tls: enabled: false ca_common_name: \u0026#34;eu-west-2.compute.internal\u0026#34; etcd_common_name: \u0026#34;*.eu-west-2.compute.internal\u0026#34; cert_dir: \u0026#34;/etc/etcdtls\u0026#34; ca_cert_file: \u0026#34;etcd-ca.pem\u0026#34; etcd_server_cert_file: \u0026#34;server.pem\u0026#34; etcd_server_key_file: \u0026#34;server-key.pem\u0026#34; etcd_client_cert_file: \u0026#34;etcd-client.crt\u0026#34; etcd_client_key_file: \u0026#34;etcd-client.key\u0026#34; $ vi group_vars/all  Choose between using IP addressing or FQDN in the advertise_format parameter. It allows you to decide how Etcd advertises its address to clients. This is particularly relevant when using TLS.\n  If enabling TLS, it is recomended to generate your own CA certificate and key. You can do it by generating the CA from the machine running Ansible by: ansible-playbook create_ca.yaml.\n   Install\nansible-playbook -i hosts install.yaml   Installation Verification\n The playbook installs the etcdctl binary on the nodes, at /usr/local/bin.\n $ ssh $NODE # Any node running the new etcd $ ETCDCTL_API=3 etcdctl --endpoints=127.0.0.1:2379 member list 66946cff1224bb5, started, etcd-b94bqkb9rf, http://172.28.0.1:2380, http://172.28.0.1:2379 17e7256953f9319b, started, etcd-gjr25s4sdr, http://172.28.0.2:2380, http://172.28.0.2:2379 8b698843a4658823, started, etcd-rqdf9thx5p, http://172.28.0.3:2380, http://172.28.0.3:2379   Benefits of Running External to Kubernetes Etcd is a distributed key-value store database focused on strong consistency. That means that etcd nodes perform operations across the cluster to ensure quorum. In the case that quorum is lost, an etcd node stops and marks its contents as read-only. Another peer might have a newer version that has not been committed to the database. Quorum is fundamental for etcd operations.\nIn a Kubernetes environment, applications are scheduled across and in some scenarios such as \u0026ldquo;DiskPressure\u0026rdquo; they may need to be evicted from a node, and be scheduled onto a different node. With an application such as etcd, the scenario described can result in quorum being lost, making the cluster unable to recover automatically. Usually a 3 node etcd cluster can survive losing one node and recover. However, losing a second node at the same time or even having a network partition between them will result in quorum lost.\nBind Etcd IPs to Kubernetes Service Kubernetes external services use a DNS name to reference external endpoints, making them easy to reference from inside the cluster. You can use the example from the helper github repository to deploy the external Service. Using an external service can make monitoring of etcd from Prometheus easier.\n  Using Etcd with StorageOS During installation of StorageOS the kvBackend.address parameter of the StorageOS operator is used to specify the address of the etcd cluster. See the StorageOS cluster operator configuration examples for more information.\n","excerpt":"StorageOS requires an etcd cluster in order to function. For more information on why etcd is …","ref":"/docs/prerequisites/etcd/","title":"Etcd"},{"body":"Follow the recipes on this page to create your first PVC (Persistent Volume Claim) using StorageOS. StorageOS implements dynamic provisioning, so the creation of a PVC will automatically provision a PV (PersistentVolume) that can be used to persist data written by a Pod.\nCreate the PersistentVolumeClaim   You can find the basic examples in the StorageOS use-cases repository, in the 00-basic directory.\ngit clone https://github.com/storageos/use-cases.git storageos-usecases cd storageos-usecases/00-basic PVC definition\napiVersion:v1kind:PersistentVolumeClaimmetadata:name:my-vol-1spec:storageClassName:\u0026#34;fast\u0026#34;# StorageOS StorageClassaccessModes:- ReadWriteOnceresources:requests:storage:5GiThe above PVC will dynamically provision a 5GB volume using the fast StorageClass. This StorageClass was created during the StorageOS install and triggers creation of a PersistentVolume by StorageOS.\nFor installations with CSI, you can create multiple StorageClasses in order to specify default labels.\napiVersion:storage.k8s.io/v1kind:StorageClassmetadata:name:storageos-replicatedprovisioner:csi.storageos.com# Provisioner when using CSIparameters:fsType:ext4pool:defaultstorageos.com/replicas:\u0026#34;1\u0026#34;# Enforces 1 replica for the Volume# Change the Namespace below if StorageOS doesn\u0026#39;t run in kube-systemcsi.storage.k8s.io/node-publish-secret-namespace:kube-system# Namespace that runs StorageOS Daemonsetcsi.storage.k8s.io/provisioner-secret-namespace:kube-system# Namespace that runs StorageOS Daemonsetcsi.storage.k8s.io/controller-publish-secret-namespace:kube-system# Namespace that runs StorageOS Daemonsetcsi.storage.k8s.io/node-publish-secret-name:csi-node-publish-secretcsi.storage.k8s.io/provisioner-secret-name:csi-provisioner-secretcsi.storage.k8s.io/controller-publish-secret-name:csi-controller-publish-secretThe above StorageClass has the storageos.com/replicas label set. This label tells StorageOS to create a volume with a replica. Adding StorageOS feature labels to the StorageClass ensures all volumes created with the StorageClass have the same labels. For simplicity\u0026rsquo;s sake this example will use unreplicated volumes.\napiVersion:v1kind:PersistentVolumeClaimmetadata:name:my-vol-1spec:storageClassName:\u0026#34;storageos-replicated\u0026#34;# Reference to the StorageClassaccessModes:- ReadWriteOnceresources:requests:storage:5GiYou can also choose to add the label in the PVC definition rather than the StorageClass. The PVC definition takes precedence over the SC.\napiVersion:v1kind:PersistentVolumeClaimmetadata:name:my-vol-1labels:storageos.com/replicas:\u0026#34;1\u0026#34;spec:storageClassName:\u0026#34;fast\u0026#34;accessModes:- ReadWriteOnceresources:requests:storage:5GiThe above PVC has the storageos.com/replicas label set. This label tells StorageOS to add a replica for the volume that is created. For the sake of keeping this example simple an unreplicated volume will be used.\n  Move into the examples folder and create a PVC using the PVC definition above.\n$ # from storageos-usecases/00-basic $ kubectl create -f ./pvc-basic.yaml You can view the PVC that you have created with the command below\n$ kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE pvc-1 Bound pvc-f8ffa027-e821-11e8-bc0b-0ac77ccc61fa 5Gi RWO fast 1m   Create a pod that mounts the PVC created in step 2.\n$ kubectl create -f ./pod.yaml The command above creates a Pod that uses the PVC that was created in step 1.\napiVersion:v1kind:Podmetadata:name:d1spec:containers:- name:debianimage:debian:9-slimcommand:[\u0026#34;/bin/sleep\u0026#34;]args:[\u0026#34;3600\u0026#34;]volumeMounts:- mountPath:/mntname:v1volumes:- name:v1persistentVolumeClaim:claimName:pvc-1In the Pod definition above volume v1 references the PVC created in step 2, and is mounted in the pod at /mnt. In this example a debian image is used for the container but any container image with a shell would work for this example.\n  Confirm that the pod is up and running\n$ kubectl get pods NAME READY STATUS RESTARTS AGE d1 1/1 Running 0 1m   Execute a shell inside the container and write some contents to a file\n$ kubectl exec -it d1 -- bash root@d1:/# echo \u0026#34;Hello World!\u0026#34; \u0026gt; /mnt/helloworld root@d1:/# cat /mnt/helloworld Hello World! By writing to /mnt inside the container, the StorageOS volume created by the PVC is being written to. If you were to kill the pod and start it again on a new node, the helloworld file would still be avaliable.\nIf you wish to see more use cases with actual applications please see our Use Cases documentation.\n  ","excerpt":"Follow the recipes on this page to create your first PVC (Persistent Volume Claim) using StorageOS. …","ref":"/docs/operations/firstpvc/","title":"StorageOS Volume Guide"},{"body":"$ storageos get --help Fetch basic details for resources Usage: storageos get [command] Available Commands: cluster Fetch cluster-wide configuration details diagnostics Fetch a cluster diagnostic bundle namespace Retrieve basic details of cluster namespaces node Retrieve basic details of nodes in the cluster user Fetch user details volume Retrieve basic details of volumes Flags: -h, --help help for get Global Flags: --endpoints stringArray set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705]) -n, --namespace string specifies the namespace to operate within for commands that require one (default \u0026#34;default\u0026#34;) -o, --output string specifies the output format (one of [json yaml text]) (default \u0026#34;text\u0026#34;) --password string set the StorageOS account password to authenticate with (default \u0026#34;storageos\u0026#34;) --timeout duration set the timeout duration to use for execution of the command (default 5s) --use-ids specify existing StorageOS resources by their unique identifiers instead of by their names --username string set the StorageOS account username to authenticate as (default \u0026#34;storageos\u0026#34;) Use \u0026#34;storageos get [command] --help\u0026#34; for more information about a command. get cluster $ storageos get cluster --help Fetch cluster-wide configuration details Usage: storageos get cluster [flags] Examples: $ storageos get cluster Flags: -h, --help help for cluster get diagnostics $ storageos get diagnostics --help Fetch a cluster diagnostic bundle Usage: storageos get diagnostics [flags] Examples: $ storageos get diagnostics $ storageos get diagnostics --output-file ~/my-diagnostics Flags: -h, --help help for diagnostics --output-file string writes the generated diagnostic bundle to a specified file path get namespace $ storageos get namespace -h Retrieve basic details of cluster namespaces Usage: storageos get namespace [namespace names...] [flags] Aliases: namespace, namespaces Examples: $ storageos get namespaces $ storageos get namespace my-namespace-name Flags: -h, --help help for namespace -l, --selector stringArray filter returned results by a set of comma-separated label selectors get node $ storageos get node -h Retrieve basic details of nodes in the cluster Usage: storageos get node [node names...] [flags] Aliases: node, nodes Examples: $ storageos get node my-node-name Flags: -h, --help help for node -l, --selector stringArray filter returned results by a set of comma-separated label selectors get user $ storageos get user -h Fetch user details Usage: storageos get user [user names...] [flags] Aliases: user, users Examples: $ storageos get user my-username $ storageos get user my-username-1 my-username-2 $ storageos get user --use-ids my-userid $ storageos get user --use-ids my-userid-1 my-userid-2 Flags: -h, --help help for user get volume $ storageos get volume -h Retrieve basic details of volumes Usage: storageos get volume [volume names...] [flags] Aliases: volume, volumes Examples: $ storageos get volumes --all-namespaces $ storageos get volume --namespace my-namespace-name my-volume-name Flags: --all-namespaces retrieves volumes from all accessible namespaces. This option overrides the namespace configuration -h, --help help for volume -l, --selector stringArray filter returned results by a set of comma-separated label selectors ","excerpt":"$ storageos get --help Fetch basic details for resources Usage: storageos get [command] Available …","ref":"/docs/reference/cli/get/","title":"Get"},{"body":"StorageOS provides a GUI for cluster and volume management.\nThe GUI is available at port 5705 on any of the nodes in the cluster. Initially you can log in as the default administrator, with the username and password from the storageos-api Secret. By default storageos, storageos.\n You can access the GUI by either port-forwarding with kubectl or using an Ingress rule. i.e kubectl -n kube-system port-forward svc/storageos 5705.\n Nodes Volumes You can create volumes, including replicated volumes, and view volume details:\nLicensing Cluster info Namespaces Volumes can be namespaced across different projects or teams, and you can switch namespace using the left hand panel:\n","excerpt":"StorageOS provides a GUI for cluster and volume management.\nThe GUI is available at port 5705 on any …","ref":"/docs/reference/gui/","title":"Graphical user interface (GUI)"},{"body":"Various tools are available for checking on the status of a cluster.\nThe StorageOS CLI displays the status of nodes in the cluster.\n$ storageos get nodes NAME HEALTH AGE LABELS node1 online 44 minutes ago node2 online 44 minutes ago node3 online 44 minutes ago node4 online 44 minutes ago node5 online 44 minutes ago ","excerpt":"Various tools are available for checking on the status of a cluster.\nThe StorageOS CLI displays the …","ref":"/docs/operations/health/","title":"Cluster health"},{"body":"$ storageos help Storage for Cloud Native Applications. By using this product, you are agreeing to the terms of the the StorageOS Ltd. End User Subscription Agreement (EUSA) found at: https://storageos.com/legal/#eusa To be notified about stable releases and latest features, sign up at https://my.storageos.com. Usage: storageos [command] Available Commands: apply Make changes to existing resources attach Attach a volume to a node create Create new resources delete Delete resources in the cluster describe Fetch extended details for resources detach Detach a volume from its current location get Fetch basic details for resources help Help about any command version View version information for the StorageOS CLI Flags: --endpoints stringArray set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705]) -h, --help help for storageos -n, --namespace string specifies the namespace to operate within for commands that require one (default \u0026#34;default\u0026#34;) -o, --output string specifies the output format (one of [json yaml text]) (default \u0026#34;text\u0026#34;) --password string set the StorageOS account password to authenticate with (default \u0026#34;storageos\u0026#34;) --timeout duration set the timeout duration to use for execution of the command (default 5s) --use-ids specify existing StorageOS resources by their unique identifiers instead of by their names --username string set the StorageOS account username to authenticate as (default \u0026#34;storageos\u0026#34;) Additional help topics: storageos env View documentation for configuration settings which can be set in the environment storageos exitcodes View documentation for the exit codes used by the StorageOS CLI Use \u0026#34;storageos [command] --help\u0026#34; for more information about a command. ","excerpt":"$ storageos help Storage for Cloud Native Applications. By using this product, you are agreeing to …","ref":"/docs/reference/cli/help/","title":"Help"},{"body":"StorageOS uses the storage available on the nodes where it is installed to present as available for volumes.\nIn order to mitigate against problems caused by filling the host root disk, we recommend mounting a separate device into the /var/lib/storageos directory. StorageOS is agnostic to the type of filesystem mounted in /var/lib/storageos.\nExtending Available Storage StorageOS uses subdirectories of /var/lib/storageos/data to hold user data. By default, the directory /var/lib/storageos/data/dev1 will be created when a node is bootstrapped, and used for pool data. It is possible to shard the data by creating more directories into this structure. StorageOS will save data in any directory that conforms to the pattern /var/lib/storageos/data/dev[0-9]+, such as /var/lib/storageos/data/dev2 or /var/lib/storageos/data/dev5. This functionality enables operators to mount different devices into devX directories and StorageOS will recognise them as available storage automatically.\nThere are two possible options to expand the available disk space for StorageOS to allocate:\n Mount filesystem in /var/lib/storageos/data/devX Use LVM to expand the logical volume available to StorageOS  Option 1: Mount Additional Devices This option enables operators to expand the cluster\u0026rsquo;s available space at any time without having to stop applications or forcing operational downtime. The expansion of disk is transparent for applications and StorageOS Volumes. StorageOS will use the new available space to create new data files.\n  Context\nWe assume that there is a disk available in our Linux system without formatting in addition to the root filesystem. StorageOS data dir dev1 (/var/lib/storageos/data/dev1) is using /dev/xvda1. We will use the device /dev/xvdf to expand StorageOS available space.\nList available block devices in the host.\nroot@node0:~# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT xvda 202:0 0 128G 0 disk `-xvda1 202:1 0 128G 0 part / xvdf 202:80 0 100G 0 disk Check StorageOS cluster\u0026rsquo;s available capacity.\n$ storageos get node -ojson | jq -r '.[] | { name: .name, capacity: .capacity.total }' \u0026quot;node0\u0026quot; 137,438,953,472 \u0026quot;node1\u0026quot; 137,438,953,472 \u0026quot;node2\u0026quot; 137,438,953,472   Format device\nroot@node0:/var/lib/storageos/data# mkfs -t ext4 /dev/xvdf mke2fs 1.42.12 (29-Aug-2014) Creating filesystem with 26214400 4k blocks and 6553600 inodes Filesystem UUID: 380712fa-6f82-477a-81a5-d7466d4c6b7f Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 4096000, 7962624, 11239424, 20480000, 23887872 Allocating group tables: done Writing inode tables: done Creating journal (32768 blocks): done Writing superblocks and filesystem accounting information: done   Mount filesystem\nroot@node0:~# mkdir -p /var/lib/storageos/data/dev2 root@node0:~# mount /dev/xvdf /var/lib/storageos/data/dev2   Verify available storage\nIn less than 30 seconds, StorageOS will see the new available capacity.\n$ storageos get node -ojson | jq -r '.[] | { name: .name, capacity: .capacity.total }' \u0026quot;node0\u0026quot; 244,491,013,324 \u0026quot;node1\u0026quot; 137,438,953,472 \u0026quot;node2\u0026quot; 137,438,953,472 Note that the node node0 has increased the TOTAL capacity in 100Gi.\n   Persist the mount at boot by adding the mount endpoint to /etc/fstab\n Option 2: Expand Existing Devices Backed by LVM This option enables operators to take advantage of LVM to manage disks.\n  Context\nWe assume that /var/lib/storageos is mounted onto an LVM volume. We are using a volumegroup named storageos and logical volume called data. There is a second physical disk /dev/xvdg unused.\nList available block devices in the host.\nroot@node2:~# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT xvda 202:0 0 128G 0 disk `-xvda1 202:1 0 128G 0 part / xvdf 202:80 0 100G 0 disk `-storageos-data 254:0 0 99G 0 lvm /var/lib/storageos xvdg 202:96 0 100G 0 disk Check StorageOS cluster\u0026rsquo;s available capacity.\n$ storageos get node -ojson | jq -r '.[] | { name: .name, capacity: .capacity.total }' \u0026quot;node0\u0026quot; 137,438,953,472 \u0026quot;node1\u0026quot; 137,438,953,472 \u0026quot;node2\u0026quot; 107,696,304,947 # --\u0026gt; LVM storageos/data volume   Add physical disk to LVM\nroot@node2:~# vgextend storageos /dev/xvdg Volume group \u0026quot;storageos\u0026quot; successfully extended The volume group storageos must have 2 physical volumes (#PV)\nroot@node2:~# vgs VG #PV #LV #SN Attr VSize VFree storageos 2 1 0 wz--n- 199.99g 104.99g   Extend logical volume data\nroot@node2:~# lvextend -L+100G /dev/storageos/data Size of logical volume storageos/data changed from 95.00 GiB (24320 extents) to 195.00 GiB (49920 extents). Logical volume data successfully resized   Resize the FileSystem\n Your filesystem must support the option to be expanded, and to do so while in use. Otherwise, you need to unmount first.\n root@node2:~# resize2fs /dev/storageos/data resize2fs 1.42.12 (29-Aug-2014) Filesystem at /dev/storageos/data is mounted on /var/lib/storageos; on-line resizing required old_desc_blocks = 6, new_desc_blocks = 13 The filesystem on /dev/storageos/data is now 51118080 (4k) blocks long.   Check new available space\nThe mounted file system to /var/lib/storageos has increased its size.\nroot@node2:~# df -h /dev/mapper/storageos-data Filesystem Size Used Avail Use% Mounted on /dev/mapper/storageos-data 192G 60M 183G 1% /var/lib/storageos StorageOS available storage has increased too.\n$ storageos get node -ojson | jq -r '.[] | { name: .name, capacity: .capacity.total }' \u0026quot;node0\u0026quot; 137,438,953,472 \u0026quot;node1\u0026quot; 137,438,953,472 \u0026quot;node2\u0026quot; 206,158,430,208 # --\u0026gt; 100G more available    Persist the mount at boot by adding the mount point to /etc/fstab\n ","excerpt":"StorageOS uses the storage available on the nodes where it is installed to present as available for …","ref":"/docs/operations/managing-host-storage/","title":"Managing Host Storage"},{"body":"InfluxDB with StorageOS InfluxDB is a popular open source time series database application optimised for managing datasets consisting of many small measurements. Its advantages include the ability to handle very high write and query loads. Its uses include monitoring, analytics and the recording and analysis of data from sensors.\nBefore you start, ensure you have StorageOS installed and ready on a Kubernetes cluster. See our guide on how to install StorageOS on Kubernetes for more information.\nDeploying InfluxDB on Kubernetes   You can find the latest files in the StorageOS use cases repository\ngit clone https://github.com/storageos/use-cases.git storageos-usecases cd storageos-usecases StatefulSet definition:\napiVersion:apps/v1kind:StatefulSetmetadata:name:influxdbspec:replicas:1selector:matchLabels:app:influxdbserviceName:influxdb...spec:serviceAccountName:influxdb...volumeMounts:- mountPath:/var/lib/influxdbname:data...volumeClaimTemplates:- metadata:name:dataspec:accessModes:[\u0026#34;ReadWriteOnce\u0026#34;]storageClassName:\u0026#34;fast\u0026#34;# StorageOS storageClassresources:requests:storage:20GiThis excerpt is from the StatefulSet definition. This file contains the VolumeClaimTemplate that will dynamically provision storage, using the StorageOS storage class. Dynamic provisioning occurs as a volumeMount has been declared with the same name as a VolumeClaim.\n  Create the InfluxDB objects\nkubectl create -f ./influxdb   Confirm InfluxDB is up and running.\n$ kubectl get pods NAME READY STATUS RESTARTS AGE influxdb-client 1/1 Running 0 1m influxdb-0 1/1 Running 0 1m   Connect to the InfluxDB client pod, then to the InfluxDB server through the service (this reflects the common kubernetes pattern of maintaining a client pod to conveniently inspect a resource interactively). The default user (admin) and password (admin) are defined in the StatefulSet.\n$ kubectl exec -it influxdb-client -- bash root@influxdb-client:/# influx -host influxdb-0.influxdb Connected to http://influxdb-0.influxdb:8086 version 1.8.2 InfluxDB shell version: 1.8.2 \u0026gt; auth username: admin password: \u0026gt; show databases name: databases name _internal \u0026gt; CREATE DATABASE weather; \u0026gt; USE weather Using database weather \u0026gt; INSERT temperature,location=London value=26.4 \u0026gt; INSERT temperature,location=London value=24.9 \u0026gt; INSERT temperature,location=London value=22.2 \u0026gt; INSERT temperature,location=London value=14.7 \u0026gt; INSERT temperature,location=London value=19.5 \u0026gt; INSERT temperature,location=Paris value=27.1 \u0026gt; INSERT temperature,location=Paris value=27.5 \u0026gt; INSERT temperature,location=Paris value=21.3 \u0026gt; INSERT temperature,location=Paris value=26.7 \u0026gt; INSERT temperature,location=Paris value=30.0 \u0026gt; SELECT MEAN(*) FROM \u0026#34;temperature\u0026#34; GROUP BY \u0026#34;location\u0026#34; name: temperature tags: location=London time mean_value 0 25.65 name: temperature tags: location=Paris time mean_value 0 26.90   In the above steps we have inserted some time series data on the temperature at two locations, and calculated the mean for both. InfluxDB offers a variety of such aggregations (see the docs here), allowing convenient analysis of time series data.\nConfiguration If you need custom startup options, you can edit or add to the environment variables within the 20-statefulset.yaml file.\nBackups In this example of how to perform backups of an InfluxDB database on a Kubernetes cluster, we write the output backup file to an Amazon Web Services (AWS) S3 bucket. Other approaches, such as backing up to internal servers or other StorageOS volumes, are possible. For this example to run successfully, Base64-encoded AWS credentials and an S3 bucket name should be inserted into the data field of the backup/50-secret-config.yaml file.\n$ echo -n \u0026#39;\u0026lt;your-aws-access-key-id\u0026gt;\u0026#39; | base64 XXXXXXXXXXXX $ echo -n \u0026#39;\u0026lt;your-aws-secret-access-key\u0026gt;\u0026#39; | base64 XXXXXXXXXXXX $ echo -n \u0026#39;\u0026lt;your-aws-default-region\u0026gt;\u0026#39; | base64 XXXXXXXXXXXX $ echo -n \u0026#39;\u0026lt;your-S3-bucket-name\u0026gt;\u0026#39; | base64 XXXXXXXXXXXX Secret definition:\napiVersion:v1kind:Secretmetadata:name:backup-pod-environmenttype:Opaquedata:AWS_ACCESS_KEY_ID:XXXXXXXXXXXXAWS_SECRET_ACCESS_KEY:XXXXXXXXXXXXAWS_DEFAULT_REGION:XXXXXXXXXXXXBUCKET_NAME:XXXXXXXXXXXXDB_NAME:d2VhdGhlcg==DB_HOST:aW5mbHV4ZGItMC5pbmZsdXhkYjo4MDg4To perform the backup, create the secret and job from the manifest files in the backup directory.\nkubectl create -f ./influxdb/backup/ Confirm that the backup pod has been created, and the backup performed successfully.\nNAME READY STATUS RESTARTS AGE backup-ks976 0/1 Completed 0 1m client 1/1 Running 0 10m influxdb-0 1/1 Running 0 10m mysql-0 1/1 Running 0 10m The files generated by the backup operation should now be available in you S3 bucket. Re-perform the operation at any time by deleting and re-creating the backup job.\n$ kubectl delete -f \u0026#34;influxdb/40-backup-job.yaml\u0026#34; job.batch \u0026#34;backup\u0026#34; deleted $ kubectl create -f \u0026#34;influxdb/40-backup-job.yaml\u0026#34; job.batch/backup created ","excerpt":"InfluxDB with StorageOS InfluxDB is a popular open source time series database application optimised …","ref":"/docs/usecases/influxdb/","title":"InfluxDB"},{"body":"StorageOS has requirements for the configuration of host systems. As such, StorageOS starts an init container that sets the system configuration for StorageOS. The container also manages configuration changes required when upgrading StorageOS versions.\nThe container belongs to the DaemonSet that the StorageOS Cluster Operator starts when a StorageOSCluster resource is created. The storageos-init container is executed as an initContainer as part of a Kubernetes Pod. Therefore, only successful execution of the storageos-init container processes will result in the main container starting.\nScript Framework The code responsible for fulfilling the requirements is based on a Script Framework.\nThe script framework executes a set of scripts, performing checks, verifications and other procedures needed for StorageOS to be able to start. The scripts stdout and stderr are written to the stdout and stderr of the init app. The container logs contain all the logs of the individual scripts that run. The exit statuses of the scripts are used to determine initialization failure or success. Any non-zero exit status is logged as an event in the Kubernetes Pod events.\nIf any of the scripts fail, the storageos-init container will propagate the failure to Kubernetes, showing the status of the Pod as Init:Err.\nTo view the output of all storageos-init containers the following command can be used:\nkubectl -n kube-system logs -l app=storageos,kind=daemonset -c storageos-init For more details, check the StorageOS init container project.\nScripts executed The storageos-init container executes the following scripts.\n enable-lio dbupgrade  ","excerpt":"StorageOS has requirements for the configuration of host systems. As such, StorageOS starts an init …","ref":"/docs/reference/init-container/","title":"Init container"},{"body":"Jenkins with StorageOS This example shows an example of how to deploy Jenkins on Kubernetes with a StorageOS persistent volume being mounted on /var/jenkins_home. Deploying Jenkins using StorageOS offers multiple benefits. Firstly Jenkins can spin up multiple build pods at once to allow concurrent builds of different projects. Secondly Jenkins configuration is on a PersistentVolume so even if the Jenkins pod is rescheduled the configuration will persist.\nUsing StorageOS volume replicas  allows for failure of nodes holding the PersistentVolume without interrupting Jenkins. Lastly by enabling StorageOS fencing  Jenkins time to recover, in case of node failures, is greatly reduced.\nBefore you start, ensure you have StorageOS installed and ready on a Kubernetes cluster. See our guide on how to install StorageOS on Kubernetes for more information\nDeploying Jenkins on Kubernetes   You can find the latest files in the StorageOS example deployment repository.\n$ git clone https://github.com/storageos/use-cases.git storageos-usecases $ cd storageos-usecases $ kubectl create -f ./jenkins   Confirm that Jenkins is up and running\n$ kubectl get pods -w -l app=jenkins NAME READY STATUS RESTARTS AGE jenkins-0 1/1 Running 0 1m   Connect to the Jenkins UI through the Jenkins service.\nYou can do this by port forwarding the Jenkins Kubernetes service to your localhost and accessing the UI via your browser. Alternatively if you have network access to your Kubernetes nodes then you can create a NodePort service and access Jenkins like that. A NodePort service has been left in 10-service.yaml commented out.\nTo port-foward the Jenkins service use the following command.\n$ kubectl port-foward svc/jenkins 8080 To login to the Jenkins UI use the credentials specified in 07-config.yaml, unless these have been changed from the defaults the username/password is admin/password.\n  Create a Jenkins job.\nOnce you are logged into the UI you can create a job that will be farmed out to a Kubernetes plugin build agent. Click New Item, enter a name for the project and select Freestyle project. Next add an Execute shell build step. As a proof of concept you can use the bash below to have the pod execute a sleep.\n#!/bin/bash sleep 1000 Save the project and select Schedule a build of your project. You can watch for the appearance of a build pod using kubectl get pods -l jenkins=agent -w. Once the pod is created you should see the Build Executor status in the Jenkins UI display the pod.\nTo see multiple projects being built at once create another project and try scheduling a build of both projects at the same time.\n  ","excerpt":"Jenkins with StorageOS This example shows an example of how to deploy Jenkins on Kubernetes with a …","ref":"/docs/usecases/jenkins/","title":""},{"body":"Kafka with StorageOS  Kafka is a popular stream processing platform combining features from pub/sub and traditional queues.\nUsing StorageOS persistent volumes with Apache Kafka means that if a pod fails, the cluster is only in a degraded state for as long as it takes Kubernetes to restart the pod. When the pod comes back up, the pod data is immediately available. Should Kubernetes schedule the kafka pod on a new node, StorageOS allows for the data to be available to the pod, irrespective of whether or not the original StorageOS master volume is located on the same node.\nKafka has features to allow it to handle replication, and as such careful consideration of whether to allow StorageOS or Kafka to handle replication is required.\nBefore you start, ensure you have StorageOS installed and ready on a Kubernetes cluster. See our guide on how to install StorageOS on Kubernetes for more information.\nPrerequisites  Apache Zookeeper is required by Kafka to function; we assume it to already exist and be accessible within the Kubernetes cluster as zookeeper, see how to run Zookeeper with StorageOS here StorageOS is assumed to have been installed; please check for the latest available version here Kafka pods require 1536 MB of memory for successful scheduling  Helm To simplify the deployment of kafka, we\u0026rsquo;ve used this Kafka helm chart (incubator) (version 0.13.8, app version 5.0.1) and rendered it into the example deployment files you can find in our GitHub repo.\nClone the use cases repo You can find the latest files in the StorageOS use cases repository in /kafka/\ngit clone https://github.com/storageos/use-cases.git storageos-usecases cd storageos-usecases StatefulSet definition\n---apiVersion:apps/v1beta1kind:StatefulSetmetadata:name:kafkalabels:app:kafka...spec:serviceName:kafka-headlesspodManagementPolicy:OrderedReadyupdateStrategy:type:OnDeletereplicas:3# \u0026lt;--- number of kafa pods to runtemplate:...spec:serviceAccountName:kafkacontainers:...- name:kafka-brokerimage:\u0026#34;confluentinc/cp-kafka:5.0.1\u0026#34;imagePullPolicy:\u0026#34;IfNotPresent\u0026#34;...volumeMounts:- name:datadirmountPath:\u0026#34;/var/data\u0026#34;volumes:- name:jmx-configconfigMap:name:kafka-metricsterminationGracePeriodSeconds:60volumeClaimTemplates:- metadata:name:datadirspec:accessModes:[\u0026#34;ReadWriteOnce\u0026#34;]resources:requests:storage:10Gi# \u0026lt;--- storage requested for each podstorageClassName:\u0026#34;fast\u0026#34;# \u0026lt;--- the StorageClass to useThis excerpt is from the StatefulSet definition (10-statefulset.yaml). The file contains the PersistentVolumeClaim template that will dynamically provision the necessary storage, using the StorageOS storage class.\nDynamic provisioning occurs as a volumeMount has been declared with the same name as a VolumeClaimTemplate.\n  Create the kubernetes objects\nkubectl apply -f ./kafka/   Confirm kafka is up and running\n$ kubectl get pods -l app=kafka NAME READY STATUS RESTARTS AGE kafka-0 2/2 Running 0 10m kafka-1 2/2 Running 0 9m26s kafka-2 2/2 Running 0 7m59s   Connect to kafka\nConnect to the kafka test client pod and send some test data to kafka through its service endpoint\n  Connect to the pod\nkubectl exec -it kafka-test-client /bin/bash   Create a topic\n/usr/bin/kafka-topics --zookeeper zookeeper:2181 --create --topic test-rep-one --partitions 6 --replication-factor 1   Send some test data\n/usr/bin/kafka-run-class org.apache.kafka.tools.ProducerPerformance --topic test-rep-one --num-records 5000 --record-size 100 --throughput -1 --print-metrics --producer-props acks=1 bootstrap.servers=kafka:9092 buffer.memory=67108864 batch.size=8196   ","excerpt":"Kafka with StorageOS  Kafka is a popular stream processing platform combining features from pub/sub …","ref":"/docs/usecases/kafka/","title":""},{"body":"Kubevirt with StorageOS Kubevirt is a CNCF sandbox project that allows the running of virtual machines (VMs) in Kubernetes pods.\nDeploying Kubevirt using StorageOS offers multiple benefits. Kubevirt can spin up VMs as Kubernetes pods, using images on StorageOS persistent volumes. Doing this allows the VM data to persist through restarts and rescheduling. Using StorageOS volume replicas also allows for failure of nodes holding the PersistentVolume without interrupting the VM running off the PersistentVolume. Containerized Data Importer (CDI) can also be used to prepare StorageOS volumes with disk images in an automated fashion. Simply by declaring that a VirtualMachine will use a DataVolume and providing the disk image URL, a StorageOS volume can be dynamically provisioned and automatically prepared with the disk image.\nThis usecase will guide you through installing KubeVirt and CDI on your Kubernetes cluster, and create a VM. By the end of the guide you\u0026rsquo;ll be able to launch a shell inside the KubeVirt VM that\u0026rsquo;s running as a Kubernetes pod.\nBefore you start, ensure you have StorageOS installed and ready on a Kubernetes cluster. See our guide on how to install StorageOS on Kubernetes for more information.\nPrerequisites Please ensure you have met the Kubevirt prerequisites, please see the Kubevirt installation instructions  for more information.\nAs part of this installation it is assumed that you are running a Kubernetes cluster on VMs. As such nested virtualization or hardware emulation need to be enabled.\nDeploying KubeVirt on Kubernetes  For ease of installation we have enabled hardware emulation. If your VMs support nested virtualization then edit the Kubevirt ConfigMap ./kubevirt-install/10-cm.yaml, removing the line debug.useEmulation: \u0026quot;true\u0026quot;.\n   In order to deploy Kubevirt you just need to clone this repository and use kubectl to create the Kubernetes objects.\n$ git clone https://github.com/storageos/use-cases.git storageos-usecases $ cd storageos-usecases/kubevirt $ kubectl create -f ./kubevirt-install   Check that the Kubevirt pods are running.\n$ kubectl get pods -w -n kubevirt NAME READY STATUS RESTARTS AGE virt-api-57546d479b-p26d4 1/1 Running 0 1m virt-api-57546d479b-zs5dw 1/1 Running 0 1m virt-controller-56b5498854-7xsfz 1/1 Running 1 1m virt-controller-56b5498854-bz559 1/1 Running 1 1m virt-handler-6z4kq 1/1 Running 0 1m virt-handler-7szhl 1/1 Running 0 1m virt-handler-jmm6w 1/1 Running 0 1m virt-operator-79c9bdd859-8xq98 1/1 Running 0 1m virt-operator-79c9bdd859-kfjz6 1/1 Running 0 1m   Once Kubevirt is running install CDI.\n$ kubectl create -f ./cdi   Check that the CDI pods are running correctly.\n$ kubectl get pods -n cdi NAME READY STATUS RESTARTS AGE cdi-apiserver-8668f888df-s6pp4 1/1 Running 0 1m cdi-deployment-5cf794896b-whh4j 1/1 Running 0 1m cdi-operator-5887f96c-dz2hg 1/1 Running 0 1m cdi-uploadproxy-97fbbfcbf-6f9xs 1/1 Running 0 1m   Now that CDI and Kubevirt are running, VMs can be created. In this example VMs running Cirros, a small and lightweight OS, will be created. The vm-cirros.yaml manifest creates a VirtualMachine that uses a DataVolume. This means that CDI will create a StorageOS backed PVC and download the image that the VirtualMachineInstance (VMI) will boot from onto the PVC.\n$ kubectl create -f ./vm-cirros.yaml   Check that the VMI is running. Note that the VMI will only be created after CDI has downloaded the Cirros disk image onto a StorageOS persistent volume so depending on your connection speed this may take some time.\n$ kubectl get vmi NAME AGE PHASE IP NODENAME cirros 1m Running 10.244.2.12 ip-10-1-10-154.storageos.net $ kubectl get pods NAME READY STATUS RESTARTS AGE virt-launcher-cirros-drqhr 1/1 Running 0 1m   Connect to the VM console.\nThis example uses the virtctl kubectl plugin in order to connect to the VMs console. The escape sequence ^] is ctrl + ]\n$ kubectl virt console cirros Successfully connected to cirros console. The escape sequence is ^] login as \u0026#39;cirros\u0026#39; user. default password: \u0026#39;gocubsgo\u0026#39;. use \u0026#39;sudo\u0026#39; for root. cirros login: cirros Password: $   Cloning Volumes CDI allows for images to be cloned using a DataVolume manifest. Verify that the cirros pvc, created as part of the vm-cirros.yaml file, exists before attempting to clone the volume.\n N.B. Ensure that the VMI is stopped before continuing!\n   Verify that the VMI is stopped before continuing, and that the cirros pvc, created as part of the vm-cirros.yaml file, exists before attempting to clone the volume.\n$ kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE cirros Bound pvc-f4833060-5a77-420c-927e-6bc518d9df3c 12Gi RWO fast 1m   Once the PVC\u0026rsquo;s existence is confirmed then create a new DataVolume that uses the cirros PVC as its source.\n$ kubectl create -f ./cloned.yaml   Watch as the CDI pods are created.\n$ kubectl get pods -w You\u0026rsquo;ll see that a cdi-upload-cloned-datavolume pod is created and then a cdi-clone-source pod is created. The cdi-source pod mounts the original cirros volume and sends the contents of the volume to the cdi-upload pod. The cdi-upload pod creates and mounts a new PVC and writes the contents of the original volume to it.\n  ","excerpt":"Kubevirt with StorageOS Kubevirt is a CNCF sandbox project that allows the running of virtual …","ref":"/docs/usecases/kubevirt/","title":""},{"body":"Feature labels are a powerful and flexible way to control storage features.\nLabels can be applied to various StorageOS artefacts. Applying specific feature labels triggers compression, replication and other storage features. No feature labels are present by default.\nStorageOS Node labels Nodes do not have any feature labels present by default. When StorageOS is run within Kubernetes, the Cluster Operator syncs any node labels set on Kubernetes nodes within StorageOS. Node labels may also be set with the CLI or UI.\n   Feature Label Values Description     Compute only storageos.com/computeonly true / false Specifies whether a node should be computeonly where it only acts as a client and does not host volume data locally, otherwise the node is hyperconverged (the default), where the node can operate in both client and server modes.    You can set the computeonly label on the Kubernetes node and the label will be sync\u0026rsquo;d to the StorageOS node (labels take an eventual consistency reconciliation time of ~1min).\nkubectl label node $NODE storageos.com/computeonly=true StorageOS Volume labels Volumes do not have any feature labels present by default. WARNING: The caching and compression labels can only apply during provision time, they can\u0026rsquo;t be changed during execution.\n   Feature Label Values Description     Caching storageos.com/nocache true / false Switches off caching.   Compression storageos.com/nocompress true / false Switches off compression of data at rest and in transit.   Replication storageos.com/replicas integers [0, 5] Replicates entire volume across nodes. Typically 1 replica is sufficient (2 copies of the data); more than 2 replicas is not recommended.    To create a volume with a feature label:\n  Option 1: PVC Label\nAdd the label in the PVC definition, for instance:\napiVersion:v1kind:PersistentVolumeClaimmetadata:name:pvc-3labels:storageos.com/replicas:\u0026#34;1\u0026#34;# Label \u0026lt;-----spec:storageClassName:\u0026#34;fast\u0026#34;accessModes:- ReadWriteOnceresources:requests:storage:1G  Option 2: Set label in the StorageClass\nAny PVC using the StorageClass inherits the label. The PVC label takes precedence over the StorageClass parameters.\napiVersion:storage.k8s.io/v1kind:StorageClassmetadata:name:storageos-replicatedparameters:fsType:ext4pool:defaultstorageos.com/replicas:\u0026#34;1\u0026#34;# Label \u0026lt;--------provisioner:storageos# CSI driver (recommended)   Option 3:\nOnce a PVC is created, you can update the Labels in StorageOS both in the UI or CLI. Those labels are going to be visible only for StorageOS and will not be synced to the Kubernetes resource.\n  ","excerpt":"Feature labels are a powerful and flexible way to control storage features.\nLabels can be applied to …","ref":"/docs/reference/labels/","title":"StorageOS Feature labels"},{"body":"A newly installed StorageOS cluster does not include a licence. A cluster can run unlicensed for 24 hours. After that, new operations such as volume provisioning or adding nodes are not permitted. Normal functioning of the cluster can be unlocked by applying a Free Developer licence.\nTo learn how to apply a licence to your cluster, check the operations licensing page.\nFree Developer licence The developer licence only requires the user to register and issue a licence through StorageOS.\nThe developer licence is free and grants a licence for a 5 node StorageOS cluster with 5TB of provisioned capacity. It is designed to enable basic cloud native workflows in Kubernetes that require the persistence of stateful application data. Dynamic provisioning, distributed access to data and high availability of volumes through synchronous replication and automatic failover are some of the features that are available under the developer licence.\nThe free developer licence includes community support through our public Slack channel..\nCommercial licences For information on our commercial offerings, including support, please contact sales@storageos.com.\nNote about capacity limits Some StorageOS licences have limits on capacity. StorageOS allows provisioning volumes until the limit of the licence is reached. Only the size of the volume requested by the Persistent Volume Claim counts for the licence limit, regardless whether of whether the volume has replication enabled.\nOnce the licence limit is reached, new volumes are not able to provision unless provisioned capacity is released, i.e deleting volumes. That behaviour is not tight to the capacity of the backend disks on your nodes.\n","excerpt":"A newly installed StorageOS cluster does not include a licence. A cluster can run unlicensed for 24 …","ref":"/docs/reference/licence/","title":"Licence"},{"body":"A newly installed StorageOS cluster does not include a licence. A cluster can run unlicensed for 24 hours. After that, new operations such as volume provisioning or adding nodes are not permitted. Normal functioning of the cluster can be unlocked by applying for a Free Developer licence.\nObtaining a Developer licence via the GUI You will need access to the StorageOS GUI on port 5705 of any of your nodes. For convenience, it is often easiest to port forward the service using the following kubectl incantation (this will block, so a second terminal window may be advisable):\n$ kubectl port-forward -n kube-system svc/storageos 5705 As an alternative, an Ingress controller may be preferred.\nOnce you have obtained access to the GUI, login using whatever credentials you used to create the cluster.\nYou can obtain and apply a free Developer licence in the StorageOS web GUI by creating or logging in with a StorageOS account on the StorageOS portal via the licence page of the GUI:\nWait a few seconds for the licence generation process to complete, at which point your licence will be visible.\nApplying a previously obtained licence via the GUI Commercial licences are delivered through contact with the StorageOS team. To apply such keys, via the web GUI, visit the licence section of the GUI and click on the tab \u0026ldquo;Upgrade\u0026rdquo;, for the specific licence level you purchased. Then paste the licence key and click on \u0026ldquo;UPLOAD KEY TO CLUSTER\u0026rdquo;. Note that you can also view your cluster ID on the same page.\nApplying a licence via the CLI Before getting a licence, you need to know the ID of your StorageOS cluster.\nThis CLI command can print the cluster ID:\n$ storageos get cluster ID: 704dd165-9580-4da4-a554-0acb96d328cb Licence: expiration: 2021-03-25T13:48:46Z (1 year from now) capacity: 5.0 TiB kind: professional customer name: storageos Created at: 2020-03-25T13:48:33Z (1 hour ago) Updated at: 2020-03-25T13:48:46Z (1 hour ago) Given the Cluster ID, the StorageOS team can generate a licence. Once, given the key, you can apply the licence by using the following command.\n$ echo PASTE-THE-LICENCE-KEY-HERE | storageos apply licence --from-stdin Read the licence CLI command reference for further information.\nObtaining an Enterprise licence Please contact sales@storageos.com to discuss pricing for commercial licences.\n","excerpt":"A newly installed StorageOS cluster does not include a licence. A cluster can run unlicensed for 24 …","ref":"/docs/operations/licensing/","title":"Licensing"},{"body":"This procedure aims to replace the current nodes of your Etcd cluster with new ones. Migrating Etcd is a delicate operation, so we recommend a careful assessment of the steps required before starting.\nThe following procedure is scoped for Etcd clusters that are provisioned outside Kubernetes on their own nodes.\n Please note that this procedure involves full downtime of StorageOS and access to its data\n  The following procedure is designed for Etcd clusters that are provisioned externally to Kubernetes, with members that are situated on their own nodes.\n Migration To perform Etcd migrations safely, we recommend adding new Etcd nodes to the existing Etcd cluster, before editing the StorageOS custom resource definition and finally deleting the old Etcd nodes one by one. The full procedure is as follows:\n  Prepare all the nodes that will host etcd. The Etcd provisioning steps can be found in the etcd section of the StorageOS prerequisites page.\nOn each new node, edit the /etc/etcd.conf file to contain the IP addresses of all old and new nodes. In the /etc/systemd/system/etcd3.service file, ensure that the --initial-cluster argument includes all old and new nodes. Also ensure that the --initial-cluster-state argument is set to existing.\nFor example, adding 3 further nodes to an initially 3 node cluster would involve populating /etc/etcd.conf on each of the fourth, fifth and sixth nodes with the IP addresses of all 6 nodes intended for the cluster, as well as environment variables for the client port and peers port:\n# NODE_IP is the IP of the node where this file resides. NODE_IP=192.168.152.142 # Node 1 IP NODE1_IP=192.168.195.168 # Node 2 IP NODE2_IP=192.168.202.40 # Node 3 IP NODE3_IP=192.168.174.117 # Node 4 IP NODE4_IP=192.168.152.142 # Node 5 IP NODE5_IP=192.168.198.252 # Node 6 IP NODE6_IP=192.168.150.122 CLIENT_PORT=2379 PEERS_PORT=2380 The executable section of /etc/systemd/system/etcd3.service on each of the new nodes should then be edited to reflect these environment variables. For example, the service file for the second new node to be added to an initially 3 node cluster would refer to the three old nodes and two new nodes. Note that in the --initial-cluster variable we do not include members that have not yet been added to the cluster, apart from the member we are currently adding.\n... ExecStart=/usr/local/sbin/etcd3 --name etcd-${NODE_IP} \\ ... --initial-cluster-state existing \\ ... --initial-cluster etcd-${NODE1_IP}=http://${NODE1_IP}:${PEERS_PORT},etcd-${NODE2_IP}=http://${NODE2_IP}:${PEERS_PORT},etcd-${NODE3_IP}=http://${NODE3_IP}:${PEERS_PORT},etcd-${NODE4_IP}=http://${NODE4_IP}:${PEERS_PORT},etcd-${NODE5_IP}=http://${NODE5_IP}:${PEERS_PORT} ... At runtime, the value of initial-cluster will resolve to the following:\n\u0026quot;etcd-192.168.195.168=http://192.168.195.168:2380,etcd-192.168.202.40=http://192.168.202.40:2380,etcd-192.168.174.117=http://192.168.174.117:2380,etcd-192.168.152.1\u0026quot;   Connect to an Etcd node that already belongs to the cluster, and add one of the new nodes to the cluster, using the etcdctl member add --learner command per the etcd documentation. The output includes environment variables with the name of the new member and the state and constituents of the cluster now that the new node has been added:\n# Name of the new member of etcd ETCD_NEW_MEMBER=\u0026#34;etcd-192.168.152.142\u0026#34; # Peer url for the new member of etcd, including the port ETCD_NEW_MEMBER_PEER=\u0026#34;http://192.168.152.142:2380\u0026#34; # Client url for the new member of etcd, including the port ETCD_NEW_MEMBER_CLIENT=\u0026#34;http://192.168.195.168:2379\u0026#34; # Add the new member to the cluster ETCDCTL_API=3 etcdctl member add $ETCD_NEW_MEMBER \\  --peer-urls=\u0026#34;$ETCD_NEW_MEMBER_PEER\u0026#34; \\  --endpoints=\u0026#34;$ETCD_NEW_MEMBER_CLIENT\u0026#34; Member 4a5820690b65300c added to cluster 97f788d9ca9b3357 ETCD_NAME=\u0026#34;etcd-192.168.152.142\u0026#34; ETCD_INITIAL_CLUSTER=\u0026#34;etcd-192.168.202.40=http://192.168.202.40:2380,etcd-192.168.174.117=http://192.168.174.117:2380,etcd-192.168.152.142=http://192.168.152.142:2380,etcd-192.168.195.168=http://192.168.195.168:2380\u0026#34; ETCD_INITIAL_ADVERTISE_PEER_URLS=\u0026#34;http://192.168.152.142:2380\u0026#34; ETCD_INITIAL_CLUSTER_STATE=\u0026#34;existing\u0026#34;   Start the Etcd service on the new node:\nsystemctl daemon-reload systemctl enable etcd3.service systemctl start etcd3.service List the members in the cluster and confirm that the new node is present and has started, and has therefore successfully been added:\n$ ETCDCTL_API=3 etcdctl --endpoints=http://127.0.0.1:2379 member list aa9dec86cc9c2d1, started, etcd-192.168.202.40, http://192.168.202.40:2380, http://192.168.202.40:2379, false 48823f29326a24f7, started, etcd-192.168.174.117, http://192.168.174.117:2380, http://192.168.174.117:2379, false 4a5820690b65300c, started, etcd-192.168.152.142, http://192.168.152.142:2380, http://192.168.152.142:2379, true gwk09ksjs5a6n862, started, etcd-192.168.152.142, http://192.168.152.142:2380, http://192.168.195.168:2379, false   Promote the learner to be a full voting member of the cluster with the etcdctl member promote \u0026lt;member id\u0026gt; command.\n As per the etcd documentation, the learner will fail to be promoted if it is not ready.\n   Repeat the previous two steps for all the remaining nodes onto which you wish to migrate etcd. Per the Etcd documentation, when adding more than one member to a cluster, it best practice to configure each member one at a time, verifying that each starts successfully before moving on to the next.\n  Delete the StorageOS Custom Resource per the instructions in the uninstall operations page.\n Although deleting the StorageOSCluster Custom Resource will stop StorageOS, the data will be safe. When StorageOS starts again, the Volumes will be available.\n   Connect to one of the newly-added nodes, and use etcdctl to remove the old Etcd nodes 1 by 1:\n$ ETCDCTL_API=3 etcdctl --endpoints=http://127.0.0.1:2379 member remove 48823f29326a24f7 Member 48823f29326a24f7 removed from cluster 97f788d9ca9b3357 After removing a node, check that Etcd is healthy and quorum has been maintained with the following commands:\nexport ETCD_ENDPOINTS=\u0026quot;http://192.168.2020.40:2379,http://192.168.174.117.2379,...\u0026quot; etcdctl --endpoints $ETCD_ENDPOINTS endpoint health -w table etcdctl --endpoints $ETCD_ENDPOINTS endpoint status -w table   Edit the kvBackend.address parameter of the StorageOS Custom Resource Definition to reflect the locations of the new Etcd nodes, removing the references to the old nodes\n  Recreate the StorageOS Custom Resource with the newly-edited kvBackend.address parameter\n  Perform a member list again, to confirm that only the new machines are still present in the cluster:\n$ ETCDCTL_API=3 etcdctl --endpoints=http://127.0.0.1:2379 member list 4a5820690b65300c, started, etcd-192.168.152.142, http://192.168.152.142:2380, http://192.168.152.142:2379, false b1780933f495adb3, started, etcd-192.168.150.122, http://192.168.150.122:2380, http://192.168.150.122:2379, false ee47ff85984afcc0, started, etcd-192.168.198.252, http://192.168.198.252:2380, http://192.168.198.252:2379, false At this stage Etcd has been successfully migrated.\n  Etcd Learners As of v3.4, Etcd includes the functionality to add a new member as a \u0026ldquo;learner\u0026rdquo; (a non-voting member of the cluster). This makes the process of adding new members safer, as it allows the new member to complete the process of synchronisation before participating in quorum, minimising cluster downtime.\nIf you are using an earlier version of etcd, make the following substitutions to the above process:\n In step 2, replace the etcdctl member add --learner command with etcdctl member add Skip theetcdctl member promote \u0026lt;member id\u0026gt; command  Using Etcd learners provides added resilience, so consider upgrading to a version of Etcd that supports this as soon as possible.\n","excerpt":"This procedure aims to replace the current nodes of your Etcd cluster with new ones. Migrating Etcd …","ref":"/docs/operations/etcd/etcd-migration/","title":"Migrate Etcd"},{"body":"MS SQL with StorageOS  Beginning with Microsoft SQL Server 2017, Microsoft has supported MSSQL on linux.\nBefore you start, ensure you have StorageOS installed and ready on a Kubernetes cluster. See our guide on how to install StorageOS on Kubernetes for more information.\nDeploying MS SQL on Kubernetes   You can find the latest files in the StorageOS use cases repository\ngit clone https://github.com/storageos/use-cases.git storageos-usecases StatefulSet defintion\nkind:StatefulSetmetadata:name:mssqlspec:selector:matchLabels:app:mssqlenv:prodserviceName:mssqlreplicas:1...spec:serviceAccountName:mssql...volumeMounts:- name:datamountPath:/var/opt/mssql...volumeClaimTemplates:- metadata:name:datalabels:env:prodspec:accessModes:[\u0026#34;ReadWriteOnce\u0026#34;]storageClassName:\u0026#34;fast\u0026#34;# StorageOS storageClass resources:requests:storage:5GiThis excerpt is from the StatefulSet definition. This file contains the VolumeClaim template that will dynamically provision storage, using the StorageOS storage class. Dynamic provisioning occurs as a volumeMount has been declared with the same name as a VolumeClaim.\n  Move into the MS SQL examples folder and create the objects\ncd storageos-usecases kubectl create -f ./mssql   Confirm MS SQL is up and running.\n$ kubectl get pods -w -l app=mssql NAME READY STATUS RESTARTS AGE mssql-0 1/1 Running 0 1m   Connect to the MS SQL client pod and connect to the MS SQL server through the service\n$ kubectl exec -it mssql-0 -- /opt/mssql-tools/bin/sqlcmd -S mssql-0.mssql -U SA -P \u0026#39;Password15\u0026#39; 1\u0026gt; USE master; 2\u0026gt; GO Changed database context to \u0026#39;master\u0026#39;. 1\u0026gt; SELECT name, database_id, create_date FROM sys.databases ; 2\u0026gt; GO name database_id create_date --------------------------- ----------- ----------------------- master 1 2003-04-08 09:13:36.390 tempdb 2 2018-11-02 16:30:37.907 model 3 2003-04-08 09:13:36.390 msdb 4 2018-10-19 01:18:57.300 (4 rows affected)   Configuration If you need custom startup options, you can edit the ConfigMap file 15-mssql-configmap.yaml with your desired MS SQL configuration settings.\n","excerpt":"MS SQL with StorageOS  Beginning with Microsoft SQL Server 2017, Microsoft has supported MSSQL on …","ref":"/docs/usecases/mssql/","title":""},{"body":"StorageOS supports secure communication with an external etcd cluster using mutual TLS (mTLS). With mTLS both StorageOS and etcd authenticate each other ensuring that communication only happens between mutually authenticated end points, and that all communication is encrypted.\nStorageOS uses the certificates and keys from a Secret to cypher and authenticate Etcd traffic.\nHow to create the certificates Secret The client auth certificates need the following file names, in the Secret.\n etcd-client-ca.crt - containing the etcd Certificate Authority certificate etcd-client.crt - containing the etcd Client certificate etcd-client.key - cotaining the etcd Client key  kubectl create secret -n storageos-etcd generic \\  etcd-client-tls \\  --from-file=\u0026#34;etcd-client-ca.crt\u0026#34; \\  --from-file=\u0026#34;etcd-client.crt\u0026#34; \\  --from-file=\u0026#34;etcd-client.key\u0026#34; How to use the mTLS certificates Secret with StorageOS Below is an example StorageOSCluster resource that can be used to setup StorageOS with etcd using mTLS.\napiVersion:storageos.com/v1kind:StorageOSClustermetadata:name:storageos-clusternamespace:\u0026#34;storageos-operator\u0026#34;spec:# StorageOS Pods are in kube-system by defaultsecretRefName:\u0026#34;storageos-api\u0026#34;secretRefNamespace:\u0026#34;default\u0026#34;images:nodeContainer:\u0026#34;storageos/node:v2.3.2\u0026#34;namespace:\u0026#34;storageos\u0026#34;# External mTLS secured etcd cluster specific propertiestlsEtcdSecretRefName:\u0026#34;etcd-client-tls\u0026#34;# Secret containing etcd client certificatestlsEtcdSecretRefNamespace:\u0026#34;etcd\u0026#34;# Namespace of the client certificates secretkvBackend:address:\u0026#34;https://storageos-etcd-cluster-client.storagos-etcd.svc:2379\u0026#34;# Etcd client service address.backend:\u0026#34;etcd\u0026#34;# Backend typetlsEtcdSecretRefName and tlsEtcdSecretRefNamespace are used to pass a reference to the Secret.\nThe StorageOS operator uses the etcd secret that contains the client certificates, to build a secret in the StorageOS installation namespace. This secret contains the certificate filenames and certificate file contents. The StorageOS daemonset that is created by the operator mounts the secret as a volume so that the certificate files are available inside the pod. Environment variables containing the file paths are passed to the StorageOS process in order to use the files from the mounted path.\nA worked example of setting up StorageOS with external etcd using mTLS is avaliable here. For ease of use the example uses the CoreOS etcd operator and the CoreOS guide The example uses the CoreOS etcd operator and follows the CoreOS guide for Cluster TLS.\n","excerpt":"StorageOS supports secure communication with an external etcd cluster using mutual TLS (mTLS). With …","ref":"/docs/operations/etcd/storageos-secret-info/","title":"Encrypting communication with Etcd"},{"body":"MySQL with StorageOS MySQL is a popular SQL open source database for a wide range of popular web-based applications including WordPress.\nBefore you start, ensure you have StorageOS installed and ready on a Kubernetes cluster. See our guide on how to install StorageOS on Kubernetes for more information.\nDeploying MySQL on Kubernetes   You can find the latest files in the StorageOS use cases repository\ngit clone https://github.com/storageos/use-cases.git storageos-usecases StatefulSet defintion\napiversion:apps/v1kind:statefulsetmetadata:name:mysqlspec:selector:matchlabels:app:mysqlenv:prodservicename:mysqlreplicas:1...spec:serviceaccountname:mysql...volumemounts:- name:datamountpath:/var/lib/mysqlsubpath:mysql- name:confmountpath:/etc/mysql/mysql.conf.d...volumeclaimtemplates:- metadata:name:datalabels:env:prodspec:accessmodes:[\u0026#34;readwriteonce\u0026#34;]storageclassname:\u0026#34;fast\u0026#34;# storageos storageclass resources:requests:storage:5giThis excerpt is from the StatefulSet definition. This file contains the VolumeClaim template that will dynamically provision storage, using the StorageOS storage class. Dynamic provisioning occurs as a volumeMount has been declared with the same name as a VolumeClaim.\n  Move into the MySQL examples folder and create the objects\ncd storageos-usecases kubectl create -f ./mysql   Confirm MySQL is up and running.\n$ kubectl get pods -w -l app=mysql NAME READY STATUS RESTARTS AGE mysql-0 1/1 Running 0 1m   Connect to the MySQL client pod and connect to the MySQL server through the service\n$ kubectl exec client -- mysql -h mysql-0.mysql -e \u0026#34;show databases;\u0026#34; Database information_schema mysql performance_schema   Configuration If you need custom startup options, you can edit the ConfigMap file 15-mysqld-configmap.yaml with your desired MySQL configuration settings.\n","excerpt":"MySQL with StorageOS MySQL is a popular SQL open source database for a wide range of popular …","ref":"/docs/usecases/mysql/","title":""},{"body":"StorageOS namespaces are an identical concept to Kubernetes namespaces. They are intended to allow a StorageOS cluster to be used by multiple teams across multiple projects.\nIt is not necessary to create StorageOS namespaces manually, as StorageOS maps Kubernetes namespaces on a one-to-one basis when PersistentVolumeClaims using the StorageOS StorageClass are created.\nAccess to Namespaces is controlled through user or group level policies.\n","excerpt":"StorageOS namespaces are an identical concept to Kubernetes namespaces. They are intended to allow a …","ref":"/docs/concepts/namespaces/","title":"Namespaces"},{"body":"Namespaces help different projects or teams share a StorageOS cluster. Only the default namespace is created by default.\nNamespaces apply to volumes.\nManaging Namespaces In order to create a new namespace navigate to \u0026ldquo;Namespaces\u0026rdquo; in the GUI, and select \u0026ldquo;Create Namespace\u0026rdquo;.\nWhen a Kubernetes PVC is created in a namespace, StorageOS automatically maps the Volume in the same namespace. Namespaces are created by StorageOS to fulfil the RBAC rules enforced by Kubernetes roles.\nIn order to delete a namespace, all volumes must be deleted from the namespace before the namespace can be deleted.\n","excerpt":"Namespaces help different projects or teams share a StorageOS cluster. Only the default namespace is …","ref":"/docs/operations/namespaces/","title":"Namespaces"},{"body":"Nginx with StorageOS  Nginx is a popular web server that can be used as a reverse proxy, load balancer or even as a Kubernetes ingress controller.\nBefore you start, ensure you have StorageOS installed and ready on a Kubernetes cluster. See our guide on how to install StorageOS on Kubernetes for more information.\nDeploying Nginx on Kubernetes   You can find the latest files in the StorageOS use cases repostiory\ngit clone https://github.com/storageos/use-cases.git storageos-usecases StatefulSet definition\napiVersion:apps/v1kind:StatefulSetmetadata:name:nginxspec:serviceName:nginxspec:serviceAccountName:nginxcontainers:- name:nginximage:nginxports:- containerPort:80volumeMounts:- name:nginx-datamountPath:/usr/share/nginx/htmlsubPath:html- name:nginx-configmountPath:/etc/nginx/conf.dvolumes:- name:nginx-configconfigMap:name:nginxvolumeClaimTemplates:- metadata:name:nginx-dataspec:accessModes:[\u0026#34;ReadWriteOnce\u0026#34;]storageClassName:\u0026#34;fast\u0026#34;# StorageOS storageClass resources:requests:storage:5GiThis excerpt is from the StatefulSet definition. This file contains the VolumeClaim template that will dynamically provision storage, using the StorageOS storage class. Dynamic provisioning occurs as a volumeMount has been declared with the same name as a VolumeClaim.\n  Move into the Nginx examples folder and create the objects\ncd storageos-usecases kubectl create -f ./nginx   Confirm Nginx is up and running.\n$ kubectl get pods -w -l app=nginx NAME READY STATUS RESTARTS AGE nginx-0 1/1 Running 0 1m   Connect to the nginx pod and write a file to /usr/share/nginx/html that Nginx will serve.\n$ kubectl exec nginx-0 -it -- bash root@nginx-0:/# echo Hello world! \u0026gt; /usr/share/nginx/html/greetings.txt   Connect to the Busybox pod and connect to the Nginx server through the service and retrieve the directory index from Nginx.\n$ kubectl exec -it busybox -- /bin/sh / # wget -q -O- nginx \u0026lt;html\u0026gt; \u0026lt;head\u0026gt;\u0026lt;title\u0026gt;Index of /\u0026lt;/title\u0026gt;\u0026lt;/head\u0026gt; \u0026lt;body\u0026gt; \u0026lt;h1\u0026gt;Index of /\u0026lt;/h1\u0026gt;\u0026lt;hr\u0026gt;\u0026lt;pre\u0026gt;\u0026lt;a href=\u0026#34;../\u0026#34;\u0026gt;../\u0026lt;/a\u0026gt; \u0026lt;a href=\u0026#34;greetings.txt\u0026#34;\u0026gt;greetings.txt\u0026lt;/a\u0026gt; 27-Feb-2019 12:04 13 \u0026lt;/pre\u0026gt;\u0026lt;hr\u0026gt;\u0026lt;/body\u0026gt; \u0026lt;/html\u0026gt;   Retrieve and display the contents of the greetings.txt file\n/ # wget -q -O- nginx/greetings.txt Hello world!   Configuration If you need custom startup options, you can edit the ConfigMap file 15-nginx-configmap.yaml with your desired Nginx configuration settings.\n","excerpt":"Nginx with StorageOS  Nginx is a popular web server that can be used as a reverse proxy, load …","ref":"/docs/usecases/nginx/","title":""},{"body":"A StorageOS node is any machine (virtual or physical) that is running the StorageOS daemonset pod. A node must be running a daemonset pod in order to consume and/or present storage.\nBy default StorageOS nodes run in hyperconverged mode. This means that the node hosts data from StorageOS volumes and can present volumes to applications.\nAlternatively, a node can run in computeonly mode, which means no storage is consumed on the node itself and the node only presents volumes hosted by other nodes. Volumes presented to applications running on compute only nodes are therefore all remote. Compute only nodes can be very useful for topologies where nodes are ephemeral and should not host data, but the ephemeral nodes host applications that require StorageOS volumes. The nodes that are not intended to hold data, but just to present StorageOS volumes, can be set as computeonly.\nA node can be marked as compute only at any point in time by adding the label storageos.com/computeonly=true, following the labels reference.\n","excerpt":"A StorageOS node is any machine (virtual or physical) that is running the StorageOS daemonset pod. A …","ref":"/docs/concepts/nodes/","title":"Nodes"},{"body":"The following document lists the open source software attributions in the StorageOS Control Plane, Data Plane and CLI.\nCNats The MIT License (MIT) Copyright (c) 2015 Apcera Inc. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \u0026quot;Software\u0026quot;), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \u0026quot;AS IS\u0026quot;, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. CppUTest  Copyright (c) 2007, Michael Feathers, James Grenning and Bas Vodde All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of the \u0026lt;organization\u0026gt; nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE EARLIER MENTIONED AUTHORS ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL \u0026lt;copyright holder\u0026gt; BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. Hayai Copyright (c) 2011 - Nick Bruun. This software is provided 'as-is', without any express or implied warranty. In no event will the authors be held liable for any damages arising from the use of this software. Permission is granted to anyone to use this software for any purpose, including commercial applications, and to alter it and redistribute it freely, subject to the following restrictions: The origin of this software must not be misrepresented; you must not claim that you wrote the original software. If you use this software in a product, an acknowledgment in the product documentation would be appreciated but is not required. Altered source versions must be plainly marked as such, and must not be misrepresented as being the original software. If you meet (any of) the author(s), you're encouraged to buy them a beer, a drink or whatever is suited to the situation, given that you like the software. This notice may not be removed or altered from any source distribution. LZ4 LZ4 Library Copyright (c) 2011-2016, Yann Collet All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \u0026quot;AS IS\u0026quot; AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. LibreSSL  LICENSE ISSUES ============== The OpenSSL toolkit stays under a dual license, i.e. both the conditions of the OpenSSL License and the original SSLeay license apply to the toolkit. See below for the actual license texts. Actually both licenses are BSD-style Open Source licenses. In case of any license issues related to OpenSSL please contact openssl-core@openssl.org. OpenSSL License --------------- ==================================================================== Copyright (c) 1998-2011 The OpenSSL Project. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. 3. All advertising materials mentioning features or use of this software must display the following acknowledgment: \u0026quot;This product includes software developed by the OpenSSL Project for use in the OpenSSL Toolkit. (http://www.openssl.org/)\u0026quot; 4. The names \u0026quot;OpenSSL Toolkit\u0026quot; and \u0026quot;OpenSSL Project\u0026quot; must not be used to endorse or promote products derived from this software without prior written permission. For written permission, please contact openssl-core@openssl.org. 5. Products derived from this software may not be called \u0026quot;OpenSSL\u0026quot; nor may \u0026quot;OpenSSL\u0026quot; appear in their names without prior written permission of the OpenSSL Project. 6. Redistributions of any form whatsoever must retain the following acknowledgment: \u0026quot;This product includes software developed by the OpenSSL Project for use in the OpenSSL Toolkit (http://www.openssl.org/)\u0026quot; THIS SOFTWARE IS PROVIDED BY THE OpenSSL PROJECT ``AS IS'' AND ANY EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE OpenSSL PROJECT OR ITS CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. ==================================================================== This product includes cryptographic software written by Eric Young (eay@cryptsoft.com). This product includes software written by Tim Hudson (tjh@cryptsoft.com). Original SSLeay License ----------------------- Copyright (C) 1995-1998 Eric Young (eay@cryptsoft.com) All rights reserved. This package is an SSL implementation written by Eric Young (eay@cryptsoft.com). The implementation was written so as to conform with Netscapes SSL. This library is free for commercial and non-commercial use as long as the following conditions are aheared to. The following conditions apply to all code found in this distribution, be it the RC4, RSA, lhash, DES, etc., code; not just the SSL code. The SSL documentation included with this distribution is covered by the same copyright terms except that the holder is Tim Hudson (tjh@cryptsoft.com). Copyright remains Eric Young's, and as such any Copyright notices in the code are not to be removed. If this package is used in a product, Eric Young should be given attribution as the author of the parts of the library used. This can be in the form of a textual message at program startup or in documentation (online or textual) provided with the package. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. 3. All advertising materials mentioning features or use of this software must display the following acknowledgement: \u0026quot;This product includes cryptographic software written by Eric Young (eay@cryptsoft.com)\u0026quot; The word 'cryptographic' can be left out if the rouines from the library being used are not cryptographic related :-). 4. If you include any Windows specific code (or a derivative thereof) from the apps directory (application code) you must include an acknowledgement: \u0026quot;This product includes software written by Tim Hudson (tjh@cryptsoft.com)\u0026quot; THIS SOFTWARE IS PROVIDED BY ERIC YOUNG ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. The licence and distribution terms for any publically available version or derivative of this code cannot be changed. i.e. this code cannot simply be copied and put under another distribution licence [including the GNU Public Licence.] protobuf This license applies to all parts of Protocol Buffers except the following: - Atomicops support for generic gcc, located in src/google/protobuf/stubs/atomicops_internals_generic_gcc.h. This file is copyrighted by Red Hat Inc. - Atomicops support for AIX/POWER, located in src/google/protobuf/stubs/atomicops_internals_power.h. This file is copyrighted by Bloomberg Finance LP. Copyright 2014, Google Inc. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of Google Inc. nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \u0026quot;AS IS\u0026quot; AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. Code generated by the Protocol Buffer compiler is owned by the owner of the input file used when generating it. This code is not standalone and requires a support library to be linked with it. This support library is itself covered by the above license. rocksdb BSD License For rocksdb software Copyright (c) 2011-present, Facebook, Inc. All rights reserved. --------------------------------------------------------------------- Copyright (c) 2011 The LevelDB Authors. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of Google Inc. nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \u0026quot;AS IS\u0026quot; AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. github.com/armon/go-metrics The MIT License (MIT) Copyright (c) 2013 Armon Dadgar Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \u0026quot;Software\u0026quot;), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \u0026quot;AS IS\u0026quot;, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. github.com/armon/go-radix The MIT License (MIT) Copyright (c) 2014 Armon Dadgar Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \u0026quot;Software\u0026quot;), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \u0026quot;AS IS\u0026quot;, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. github.com/bgentry/speakeasy  Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. \u0026quot;License\u0026quot; shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. \u0026quot;Licensor\u0026quot; shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. \u0026quot;Legal Entity\u0026quot; shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, \u0026quot;control\u0026quot; means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. \u0026quot;You\u0026quot; (or \u0026quot;Your\u0026quot;) shall mean an individual or Legal Entity exercising permissions granted by this License. \u0026quot;Source\u0026quot; form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. \u0026quot;Object\u0026quot; form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. \u0026quot;Work\u0026quot; shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). \u0026quot;Derivative Works\u0026quot; shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. \u0026quot;Contribution\u0026quot; shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, \u0026quot;submitted\u0026quot; means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as \u0026quot;Not a Contribution.\u0026quot; \u0026quot;Contributor\u0026quot; shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a \u0026quot;NOTICE\u0026quot; text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an \u0026quot;AS IS\u0026quot; BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work. To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets \u0026quot;[]\u0026quot; replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same \u0026quot;printed page\u0026quot; as the copyright notice for easier identification within third-party archives. Copyright [2013] [the CloudFoundry Authors] Licensed under the Apache License, Version 2.0 (the \u0026quot;License\u0026quot;); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \u0026quot;AS IS\u0026quot; BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. github.com/dgrijalva/jwt-go Copyright (c) 2012 Dave Grijalva Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \u0026quot;Software\u0026quot;), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \u0026quot;AS IS\u0026quot;, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. github.com/dgrijalva/jwt-go/request Copyright (c) 2012 Dave Grijalva Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \u0026quot;Software\u0026quot;), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \u0026quot;AS IS\u0026quot;, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. github.com/dgryski/go-metro MetroHash This package is a mechanical translation of the reference C++ code for MetroHash, available at https://github.com/jandrewrogers/MetroHash I claim no additional copyright over the original implementation. The MIT License (MIT) Copyright (c) 2015 J. Andrew Rogers Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \u0026quot;Software\u0026quot;), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \u0026quot;AS IS\u0026quot;, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. github.com/docker/docker/pkg/mount  Apache License Version 2.0, January 2004 https://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. \u0026quot;License\u0026quot; shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. \u0026quot;Licensor\u0026quot; shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. \u0026quot;Legal Entity\u0026quot; shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, \u0026quot;control\u0026quot; means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. \u0026quot;You\u0026quot; (or \u0026quot;Your\u0026quot;) shall mean an individual or Legal Entity exercising permissions granted by this License. \u0026quot;Source\u0026quot; form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. \u0026quot;Object\u0026quot; form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. \u0026quot;Work\u0026quot; shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). \u0026quot;Derivative Works\u0026quot; shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. \u0026quot;Contribution\u0026quot; shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, \u0026quot;submitted\u0026quot; means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as \u0026quot;Not a Contribution.\u0026quot; \u0026quot;Contributor\u0026quot; shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a \u0026quot;NOTICE\u0026quot; text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an \u0026quot;AS IS\u0026quot; BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS Copyright 2013-2016 Docker, Inc. Licensed under the Apache License, Version 2.0 (the \u0026quot;License\u0026quot;); you may not use this file except in compliance with the License. You may obtain a copy of the License at https://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \u0026quot;AS IS\u0026quot; BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. github.com/docker/libkv  Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. \u0026quot;License\u0026quot; shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. \u0026quot;Licensor\u0026quot; shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. \u0026quot;Legal Entity\u0026quot; shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, \u0026quot;control\u0026quot; means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. \u0026quot;You\u0026quot; (or \u0026quot;Your\u0026quot;) shall mean an individual or Legal Entity exercising permissions granted by this License. \u0026quot;Source\u0026quot; form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. \u0026quot;Object\u0026quot; form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. \u0026quot;Work\u0026quot; shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). \u0026quot;Derivative Works\u0026quot; shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. \u0026quot;Contribution\u0026quot; shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, \u0026quot;submitted\u0026quot; means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as \u0026quot;Not a Contribution.\u0026quot; \u0026quot;Contributor\u0026quot; shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a \u0026quot;NOTICE\u0026quot; text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an \u0026quot;AS IS\u0026quot; BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS Copyright 2014-2016 Docker, Inc. Licensed under the Apache License, Version 2.0 (the \u0026quot;License\u0026quot;); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \u0026quot;AS IS\u0026quot; BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. github.com/docker/libkv/store  Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. \u0026quot;License\u0026quot; shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. \u0026quot;Licensor\u0026quot; shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. \u0026quot;Legal Entity\u0026quot; shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, \u0026quot;control\u0026quot; means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. \u0026quot;You\u0026quot; (or \u0026quot;Your\u0026quot;) shall mean an individual or Legal Entity exercising permissions granted by this License. \u0026quot;Source\u0026quot; form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. \u0026quot;Object\u0026quot; form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. \u0026quot;Work\u0026quot; shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). \u0026quot;Derivative Works\u0026quot; shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. \u0026quot;Contribution\u0026quot; shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, \u0026quot;submitted\u0026quot; means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as \u0026quot;Not a Contribution.\u0026quot; \u0026quot;Contributor\u0026quot; shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a \u0026quot;NOTICE\u0026quot; text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an \u0026quot;AS IS\u0026quot; BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS Copyright 2014-2016 Docker, Inc. Licensed under the Apache License, Version 2.0 (the \u0026quot;License\u0026quot;); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \u0026quot;AS IS\u0026quot; BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. github.com/fsnotify/fsnotify Copyright (c) 2012 The Go Authors. All rights reserved. Copyright (c) 2012 fsnotify Authors. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of Google Inc. nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \u0026quot;AS IS\u0026quot; AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. github.com/golang/protobuf/proto Go support for Protocol Buffers - Google's data interchange format Copyright 2010 The Go Authors. All rights reserved. https://github.com/golang/protobuf Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of Google Inc. nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \u0026quot;AS IS\u0026quot; AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. github.com/gorilla/context Copyright (c) 2012 Rodrigo Moraes. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of Google Inc. nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \u0026quot;AS IS\u0026quot; AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. github.com/gorilla/handlers Copyright (c) 2013 The Gorilla Handlers Authors. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \u0026quot;AS IS\u0026quot; AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. github.com/gorilla/mux Copyright (c) 2012 Rodrigo Moraes. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of Google Inc. nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \u0026quot;AS IS\u0026quot; AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. github.com/gorilla/websocket Copyright (c) 2013 The Gorilla WebSocket Authors. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \u0026quot;AS IS\u0026quot; AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. github.com/hashicorp/consul/api Mozilla Public License, version 2.0 1. Definitions 1.1. “Contributor” means each individual or legal entity that creates, contributes to the creation of, or owns Covered Software. 1.2. “Contributor Version” means the combination of the Contributions of others (if any) used by a Contributor and that particular Contributor’s Contribution. 1.3. “Contribution” means Covered Software of a particular Contributor. 1.4. “Covered Software” means Source Code Form to which the initial Contributor has attached the notice in Exhibit A, the Executable Form of such Source Code Form, and Modifications of such Source Code Form, in each case including portions thereof. 1.5. “Incompatible With Secondary Licenses” means a. that the initial Contributor has attached the notice described in Exhibit B to the Covered Software; or b. that the Covered Software was made available under the terms of version 1.1 or earlier of the License, but not also under the terms of a Secondary License. 1.6. “Executable Form” means any form of the work other than Source Code Form. 1.7. “Larger Work” means a work that combines Covered Software with other material, in a separate file or files, that is not Covered Software. 1.8. “License” means this document. 1.9. “Licensable” means having the right to grant, to the maximum extent possible, whether at the time of the initial grant or subsequently, any and all of the rights conveyed by this License. 1.10. “Modifications” means any of the following: a. any file in Source Code Form that results from an addition to, deletion from, or modification of the contents of Covered Software; or b. any new file in Source Code Form that contains any Covered Software. 1.11. “Patent Claims” of a Contributor means any patent claim(s), including without limitation, method, process, and apparatus claims, in any patent Licensable by such Contributor that would be infringed, but for the grant of the License, by the making, using, selling, offering for sale, having made, import, or transfer of either its Contributions or its Contributor Version. 1.12. “Secondary License” means either the GNU General Public License, Version 2.0, the GNU Lesser General Public License, Version 2.1, the GNU Affero General Public License, Version 3.0, or any later versions of those licenses. 1.13. “Source Code Form” means the form of the work preferred for making modifications. 1.14. “You” (or “Your”) means an individual or a legal entity exercising rights under this License. For legal entities, “You” includes any entity that controls, is controlled by, or is under common control with You. For purposes of this definition, “control” means (a) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (b) ownership of more than fifty percent (50%) of the outstanding shares or beneficial ownership of such entity. 2. License Grants and Conditions 2.1. Grants Each Contributor hereby grants You a world-wide, royalty-free, non-exclusive license: a. under intellectual property rights (other than patent or trademark) Licensable by such Contributor to use, reproduce, make available, modify, display, perform, distribute, and otherwise exploit its Contributions, either on an unmodified basis, with Modifications, or as part of a Larger Work; and b. under Patent Claims of such Contributor to make, use, sell, offer for sale, have made, import, and otherwise transfer either its Contributions or its Contributor Version. 2.2. Effective Date The licenses granted in Section 2.1 with respect to any Contribution become effective for each Contribution on the date the Contributor first distributes such Contribution. 2.3. Limitations on Grant Scope The licenses granted in this Section 2 are the only rights granted under this License. No additional rights or licenses will be implied from the distribution or licensing of Covered Software under this License. Notwithstanding Section 2.1(b) above, no patent license is granted by a Contributor: a. for any code that a Contributor has removed from Covered Software; or b. for infringements caused by: (i) Your and any other third party’s modifications of Covered Software, or (ii) the combination of its Contributions with other software (except as part of its Contributor Version); or c. under Patent Claims infringed by Covered Software in the absence of its Contributions. This License does not grant any rights in the trademarks, service marks, or logos of any Contributor (except as may be necessary to comply with the notice requirements in Section 3.4). 2.4. Subsequent Licenses No Contributor makes additional grants as a result of Your choice to distribute the Covered Software under a subsequent version of this License (see Section 10.2) or under the terms of a Secondary License (if permitted under the terms of Section 3.3). 2.5. Representation Each Contributor represents that the Contributor believes its Contributions are its original creation(s) or it has sufficient rights to grant the rights to its Contributions conveyed by this License. 2.6. Fair Use This License is not intended to limit any rights You have under applicable copyright doctrines of fair use, fair dealing, or other equivalents. 2.7. Conditions Sections 3.1, 3.2, 3.3, and 3.4 are conditions of the licenses granted in Section 2.1. 3. Responsibilities 3.1. Distribution of Source Form All distribution of Covered Software in Source Code Form, including any Modifications that You create or to which You contribute, must be under the terms of this License. You must inform recipients that the Source Code Form of the Covered Software is governed by the terms of this License, and how they can obtain a copy of this License. You may not attempt to alter or restrict the recipients’ rights in the Source Code Form. 3.2. Distribution of Executable Form If You distribute Covered Software in Executable Form then: a. such Covered Software must also be made available in Source Code Form, as described in Section 3.1, and You must inform recipients of the Executable Form how they can obtain a copy of such Source Code Form by reasonable means in a timely manner, at a charge no more than the cost of distribution to the recipient; and b. You may distribute such Executable Form under the terms of this License, or sublicense it under different terms, provided that the license for the Executable Form does not attempt to limit or alter the recipients’ rights in the Source Code Form under this License. 3.3. Distribution of a Larger Work You may create and distribute a Larger Work under terms of Your choice, provided that You also comply with the requirements of this License for the Covered Software. If the Larger Work is a combination of Covered Software with a work governed by one or more Secondary Licenses, and the Covered Software is not Incompatible With Secondary Licenses, this License permits You to additionally distribute such Covered Software under the terms of such Secondary License(s), so that the recipient of the Larger Work may, at their option, further distribute the Covered Software under the terms of either this License or such Secondary License(s). 3.4. Notices You may not remove or alter the substance of any license notices (including copyright notices, patent notices, disclaimers of warranty, or limitations of liability) contained within the Source Code Form of the Covered Software, except that You may alter any license notices to the extent required to remedy known factual inaccuracies. 3.5. Application of Additional Terms You may choose to offer, and to charge a fee for, warranty, support, indemnity or liability obligations to one or more recipients of Covered Software. However, You may do so only on Your own behalf, and not on behalf of any Contributor. You must make it absolutely clear that any such warranty, support, indemnity, or liability obligation is offered by You alone, and You hereby agree to indemnify every Contributor for any liability incurred by such Contributor as a result of warranty, support, indemnity or liability terms You offer. You may include additional disclaimers of warranty and limitations of liability specific to any jurisdiction. 4. Inability to Comply Due to Statute or Regulation If it is impossible for You to comply with any of the terms of this License with respect to some or all of the Covered Software due to statute, judicial order, or regulation then You must: (a) comply with the terms of this License to the maximum extent possible; and (b) describe the limitations and the code they affect. Such description must be placed in a text file included with all distributions of the Covered Software under this License. Except to the extent prohibited by statute or regulation, such description must be sufficiently detailed for a recipient of ordinary skill to be able to understand it. 5. Termination 5.1. The rights granted under this License will terminate automatically if You fail to comply with any of its terms. However, if You become compliant, then the rights granted under this License from a particular Contributor are reinstated (a) provisionally, unless and until such Contributor explicitly and finally terminates Your grants, and (b) on an ongoing basis, if such Contributor fails to notify You of the non-compliance by some reasonable means prior to 60 days after You have come back into compliance. Moreover, Your grants from a particular Contributor are reinstated on an ongoing basis if such Contributor notifies You of the non-compliance by some reasonable means, this is the first time You have received notice of non-compliance with this License from such Contributor, and You become compliant prior to 30 days after Your receipt of the notice. 5.2. If You initiate litigation against any entity by asserting a patent infringement claim (excluding declaratory judgment actions, counter-claims, and cross-claims) alleging that a Contributor Version directly or indirectly infringes any patent, then the rights granted to You by any and all Contributors for the Covered Software under Section 2.1 of this License shall terminate. 5.3. In the event of termination under Sections 5.1 or 5.2 above, all end user license agreements (excluding distributors and resellers) which have been validly granted by You or Your distributors under this License prior to termination shall survive termination. 6. Disclaimer of Warranty Covered Software is provided under this License on an “as is” basis, without warranty of any kind, either expressed, implied, or statutory, including, without limitation, warranties that the Covered Software is free of defects, merchantable, fit for a particular purpose or non-infringing. The entire risk as to the quality and performance of the Covered Software is with You. Should any Covered Software prove defective in any respect, You (not any Contributor) assume the cost of any necessary servicing, repair, or correction. This disclaimer of warranty constitutes an essential part of this License. No use of any Covered Software is authorized under this License except under this disclaimer. 7. Limitation of Liability Under no circumstances and under no legal theory, whether tort (including negligence), contract, or otherwise, shall any Contributor, or anyone who distributes Covered Software as permitted above, be liable to You for any direct, indirect, special, incidental, or consequential damages of any character including, without limitation, damages for lost profits, loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses, even if such party shall have been informed of the possibility of such damages. This limitation of liability shall not apply to liability for death or personal injury resulting from such party’s negligence to the extent applicable law prohibits such limitation. Some jurisdictions do not allow the exclusion or limitation of incidental or consequential damages, so this exclusion and limitation may not apply to You. 8. Litigation Any litigation relating to this License may be brought only in the courts of a jurisdiction where the defendant maintains its principal place of business and such litigation shall be governed by laws of that jurisdiction, without reference to its conflict-of-law provisions. Nothing in this Section shall prevent a party’s ability to bring cross-claims or counter-claims. 9. Miscellaneous This License represents the complete agreement concerning the subject matter hereof. If any provision of this License is held to be unenforceable, such provision shall be reformed only to the extent necessary to make it enforceable. Any law or regulation which provides that the language of a contract shall be construed against the drafter shall not be used to construe this License against a Contributor. 10. Versions of the License 10.1. New Versions Mozilla Foundation is the license steward. Except as provided in Section 10.3, no one other than the license steward has the right to modify or publish new versions of this License. Each version will be given a distinguishing version number. 10.2. Effect of New Versions You may distribute the Covered Software under the terms of the version of the License under which You originally received the Covered Software, or under the terms of any subsequent version published by the license steward. 10.3. Modified Versions If you create software not governed by this License, and you want to create a new license for such software, you may create and use a modified version of this License if you rename the license and remove any references to the name of the license steward (except to note that such modified license differs from this License). 10.4. Distributing Source Code Form that is Incompatible With Secondary Licenses If You choose to distribute Source Code Form that is Incompatible With Secondary Licenses under the terms of this version of the License, the notice described in Exhibit B of this License must be attached. Exhibit A - Source Code Form License Notice This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain one at http://mozilla.org/MPL/2.0/. If it is not possible or desirable to put the notice in a particular file, then You may include the notice in a location (such as a LICENSE file in a relevant directory) where a recipient would be likely to look for such a notice. You may add additional accurate notices of copyright ownership. Exhibit B - “Incompatible With Secondary Licenses” Notice This Source Code Form is “Incompatible With Secondary Licenses”, as defined by the Mozilla Public License, v. 2.0. github.com/hashicorp/errwrap Mozilla Public License, version 2.0 1. Definitions 1.1. “Contributor” means each individual or legal entity that creates, contributes to the creation of, or owns Covered Software. 1.2. “Contributor Version” means the combination of the Contributions of others (if any) used by a Contributor and that particular Contributor’s Contribution. 1.3. “Contribution” means Covered Software of a particular Contributor. 1.4. “Covered Software” means Source Code Form to which the initial Contributor has attached the notice in Exhibit A, the Executable Form of such Source Code Form, and Modifications of such Source Code Form, in each case including portions thereof. 1.5. “Incompatible With Secondary Licenses” means a. that the initial Contributor has attached the notice described in Exhibit B to the Covered Software; or b. that the Covered Software was made available under the terms of version 1.1 or earlier of the License, but not also under the terms of a Secondary License. 1.6. “Executable Form” means any form of the work other than Source Code Form. 1.7. “Larger Work” means a work that combines Covered Software with other material, in a separate file or files, that is not Covered Software. 1.8. “License” means this document. 1.9. “Licensable” means having the right to grant, to the maximum extent possible, whether at the time of the initial grant or subsequently, any and all of the rights conveyed by this License. 1.10. “Modifications” means any of the following: a. any file in Source Code Form that results from an addition to, deletion from, or modification of the contents of Covered Software; or b. any new file in Source Code Form that contains any Covered Software. 1.11. “Patent Claims” of a Contributor means any patent claim(s), including without limitation, method, process, and apparatus claims, in any patent Licensable by such Contributor that would be infringed, but for the grant of the License, by the making, using, selling, offering for sale, having made, import, or transfer of either its Contributions or its Contributor Version. 1.12. “Secondary License” means either the GNU General Public License, Version 2.0, the GNU Lesser General Public License, Version 2.1, the GNU Affero General Public License, Version 3.0, or any later versions of those licenses. 1.13. “Source Code Form” means the form of the work preferred for making modifications. 1.14. “You” (or “Your”) means an individual or a legal entity exercising rights under this License. For legal entities, “You” includes any entity that controls, is controlled by, or is under common control with You. For purposes of this definition, “control” means (a) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (b) ownership of more than fifty percent (50%) of the outstanding shares or beneficial ownership of such entity. 2. License Grants and Conditions 2.1. Grants Each Contributor hereby grants You a world-wide, royalty-free, non-exclusive license: a. under intellectual property rights (other than patent or trademark) Licensable by such Contributor to use, reproduce, make available, modify, display, perform, distribute, and otherwise exploit its Contributions, either on an unmodified basis, with Modifications, or as part of a Larger Work; and b. under Patent Claims of such Contributor to make, use, sell, offer for sale, have made, import, and otherwise transfer either its Contributions or its Contributor Version. 2.2. Effective Date The licenses granted in Section 2.1 with respect to any Contribution become effective for each Contribution on the date the Contributor first distributes such Contribution. 2.3. Limitations on Grant Scope The licenses granted in this Section 2 are the only rights granted under this License. No additional rights or licenses will be implied from the distribution or licensing of Covered Software under this License. Notwithstanding Section 2.1(b) above, no patent license is granted by a Contributor: a. for any code that a Contributor has removed from Covered Software; or b. for infringements caused by: (i) Your and any other third party’s modifications of Covered Software, or (ii) the combination of its Contributions with other software (except as part of its Contributor Version); or c. under Patent Claims infringed by Covered Software in the absence of its Contributions. This License does not grant any rights in the trademarks, service marks, or logos of any Contributor (except as may be necessary to comply with the notice requirements in Section 3.4). 2.4. Subsequent Licenses No Contributor makes additional grants as a result of Your choice to distribute the Covered Software under a subsequent version of this License (see Section 10.2) or under the terms of a Secondary License (if permitted under the terms of Section 3.3). 2.5. Representation Each Contributor represents that the Contributor believes its Contributions are its original creation(s) or it has sufficient rights to grant the rights to its Contributions conveyed by this License. 2.6. Fair Use This License is not intended to limit any rights You have under applicable copyright doctrines of fair use, fair dealing, or other equivalents. 2.7. Conditions Sections 3.1, 3.2, 3.3, and 3.4 are conditions of the licenses granted in Section 2.1. 3. Responsibilities 3.1. Distribution of Source Form All distribution of Covered Software in Source Code Form, including any Modifications that You create or to which You contribute, must be under the terms of this License. You must inform recipients that the Source Code Form of the Covered Software is governed by the terms of this License, and how they can obtain a copy of this License. You may not attempt to alter or restrict the recipients’ rights in the Source Code Form. 3.2. Distribution of Executable Form If You distribute Covered Software in Executable Form then: a. such Covered Software must also be made available in Source Code Form, as described in Section 3.1, and You must inform recipients of the Executable Form how they can obtain a copy of such Source Code Form by reasonable means in a timely manner, at a charge no more than the cost of distribution to the recipient; and b. You may distribute such Executable Form under the terms of this License, or sublicense it under different terms, provided that the license for the Executable Form does not attempt to limit or alter the recipients’ rights in the Source Code Form under this License. 3.3. Distribution of a Larger Work You may create and distribute a Larger Work under terms of Your choice, provided that You also comply with the requirements of this License for the Covered Software. If the Larger Work is a combination of Covered Software with a work governed by one or more Secondary Licenses, and the Covered Software is not Incompatible With Secondary Licenses, this License permits You to additionally distribute such Covered Software under the terms of such Secondary License(s), so that the recipient of the Larger Work may, at their option, further distribute the Covered Software under the terms of either this License or such Secondary License(s). 3.4. Notices You may not remove or alter the substance of any license notices (including copyright notices, patent notices, disclaimers of warranty, or limitations of liability) contained within the Source Code Form of the Covered Software, except that You may alter any license notices to the extent required to remedy known factual inaccuracies. 3.5. Application of Additional Terms You may choose to offer, and to charge a fee for, warranty, support, indemnity or liability obligations to one or more recipients of Covered Software. However, You may do so only on Your own behalf, and not on behalf of any Contributor. You must make it absolutely clear that any such warranty, support, indemnity, or liability obligation is offered by You alone, and You hereby agree to indemnify every Contributor for any liability incurred by such Contributor as a result of warranty, support, indemnity or liability terms You offer. You may include additional disclaimers of warranty and limitations of liability specific to any jurisdiction. 4. Inability to Comply Due to Statute or Regulation If it is impossible for You to comply with any of the terms of this License with respect to some or all of the Covered Software due to statute, judicial order, or regulation then You must: (a) comply with the terms of this License to the maximum extent possible; and (b) describe the limitations and the code they affect. Such description must be placed in a text file included with all distributions of the Covered Software under this License. Except to the extent prohibited by statute or regulation, such description must be sufficiently detailed for a recipient of ordinary skill to be able to understand it. 5. Termination 5.1. The rights granted under this License will terminate automatically if You fail to comply with any of its terms. However, if You become compliant, then the rights granted under this License from a particular Contributor are reinstated (a) provisionally, unless and until such Contributor explicitly and finally terminates Your grants, and (b) on an ongoing basis, if such Contributor fails to notify You of the non-compliance by some reasonable means prior to 60 days after You have come back into compliance. Moreover, Your grants from a particular Contributor are reinstated on an ongoing basis if such Contributor notifies You of the non-compliance by some reasonable means, this is the first time You have received notice of non-compliance with this License from such Contributor, and You become compliant prior to 30 days after Your receipt of the notice. 5.2. If You initiate litigation against any entity by asserting a patent infringement claim (excluding declaratory judgment actions, counter-claims, and cross-claims) alleging that a Contributor Version directly or indirectly infringes any patent, then the rights granted to You by any and all Contributors for the Covered Software under Section 2.1 of this License shall terminate. 5.3. In the event of termination under Sections 5.1 or 5.2 above, all end user license agreements (excluding distributors and resellers) which have been validly granted by You or Your distributors under this License prior to termination shall survive termination. 6. Disclaimer of Warranty Covered Software is provided under this License on an “as is” basis, without warranty of any kind, either expressed, implied, or statutory, including, without limitation, warranties that the Covered Software is free of defects, merchantable, fit for a particular purpose or non-infringing. The entire risk as to the quality and performance of the Covered Software is with You. Should any Covered Software prove defective in any respect, You (not any Contributor) assume the cost of any necessary servicing, repair, or correction. This disclaimer of warranty constitutes an essential part of this License. No use of any Covered Software is authorized under this License except under this disclaimer. 7. Limitation of Liability Under no circumstances and under no legal theory, whether tort (including negligence), contract, or otherwise, shall any Contributor, or anyone who distributes Covered Software as permitted above, be liable to You for any direct, indirect, special, incidental, or consequential damages of any character including, without limitation, damages for lost profits, loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses, even if such party shall have been informed of the possibility of such damages. This limitation of liability shall not apply to liability for death or personal injury resulting from such party’s negligence to the extent applicable law prohibits such limitation. Some jurisdictions do not allow the exclusion or limitation of incidental or consequential damages, so this exclusion and limitation may not apply to You. 8. Litigation Any litigation relating to this License may be brought only in the courts of a jurisdiction where the defendant maintains its principal place of business and such litigation shall be governed by laws of that jurisdiction, without reference to its conflict-of-law provisions. Nothing in this Section shall prevent a party’s ability to bring cross-claims or counter-claims. 9. Miscellaneous This License represents the complete agreement concerning the subject matter hereof. If any provision of this License is held to be unenforceable, such provision shall be reformed only to the extent necessary to make it enforceable. Any law or regulation which provides that the language of a contract shall be construed against the drafter shall not be used to construe this License against a Contributor. 10. Versions of the License 10.1. New Versions Mozilla Foundation is the license steward. Except as provided in Section 10.3, no one other than the license steward has the right to modify or publish new versions of this License. Each version will be given a distinguishing version number. 10.2. Effect of New Versions You may distribute the Covered Software under the terms of the version of the License under which You originally received the Covered Software, or under the terms of any subsequent version published by the license steward. 10.3. Modified Versions If you create software not governed by this License, and you want to create a new license for such software, you may create and use a modified version of this License if you rename the license and remove any references to the name of the license steward (except to note that such modified license differs from this License). 10.4. Distributing Source Code Form that is Incompatible With Secondary Licenses If You choose to distribute Source Code Form that is Incompatible With Secondary Licenses under the terms of this version of the License, the notice described in Exhibit B of this License must be attached. Exhibit A - Source Code Form License Notice This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain one at http://mozilla.org/MPL/2.0/. If it is not possible or desirable to put the notice in a particular file, then You may include the notice in a location (such as a LICENSE file in a relevant directory) where a recipient would be likely to look for such a notice. You may add additional accurate notices of copyright ownership. Exhibit B - “Incompatible With Secondary Licenses” Notice This Source Code Form is “Incompatible With Secondary Licenses”, as defined by the Mozilla Public License, v. 2.0. github.com/hashicorp/go-cleanhttp Mozilla Public License, version 2.0 1. Definitions 1.1. \u0026quot;Contributor\u0026quot; means each individual or legal entity that creates, contributes to the creation of, or owns Covered Software. 1.2. \u0026quot;Contributor Version\u0026quot; means the combination of the Contributions of others (if any) used by a Contributor and that particular Contributor's Contribution. 1.3. \u0026quot;Contribution\u0026quot; means Covered Software of a particular Contributor. 1.4. \u0026quot;Covered Software\u0026quot; means Source Code Form to which the initial Contributor has attached the notice in Exhibit A, the Executable Form of such Source Code Form, and Modifications of such Source Code Form, in each case including portions thereof. 1.5. \u0026quot;Incompatible With Secondary Licenses\u0026quot; means a. that the initial Contributor has attached the notice described in Exhibit B to the Covered Software; or b. that the Covered Software was made available under the terms of version 1.1 or earlier of the License, but not also under the terms of a Secondary License. 1.6. \u0026quot;Executable Form\u0026quot; means any form of the work other than Source Code Form. 1.7. \u0026quot;Larger Work\u0026quot; means a work that combines Covered Software with other material, in a separate file or files, that is not Covered Software. 1.8. \u0026quot;License\u0026quot; means this document. 1.9. \u0026quot;Licensable\u0026quot; means having the right to grant, to the maximum extent possible, whether at the time of the initial grant or subsequently, any and all of the rights conveyed by this License. 1.10. \u0026quot;Modifications\u0026quot; means any of the following: a. any file in Source Code Form that results from an addition to, deletion from, or modification of the contents of Covered Software; or b. any new file in Source Code Form that contains any Covered Software. 1.11. \u0026quot;Patent Claims\u0026quot; of a Contributor means any patent claim(s), including without limitation, method, process, and apparatus claims, in any patent Licensable by such Contributor that would be infringed, but for the grant of the License, by the making, using, selling, offering for sale, having made, import, or transfer of either its Contributions or its Contributor Version. 1.12. \u0026quot;Secondary License\u0026quot; means either the GNU General Public License, Version 2.0, the GNU Lesser General Public License, Version 2.1, the GNU Affero General Public License, Version 3.0, or any later versions of those licenses. 1.13. \u0026quot;Source Code Form\u0026quot; means the form of the work preferred for making modifications. 1.14. \u0026quot;You\u0026quot; (or \u0026quot;Your\u0026quot;) means an individual or a legal entity exercising rights under this License. For legal entities, \u0026quot;You\u0026quot; includes any entity that controls, is controlled by, or is under common control with You. For purposes of this definition, \u0026quot;control\u0026quot; means (a) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (b) ownership of more than fifty percent (50%) of the outstanding shares or beneficial ownership of such entity. 2. License Grants and Conditions 2.1. Grants Each Contributor hereby grants You a world-wide, royalty-free, non-exclusive license: a. under intellectual property rights (other than patent or trademark) Licensable by such Contributor to use, reproduce, make available, modify, display, perform, distribute, and otherwise exploit its Contributions, either on an unmodified basis, with Modifications, or as part of a Larger Work; and b. under Patent Claims of such Contributor to make, use, sell, offer for sale, have made, import, and otherwise transfer either its Contributions or its Contributor Version. 2.2. Effective Date The licenses granted in Section 2.1 with respect to any Contribution become effective for each Contribution on the date the Contributor first distributes such Contribution. 2.3. Limitations on Grant Scope The licenses granted in this Section 2 are the only rights granted under this License. No additional rights or licenses will be implied from the distribution or licensing of Covered Software under this License. Notwithstanding Section 2.1(b) above, no patent license is granted by a Contributor: a. for any code that a Contributor has removed from Covered Software; or b. for infringements caused by: (i) Your and any other third party's modifications of Covered Software, or (ii) the combination of its Contributions with other software (except as part of its Contributor Version); or c. under Patent Claims infringed by Covered Software in the absence of its Contributions. This License does not grant any rights in the trademarks, service marks, or logos of any Contributor (except as may be necessary to comply with the notice requirements in Section 3.4). 2.4. Subsequent Licenses No Contributor makes additional grants as a result of Your choice to distribute the Covered Software under a subsequent version of this License (see Section 10.2) or under the terms of a Secondary License (if permitted under the terms of Section 3.3). 2.5. Representation Each Contributor represents that the Contributor believes its Contributions are its original creation(s) or it has sufficient rights to grant the rights to its Contributions conveyed by this License. 2.6. Fair Use This License is not intended to limit any rights You have under applicable copyright doctrines of fair use, fair dealing, or other equivalents. 2.7. Conditions Sections 3.1, 3.2, 3.3, and 3.4 are conditions of the licenses granted in Section 2.1. 3. Responsibilities 3.1. Distribution of Source Form All distribution of Covered Software in Source Code Form, including any Modifications that You create or to which You contribute, must be under the terms of this License. You must inform recipients that the Source Code Form of the Covered Software is governed by the terms of this License, and how they can obtain a copy of this License. You may not attempt to alter or restrict the recipients' rights in the Source Code Form. 3.2. Distribution of Executable Form If You distribute Covered Software in Executable Form then: a. such Covered Software must also be made available in Source Code Form, as described in Section 3.1, and You must inform recipients of the Executable Form how they can obtain a copy of such Source Code Form by reasonable means in a timely manner, at a charge no more than the cost of distribution to the recipient; and b. You may distribute such Executable Form under the terms of this License, or sublicense it under different terms, provided that the license for the Executable Form does not attempt to limit or alter the recipients' rights in the Source Code Form under this License. 3.3. Distribution of a Larger Work You may create and distribute a Larger Work under terms of Your choice, provided that You also comply with the requirements of this License for the Covered Software. If the Larger Work is a combination of Covered Software with a work governed by one or more Secondary Licenses, and the Covered Software is not Incompatible With Secondary Licenses, this License permits You to additionally distribute such Covered Software under the terms of such Secondary License(s), so that the recipient of the Larger Work may, at their option, further distribute the Covered Software under the terms of either this License or such Secondary License(s). 3.4. Notices You may not remove or alter the substance of any license notices (including copyright notices, patent notices, disclaimers of warranty, or limitations of liability) contained within the Source Code Form of the Covered Software, except that You may alter any license notices to the extent required to remedy known factual inaccuracies. 3.5. Application of Additional Terms You may choose to offer, and to charge a fee for, warranty, support, indemnity or liability obligations to one or more recipients of Covered Software. However, You may do so only on Your own behalf, and not on behalf of any Contributor. You must make it absolutely clear that any such warranty, support, indemnity, or liability obligation is offered by You alone, and You hereby agree to indemnify every Contributor for any liability incurred by such Contributor as a result of warranty, support, indemnity or liability terms You offer. You may include additional disclaimers of warranty and limitations of liability specific to any jurisdiction. 4. Inability to Comply Due to Statute or Regulation If it is impossible for You to comply with any of the terms of this License with respect to some or all of the Covered Software due to statute, judicial order, or regulation then You must: (a) comply with the terms of this License to the maximum extent possible; and (b) describe the limitations and the code they affect. Such description must be placed in a text file included with all distributions of the Covered Software under this License. Except to the extent prohibited by statute or regulation, such description must be sufficiently detailed for a recipient of ordinary skill to be able to understand it. 5. Termination 5.1. The rights granted under this License will terminate automatically if You fail to comply with any of its terms. However, if You become compliant, then the rights granted under this License from a particular Contributor are reinstated (a) provisionally, unless and until such Contributor explicitly and finally terminates Your grants, and (b) on an ongoing basis, if such Contributor fails to notify You of the non-compliance by some reasonable means prior to 60 days after You have come back into compliance. Moreover, Your grants from a particular Contributor are reinstated on an ongoing basis if such Contributor notifies You of the non-compliance by some reasonable means, this is the first time You have received notice of non-compliance with this License from such Contributor, and You become compliant prior to 30 days after Your receipt of the notice. 5.2. If You initiate litigation against any entity by asserting a patent infringement claim (excluding declaratory judgment actions, counter-claims, and cross-claims) alleging that a Contributor Version directly or indirectly infringes any patent, then the rights granted to You by any and all Contributors for the Covered Software under Section 2.1 of this License shall terminate. 5.3. In the event of termination under Sections 5.1 or 5.2 above, all end user license agreements (excluding distributors and resellers) which have been validly granted by You or Your distributors under this License prior to termination shall survive termination. 6. Disclaimer of Warranty Covered Software is provided under this License on an \u0026quot;as is\u0026quot; basis, without warranty of any kind, either expressed, implied, or statutory, including, without limitation, warranties that the Covered Software is free of defects, merchantable, fit for a particular purpose or non-infringing. The entire risk as to the quality and performance of the Covered Software is with You. Should any Covered Software prove defective in any respect, You (not any Contributor) assume the cost of any necessary servicing, repair, or correction. This disclaimer of warranty constitutes an essential part of this License. No use of any Covered Software is authorized under this License except under this disclaimer. 7. Limitation of Liability Under no circumstances and under no legal theory, whether tort (including negligence), contract, or otherwise, shall any Contributor, or anyone who distributes Covered Software as permitted above, be liable to You for any direct, indirect, special, incidental, or consequential damages of any character including, without limitation, damages for lost profits, loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses, even if such party shall have been informed of the possibility of such damages. This limitation of liability shall not apply to liability for death or personal injury resulting from such party's negligence to the extent applicable law prohibits such limitation. Some jurisdictions do not allow the exclusion or limitation of incidental or consequential damages, so this exclusion and limitation may not apply to You. 8. Litigation Any litigation relating to this License may be brought only in the courts of a jurisdiction where the defendant maintains its principal place of business and such litigation shall be governed by laws of that jurisdiction, without reference to its conflict-of-law provisions. Nothing in this Section shall prevent a party's ability to bring cross-claims or counter-claims. 9. Miscellaneous This License represents the complete agreement concerning the subject matter hereof. If any provision of this License is held to be unenforceable, such provision shall be reformed only to the extent necessary to make it enforceable. Any law or regulation which provides that the language of a contract shall be construed against the drafter shall not be used to construe this License against a Contributor. 10. Versions of the License 10.1. New Versions Mozilla Foundation is the license steward. Except as provided in Section 10.3, no one other than the license steward has the right to modify or publish new versions of this License. Each version will be given a distinguishing version number. 10.2. Effect of New Versions You may distribute the Covered Software under the terms of the version of the License under which You originally received the Covered Software, or under the terms of any subsequent version published by the license steward. 10.3. Modified Versions If you create software not governed by this License, and you want to create a new license for such software, you may create and use a modified version of this License if you rename the license and remove any references to the name of the license steward (except to note that such modified license differs from this License). 10.4. Distributing Source Code Form that is Incompatible With Secondary Licenses If You choose to distribute Source Code Form that is Incompatible With Secondary Licenses under the terms of this version of the License, the notice described in Exhibit B of this License must be attached. Exhibit A - Source Code Form License Notice This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain one at http://mozilla.org/MPL/2.0/. If it is not possible or desirable to put the notice in a particular file, then You may include the notice in a location (such as a LICENSE file in a relevant directory) where a recipient would be likely to look for such a notice. You may add additional accurate notices of copyright ownership. Exhibit B - \u0026quot;Incompatible With Secondary Licenses\u0026quot; Notice This Source Code Form is \u0026quot;Incompatible With Secondary Licenses\u0026quot;, as defined by the Mozilla Public License, v. 2.0. github.com/hashicorp/go-msgpack/codec Copyright (c) 2012, 2013 Ugorji Nwoke. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of the author nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \u0026quot;AS IS\u0026quot; AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. github.com/hashicorp/go-multierror Mozilla Public License, version 2.0 1. Definitions 1.1. “Contributor” means each individual or legal entity that creates, contributes to the creation of, or owns Covered Software. 1.2. “Contributor Version” means the combination of the Contributions of others (if any) used by a Contributor and that particular Contributor’s Contribution. 1.3. “Contribution” means Covered Software of a particular Contributor. 1.4. “Covered Software” means Source Code Form to which the initial Contributor has attached the notice in Exhibit A, the Executable Form of such Source Code Form, and Modifications of such Source Code Form, in each case including portions thereof. 1.5. “Incompatible With Secondary Licenses” means a. that the initial Contributor has attached the notice described in Exhibit B to the Covered Software; or b. that the Covered Software was made available under the terms of version 1.1 or earlier of the License, but not also under the terms of a Secondary License. 1.6. “Executable Form” means any form of the work other than Source Code Form. 1.7. “Larger Work” means a work that combines Covered Software with other material, in a separate file or files, that is not Covered Software. 1.8. “License” means this document. 1.9. “Licensable” means having the right to grant, to the maximum extent possible, whether at the time of the initial grant or subsequently, any and all of the rights conveyed by this License. 1.10. “Modifications” means any of the following: a. any file in Source Code Form that results from an addition to, deletion from, or modification of the contents of Covered Software; or b. any new file in Source Code Form that contains any Covered Software. 1.11. “Patent Claims” of a Contributor means any patent claim(s), including without limitation, method, process, and apparatus claims, in any patent Licensable by such Contributor that would be infringed, but for the grant of the License, by the making, using, selling, offering for sale, having made, import, or transfer of either its Contributions or its Contributor Version. 1.12. “Secondary License” means either the GNU General Public License, Version 2.0, the GNU Lesser General Public License, Version 2.1, the GNU Affero General Public License, Version 3.0, or any later versions of those licenses. 1.13. “Source Code Form” means the form of the work preferred for making modifications. 1.14. “You” (or “Your”) means an individual or a legal entity exercising rights under this License. For legal entities, “You” includes any entity that controls, is controlled by, or is under common control with You. For purposes of this definition, “control” means (a) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (b) ownership of more than fifty percent (50%) of the outstanding shares or beneficial ownership of such entity. 2. License Grants and Conditions 2.1. Grants Each Contributor hereby grants You a world-wide, royalty-free, non-exclusive license: a. under intellectual property rights (other than patent or trademark) Licensable by such Contributor to use, reproduce, make available, modify, display, perform, distribute, and otherwise exploit its Contributions, either on an unmodified basis, with Modifications, or as part of a Larger Work; and b. under Patent Claims of such Contributor to make, use, sell, offer for sale, have made, import, and otherwise transfer either its Contributions or its Contributor Version. 2.2. Effective Date The licenses granted in Section 2.1 with respect to any Contribution become effective for each Contribution on the date the Contributor first distributes such Contribution. 2.3. Limitations on Grant Scope The licenses granted in this Section 2 are the only rights granted under this License. No additional rights or licenses will be implied from the distribution or licensing of Covered Software under this License. Notwithstanding Section 2.1(b) above, no patent license is granted by a Contributor: a. for any code that a Contributor has removed from Covered Software; or b. for infringements caused by: (i) Your and any other third party’s modifications of Covered Software, or (ii) the combination of its Contributions with other software (except as part of its Contributor Version); or c. under Patent Claims infringed by Covered Software in the absence of its Contributions. This License does not grant any rights in the trademarks, service marks, or logos of any Contributor (except as may be necessary to comply with the notice requirements in Section 3.4). 2.4. Subsequent Licenses No Contributor makes additional grants as a result of Your choice to distribute the Covered Software under a subsequent version of this License (see Section 10.2) or under the terms of a Secondary License (if permitted under the terms of Section 3.3). 2.5. Representation Each Contributor represents that the Contributor believes its Contributions are its original creation(s) or it has sufficient rights to grant the rights to its Contributions conveyed by this License. 2.6. Fair Use This License is not intended to limit any rights You have under applicable copyright doctrines of fair use, fair dealing, or other equivalents. 2.7. Conditions Sections 3.1, 3.2, 3.3, and 3.4 are conditions of the licenses granted in Section 2.1. 3. Responsibilities 3.1. Distribution of Source Form All distribution of Covered Software in Source Code Form, including any Modifications that You create or to which You contribute, must be under the terms of this License. You must inform recipients that the Source Code Form of the Covered Software is governed by the terms of this License, and how they can obtain a copy of this License. You may not attempt to alter or restrict the recipients’ rights in the Source Code Form. 3.2. Distribution of Executable Form If You distribute Covered Software in Executable Form then: a. such Covered Software must also be made available in Source Code Form, as described in Section 3.1, and You must inform recipients of the Executable Form how they can obtain a copy of such Source Code Form by reasonable means in a timely manner, at a charge no more than the cost of distribution to the recipient; and b. You may distribute such Executable Form under the terms of this License, or sublicense it under different terms, provided that the license for the Executable Form does not attempt to limit or alter the recipients’ rights in the Source Code Form under this License. 3.3. Distribution of a Larger Work You may create and distribute a Larger Work under terms of Your choice, provided that You also comply with the requirements of this License for the Covered Software. If the Larger Work is a combination of Covered Software with a work governed by one or more Secondary Licenses, and the Covered Software is not Incompatible With Secondary Licenses, this License permits You to additionally distribute such Covered Software under the terms of such Secondary License(s), so that the recipient of the Larger Work may, at their option, further distribute the Covered Software under the terms of either this License or such Secondary License(s). 3.4. Notices You may not remove or alter the substance of any license notices (including copyright notices, patent notices, disclaimers of warranty, or limitations of liability) contained within the Source Code Form of the Covered Software, except that You may alter any license notices to the extent required to remedy known factual inaccuracies. 3.5. Application of Additional Terms You may choose to offer, and to charge a fee for, warranty, support, indemnity or liability obligations to one or more recipients of Covered Software. However, You may do so only on Your own behalf, and not on behalf of any Contributor. You must make it absolutely clear that any such warranty, support, indemnity, or liability obligation is offered by You alone, and You hereby agree to indemnify every Contributor for any liability incurred by such Contributor as a result of warranty, support, indemnity or liability terms You offer. You may include additional disclaimers of warranty and limitations of liability specific to any jurisdiction. 4. Inability to Comply Due to Statute or Regulation If it is impossible for You to comply with any of the terms of this License with respect to some or all of the Covered Software due to statute, judicial order, or regulation then You must: (a) comply with the terms of this License to the maximum extent possible; and (b) describe the limitations and the code they affect. Such description must be placed in a text file included with all distributions of the Covered Software under this License. Except to the extent prohibited by statute or regulation, such description must be sufficiently detailed for a recipient of ordinary skill to be able to understand it. 5. Termination 5.1. The rights granted under this License will terminate automatically if You fail to comply with any of its terms. However, if You become compliant, then the rights granted under this License from a particular Contributor are reinstated (a) provisionally, unless and until such Contributor explicitly and finally terminates Your grants, and (b) on an ongoing basis, if such Contributor fails to notify You of the non-compliance by some reasonable means prior to 60 days after You have come back into compliance. Moreover, Your grants from a particular Contributor are reinstated on an ongoing basis if such Contributor notifies You of the non-compliance by some reasonable means, this is the first time You have received notice of non-compliance with this License from such Contributor, and You become compliant prior to 30 days after Your receipt of the notice. 5.2. If You initiate litigation against any entity by asserting a patent infringement claim (excluding declaratory judgment actions, counter-claims, and cross-claims) alleging that a Contributor Version directly or indirectly infringes any patent, then the rights granted to You by any and all Contributors for the Covered Software under Section 2.1 of this License shall terminate. 5.3. In the event of termination under Sections 5.1 or 5.2 above, all end user license agreements (excluding distributors and resellers) which have been validly granted by You or Your distributors under this License prior to termination shall survive termination. 6. Disclaimer of Warranty Covered Software is provided under this License on an “as is” basis, without warranty of any kind, either expressed, implied, or statutory, including, without limitation, warranties that the Covered Software is free of defects, merchantable, fit for a particular purpose or non-infringing. The entire risk as to the quality and performance of the Covered Software is with You. Should any Covered Software prove defective in any respect, You (not any Contributor) assume the cost of any necessary servicing, repair, or correction. This disclaimer of warranty constitutes an essential part of this License. No use of any Covered Software is authorized under this License except under this disclaimer. 7. Limitation of Liability Under no circumstances and under no legal theory, whether tort (including negligence), contract, or otherwise, shall any Contributor, or anyone who distributes Covered Software as permitted above, be liable to You for any direct, indirect, special, incidental, or consequential damages of any character including, without limitation, damages for lost profits, loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses, even if such party shall have been informed of the possibility of such damages. This limitation of liability shall not apply to liability for death or personal injury resulting from such party’s negligence to the extent applicable law prohibits such limitation. Some jurisdictions do not allow the exclusion or limitation of incidental or consequential damages, so this exclusion and limitation may not apply to You. 8. Litigation Any litigation relating to this License may be brought only in the courts of a jurisdiction where the defendant maintains its principal place of business and such litigation shall be governed by laws of that jurisdiction, without reference to its conflict-of-law provisions. Nothing in this Section shall prevent a party’s ability to bring cross-claims or counter-claims. 9. Miscellaneous This License represents the complete agreement concerning the subject matter hereof. If any provision of this License is held to be unenforceable, such provision shall be reformed only to the extent necessary to make it enforceable. Any law or regulation which provides that the language of a contract shall be construed against the drafter shall not be used to construe this License against a Contributor. 10. Versions of the License 10.1. New Versions Mozilla Foundation is the license steward. Except as provided in Section 10.3, no one other than the license steward has the right to modify or publish new versions of this License. Each version will be given a distinguishing version number. 10.2. Effect of New Versions You may distribute the Covered Software under the terms of the version of the License under which You originally received the Covered Software, or under the terms of any subsequent version published by the license steward. 10.3. Modified Versions If you create software not governed by this License, and you want to create a new license for such software, you may create and use a modified version of this License if you rename the license and remove any references to the name of the license steward (except to note that such modified license differs from this License). 10.4. Distributing Source Code Form that is Incompatible With Secondary Licenses If You choose to distribute Source Code Form that is Incompatible With Secondary Licenses under the terms of this version of the License, the notice described in Exhibit B of this License must be attached. Exhibit A - Source Code Form License Notice This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain one at http://mozilla.org/MPL/2.0/. If it is not possible or desirable to put the notice in a particular file, then You may include the notice in a location (such as a LICENSE file in a relevant directory) where a recipient would be likely to look for such a notice. You may add additional accurate notices of copyright ownership. Exhibit B - “Incompatible With Secondary Licenses” Notice This Source Code Form is “Incompatible With Secondary Licenses”, as defined by the Mozilla Public License, v. 2.0. github.com/hashicorp/memberlist Mozilla Public License, version 2.0 1. Definitions 1.1. “Contributor” means each individual or legal entity that creates, contributes to the creation of, or owns Covered Software. 1.2. “Contributor Version” means the combination of the Contributions of others (if any) used by a Contributor and that particular Contributor’s Contribution. 1.3. “Contribution” means Covered Software of a particular Contributor. 1.4. “Covered Software” means Source Code Form to which the initial Contributor has attached the notice in Exhibit A, the Executable Form of such Source Code Form, and Modifications of such Source Code Form, in each case including portions thereof. 1.5. “Incompatible With Secondary Licenses” means a. that the initial Contributor has attached the notice described in Exhibit B to the Covered Software; or b. that the Covered Software was made available under the terms of version 1.1 or earlier of the License, but not also under the terms of a Secondary License. 1.6. “Executable Form” means any form of the work other than Source Code Form. 1.7. “Larger Work” means a work that combines Covered Software with other material, in a separate file or files, that is not Covered Software. 1.8. “License” means this document. 1.9. “Licensable” means having the right to grant, to the maximum extent possible, whether at the time of the initial grant or subsequently, any and all of the rights conveyed by this License. 1.10. “Modifications” means any of the following: a. any file in Source Code Form that results from an addition to, deletion from, or modification of the contents of Covered Software; or b. any new file in Source Code Form that contains any Covered Software. 1.11. “Patent Claims” of a Contributor means any patent claim(s), including without limitation, method, process, and apparatus claims, in any patent Licensable by such Contributor that would be infringed, but for the grant of the License, by the making, using, selling, offering for sale, having made, import, or transfer of either its Contributions or its Contributor Version. 1.12. “Secondary License” means either the GNU General Public License, Version 2.0, the GNU Lesser General Public License, Version 2.1, the GNU Affero General Public License, Version 3.0, or any later versions of those licenses. 1.13. “Source Code Form” means the form of the work preferred for making modifications. 1.14. “You” (or “Your”) means an individual or a legal entity exercising rights under this License. For legal entities, “You” includes any entity that controls, is controlled by, or is under common control with You. For purposes of this definition, “control” means (a) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (b) ownership of more than fifty percent (50%) of the outstanding shares or beneficial ownership of such entity. 2. License Grants and Conditions 2.1. Grants Each Contributor hereby grants You a world-wide, royalty-free, non-exclusive license: a. under intellectual property rights (other than patent or trademark) Licensable by such Contributor to use, reproduce, make available, modify, display, perform, distribute, and otherwise exploit its Contributions, either on an unmodified basis, with Modifications, or as part of a Larger Work; and b. under Patent Claims of such Contributor to make, use, sell, offer for sale, have made, import, and otherwise transfer either its Contributions or its Contributor Version. 2.2. Effective Date The licenses granted in Section 2.1 with respect to any Contribution become effective for each Contribution on the date the Contributor first distributes such Contribution. 2.3. Limitations on Grant Scope The licenses granted in this Section 2 are the only rights granted under this License. No additional rights or licenses will be implied from the distribution or licensing of Covered Software under this License. Notwithstanding Section 2.1(b) above, no patent license is granted by a Contributor: a. for any code that a Contributor has removed from Covered Software; or b. for infringements caused by: (i) Your and any other third party’s modifications of Covered Software, or (ii) the combination of its Contributions with other software (except as part of its Contributor Version); or c. under Patent Claims infringed by Covered Software in the absence of its Contributions. This License does not grant any rights in the trademarks, service marks, or logos of any Contributor (except as may be necessary to comply with the notice requirements in Section 3.4). 2.4. Subsequent Licenses No Contributor makes additional grants as a result of Your choice to distribute the Covered Software under a subsequent version of this License (see Section 10.2) or under the terms of a Secondary License (if permitted under the terms of Section 3.3). 2.5. Representation Each Contributor represents that the Contributor believes its Contributions are its original creation(s) or it has sufficient rights to grant the rights to its Contributions conveyed by this License. 2.6. Fair Use This License is not intended to limit any rights You have under applicable copyright doctrines of fair use, fair dealing, or other equivalents. 2.7. Conditions Sections 3.1, 3.2, 3.3, and 3.4 are conditions of the licenses granted in Section 2.1. 3. Responsibilities 3.1. Distribution of Source Form All distribution of Covered Software in Source Code Form, including any Modifications that You create or to which You contribute, must be under the terms of this License. You must inform recipients that the Source Code Form of the Covered Software is governed by the terms of this License, and how they can obtain a copy of this License. You may not attempt to alter or restrict the recipients’ rights in the Source Code Form. 3.2. Distribution of Executable Form If You distribute Covered Software in Executable Form then: a. such Covered Software must also be made available in Source Code Form, as described in Section 3.1, and You must inform recipients of the Executable Form how they can obtain a copy of such Source Code Form by reasonable means in a timely manner, at a charge no more than the cost of distribution to the recipient; and b. You may distribute such Executable Form under the terms of this License, or sublicense it under different terms, provided that the license for the Executable Form does not attempt to limit or alter the recipients’ rights in the Source Code Form under this License. 3.3. Distribution of a Larger Work You may create and distribute a Larger Work under terms of Your choice, provided that You also comply with the requirements of this License for the Covered Software. If the Larger Work is a combination of Covered Software with a work governed by one or more Secondary Licenses, and the Covered Software is not Incompatible With Secondary Licenses, this License permits You to additionally distribute such Covered Software under the terms of such Secondary License(s), so that the recipient of the Larger Work may, at their option, further distribute the Covered Software under the terms of either this License or such Secondary License(s). 3.4. Notices You may not remove or alter the substance of any license notices (including copyright notices, patent notices, disclaimers of warranty, or limitations of liability) contained within the Source Code Form of the Covered Software, except that You may alter any license notices to the extent required to remedy known factual inaccuracies. 3.5. Application of Additional Terms You may choose to offer, and to charge a fee for, warranty, support, indemnity or liability obligations to one or more recipients of Covered Software. However, You may do so only on Your own behalf, and not on behalf of any Contributor. You must make it absolutely clear that any such warranty, support, indemnity, or liability obligation is offered by You alone, and You hereby agree to indemnify every Contributor for any liability incurred by such Contributor as a result of warranty, support, indemnity or liability terms You offer. You may include additional disclaimers of warranty and limitations of liability specific to any jurisdiction. 4. Inability to Comply Due to Statute or Regulation If it is impossible for You to comply with any of the terms of this License with respect to some or all of the Covered Software due to statute, judicial order, or regulation then You must: (a) comply with the terms of this License to the maximum extent possible; and (b) describe the limitations and the code they affect. Such description must be placed in a text file included with all distributions of the Covered Software under this License. Except to the extent prohibited by statute or regulation, such description must be sufficiently detailed for a recipient of ordinary skill to be able to understand it. 5. Termination 5.1. The rights granted under this License will terminate automatically if You fail to comply with any of its terms. However, if You become compliant, then the rights granted under this License from a particular Contributor are reinstated (a) provisionally, unless and until such Contributor explicitly and finally terminates Your grants, and (b) on an ongoing basis, if such Contributor fails to notify You of the non-compliance by some reasonable means prior to 60 days after You have come back into compliance. Moreover, Your grants from a particular Contributor are reinstated on an ongoing basis if such Contributor notifies You of the non-compliance by some reasonable means, this is the first time You have received notice of non-compliance with this License from such Contributor, and You become compliant prior to 30 days after Your receipt of the notice. 5.2. If You initiate litigation against any entity by asserting a patent infringement claim (excluding declaratory judgment actions, counter-claims, and cross-claims) alleging that a Contributor Version directly or indirectly infringes any patent, then the rights granted to You by any and all Contributors for the Covered Software under Section 2.1 of this License shall terminate. 5.3. In the event of termination under Sections 5.1 or 5.2 above, all end user license agreements (excluding distributors and resellers) which have been validly granted by You or Your distributors under this License prior to termination shall survive termination. 6. Disclaimer of Warranty Covered Software is provided under this License on an “as is” basis, without warranty of any kind, either expressed, implied, or statutory, including, without limitation, warranties that the Covered Software is free of defects, merchantable, fit for a particular purpose or non-infringing. The entire risk as to the quality and performance of the Covered Software is with You. Should any Covered Software prove defective in any respect, You (not any Contributor) assume the cost of any necessary servicing, repair, or correction. This disclaimer of warranty constitutes an essential part of this License. No use of any Covered Software is authorized under this License except under this disclaimer. 7. Limitation of Liability Under no circumstances and under no legal theory, whether tort (including negligence), contract, or otherwise, shall any Contributor, or anyone who distributes Covered Software as permitted above, be liable to You for any direct, indirect, special, incidental, or consequential damages of any character including, without limitation, damages for lost profits, loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses, even if such party shall have been informed of the possibility of such damages. This limitation of liability shall not apply to liability for death or personal injury resulting from such party’s negligence to the extent applicable law prohibits such limitation. Some jurisdictions do not allow the exclusion or limitation of incidental or consequential damages, so this exclusion and limitation may not apply to You. 8. Litigation Any litigation relating to this License may be brought only in the courts of a jurisdiction where the defendant maintains its principal place of business and such litigation shall be governed by laws of that jurisdiction, without reference to its conflict-of-law provisions. Nothing in this Section shall prevent a party’s ability to bring cross-claims or counter-claims. 9. Miscellaneous This License represents the complete agreement concerning the subject matter hereof. If any provision of this License is held to be unenforceable, such provision shall be reformed only to the extent necessary to make it enforceable. Any law or regulation which provides that the language of a contract shall be construed against the drafter shall not be used to construe this License against a Contributor. 10. Versions of the License 10.1. New Versions Mozilla Foundation is the license steward. Except as provided in Section 10.3, no one other than the license steward has the right to modify or publish new versions of this License. Each version will be given a distinguishing version number. 10.2. Effect of New Versions You may distribute the Covered Software under the terms of the version of the License under which You originally received the Covered Software, or under the terms of any subsequent version published by the license steward. 10.3. Modified Versions If you create software not governed by this License, and you want to create a new license for such software, you may create and use a modified version of this License if you rename the license and remove any references to the name of the license steward (except to note that such modified license differs from this License). 10.4. Distributing Source Code Form that is Incompatible With Secondary Licenses If You choose to distribute Source Code Form that is Incompatible With Secondary Licenses under the terms of this version of the License, the notice described in Exhibit B of this License must be attached. Exhibit A - Source Code Form License Notice This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain one at http://mozilla.org/MPL/2.0/. If it is not possible or desirable to put the notice in a particular file, then You may include the notice in a location (such as a LICENSE file in a relevant directory) where a recipient would be likely to look for such a notice. You may add additional accurate notices of copyright ownership. Exhibit B - “Incompatible With Secondary Licenses” Notice This Source Code Form is “Incompatible With Secondary Licenses”, as defined by the Mozilla Public License, v. 2.0. github.com/hashicorp/serf/coordinate Mozilla Public License, version 2.0 1. Definitions 1.1. “Contributor” means each individual or legal entity that creates, contributes to the creation of, or owns Covered Software. 1.2. “Contributor Version” means the combination of the Contributions of others (if any) used by a Contributor and that particular Contributor’s Contribution. 1.3. “Contribution” means Covered Software of a particular Contributor. 1.4. “Covered Software” means Source Code Form to which the initial Contributor has attached the notice in Exhibit A, the Executable Form of such Source Code Form, and Modifications of such Source Code Form, in each case including portions thereof. 1.5. “Incompatible With Secondary Licenses” means a. that the initial Contributor has attached the notice described in Exhibit B to the Covered Software; or b. that the Covered Software was made available under the terms of version 1.1 or earlier of the License, but not also under the terms of a Secondary License. 1.6. “Executable Form” means any form of the work other than Source Code Form. 1.7. “Larger Work” means a work that combines Covered Software with other material, in a separate file or files, that is not Covered Software. 1.8. “License” means this document. 1.9. “Licensable” means having the right to grant, to the maximum extent possible, whether at the time of the initial grant or subsequently, any and all of the rights conveyed by this License. 1.10. “Modifications” means any of the following: a. any file in Source Code Form that results from an addition to, deletion from, or modification of the contents of Covered Software; or b. any new file in Source Code Form that contains any Covered Software. 1.11. “Patent Claims” of a Contributor means any patent claim(s), including without limitation, method, process, and apparatus claims, in any patent Licensable by such Contributor that would be infringed, but for the grant of the License, by the making, using, selling, offering for sale, having made, import, or transfer of either its Contributions or its Contributor Version. 1.12. “Secondary License” means either the GNU General Public License, Version 2.0, the GNU Lesser General Public License, Version 2.1, the GNU Affero General Public License, Version 3.0, or any later versions of those licenses. 1.13. “Source Code Form” means the form of the work preferred for making modifications. 1.14. “You” (or “Your”) means an individual or a legal entity exercising rights under this License. For legal entities, “You” includes any entity that controls, is controlled by, or is under common control with You. For purposes of this definition, “control” means (a) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (b) ownership of more than fifty percent (50%) of the outstanding shares or beneficial ownership of such entity. 2. License Grants and Conditions 2.1. Grants Each Contributor hereby grants You a world-wide, royalty-free, non-exclusive license: a. under intellectual property rights (other than patent or trademark) Licensable by such Contributor to use, reproduce, make available, modify, display, perform, distribute, and otherwise exploit its Contributions, either on an unmodified basis, with Modifications, or as part of a Larger Work; and b. under Patent Claims of such Contributor to make, use, sell, offer for sale, have made, import, and otherwise transfer either its Contributions or its Contributor Version. 2.2. Effective Date The licenses granted in Section 2.1 with respect to any Contribution become effective for each Contribution on the date the Contributor first distributes such Contribution. 2.3. Limitations on Grant Scope The licenses granted in this Section 2 are the only rights granted under this License. No additional rights or licenses will be implied from the distribution or licensing of Covered Software under this License. Notwithstanding Section 2.1(b) above, no patent license is granted by a Contributor: a. for any code that a Contributor has removed from Covered Software; or b. for infringements caused by: (i) Your and any other third party’s modifications of Covered Software, or (ii) the combination of its Contributions with other software (except as part of its Contributor Version); or c. under Patent Claims infringed by Covered Software in the absence of its Contributions. This License does not grant any rights in the trademarks, service marks, or logos of any Contributor (except as may be necessary to comply with the notice requirements in Section 3.4). 2.4. Subsequent Licenses No Contributor makes additional grants as a result of Your choice to distribute the Covered Software under a subsequent version of this License (see Section 10.2) or under the terms of a Secondary License (if permitted under the terms of Section 3.3). 2.5. Representation Each Contributor represents that the Contributor believes its Contributions are its original creation(s) or it has sufficient rights to grant the rights to its Contributions conveyed by this License. 2.6. Fair Use This License is not intended to limit any rights You have under applicable copyright doctrines of fair use, fair dealing, or other equivalents. 2.7. Conditions Sections 3.1, 3.2, 3.3, and 3.4 are conditions of the licenses granted in Section 2.1. 3. Responsibilities 3.1. Distribution of Source Form All distribution of Covered Software in Source Code Form, including any Modifications that You create or to which You contribute, must be under the terms of this License. You must inform recipients that the Source Code Form of the Covered Software is governed by the terms of this License, and how they can obtain a copy of this License. You may not attempt to alter or restrict the recipients’ rights in the Source Code Form. 3.2. Distribution of Executable Form If You distribute Covered Software in Executable Form then: a. such Covered Software must also be made available in Source Code Form, as described in Section 3.1, and You must inform recipients of the Executable Form how they can obtain a copy of such Source Code Form by reasonable means in a timely manner, at a charge no more than the cost of distribution to the recipient; and b. You may distribute such Executable Form under the terms of this License, or sublicense it under different terms, provided that the license for the Executable Form does not attempt to limit or alter the recipients’ rights in the Source Code Form under this License. 3.3. Distribution of a Larger Work You may create and distribute a Larger Work under terms of Your choice, provided that You also comply with the requirements of this License for the Covered Software. If the Larger Work is a combination of Covered Software with a work governed by one or more Secondary Licenses, and the Covered Software is not Incompatible With Secondary Licenses, this License permits You to additionally distribute such Covered Software under the terms of such Secondary License(s), so that the recipient of the Larger Work may, at their option, further distribute the Covered Software under the terms of either this License or such Secondary License(s). 3.4. Notices You may not remove or alter the substance of any license notices (including copyright notices, patent notices, disclaimers of warranty, or limitations of liability) contained within the Source Code Form of the Covered Software, except that You may alter any license notices to the extent required to remedy known factual inaccuracies. 3.5. Application of Additional Terms You may choose to offer, and to charge a fee for, warranty, support, indemnity or liability obligations to one or more recipients of Covered Software. However, You may do so only on Your own behalf, and not on behalf of any Contributor. You must make it absolutely clear that any such warranty, support, indemnity, or liability obligation is offered by You alone, and You hereby agree to indemnify every Contributor for any liability incurred by such Contributor as a result of warranty, support, indemnity or liability terms You offer. You may include additional disclaimers of warranty and limitations of liability specific to any jurisdiction. 4. Inability to Comply Due to Statute or Regulation If it is impossible for You to comply with any of the terms of this License with respect to some or all of the Covered Software due to statute, judicial order, or regulation then You must: (a) comply with the terms of this License to the maximum extent possible; and (b) describe the limitations and the code they affect. Such description must be placed in a text file included with all distributions of the Covered Software under this License. Except to the extent prohibited by statute or regulation, such description must be sufficiently detailed for a recipient of ordinary skill to be able to understand it. 5. Termination 5.1. The rights granted under this License will terminate automatically if You fail to comply with any of its terms. However, if You become compliant, then the rights granted under this License from a particular Contributor are reinstated (a) provisionally, unless and until such Contributor explicitly and finally terminates Your grants, and (b) on an ongoing basis, if such Contributor fails to notify You of the non-compliance by some reasonable means prior to 60 days after You have come back into compliance. Moreover, Your grants from a particular Contributor are reinstated on an ongoing basis if such Contributor notifies You of the non-compliance by some reasonable means, this is the first time You have received notice of non-compliance with this License from such Contributor, and You become compliant prior to 30 days after Your receipt of the notice. 5.2. If You initiate litigation against any entity by asserting a patent infringement claim (excluding declaratory judgment actions, counter-claims, and cross-claims) alleging that a Contributor Version directly or indirectly infringes any patent, then the rights granted to You by any and all Contributors for the Covered Software under Section 2.1 of this License shall terminate. 5.3. In the event of termination under Sections 5.1 or 5.2 above, all end user license agreements (excluding distributors and resellers) which have been validly granted by You or Your distributors under this License prior to termination shall survive termination. 6. Disclaimer of Warranty Covered Software is provided under this License on an “as is” basis, without warranty of any kind, either expressed, implied, or statutory, including, without limitation, warranties that the Covered Software is free of defects, merchantable, fit for a particular purpose or non-infringing. The entire risk as to the quality and performance of the Covered Software is with You. Should any Covered Software prove defective in any respect, You (not any Contributor) assume the cost of any necessary servicing, repair, or correction. This disclaimer of warranty constitutes an essential part of this License. No use of any Covered Software is authorized under this License except under this disclaimer. 7. Limitation of Liability Under no circumstances and under no legal theory, whether tort (including negligence), contract, or otherwise, shall any Contributor, or anyone who distributes Covered Software as permitted above, be liable to You for any direct, indirect, special, incidental, or consequential damages of any character including, without limitation, damages for lost profits, loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses, even if such party shall have been informed of the possibility of such damages. This limitation of liability shall not apply to liability for death or personal injury resulting from such party’s negligence to the extent applicable law prohibits such limitation. Some jurisdictions do not allow the exclusion or limitation of incidental or consequential damages, so this exclusion and limitation may not apply to You. 8. Litigation Any litigation relating to this License may be brought only in the courts of a jurisdiction where the defendant maintains its principal place of business and such litigation shall be governed by laws of that jurisdiction, without reference to its conflict-of-law provisions. Nothing in this Section shall prevent a party’s ability to bring cross-claims or counter-claims. 9. Miscellaneous This License represents the complete agreement concerning the subject matter hereof. If any provision of this License is held to be unenforceable, such provision shall be reformed only to the extent necessary to make it enforceable. Any law or regulation which provides that the language of a contract shall be construed against the drafter shall not be used to construe this License against a Contributor. 10. Versions of the License 10.1. New Versions Mozilla Foundation is the license steward. Except as provided in Section 10.3, no one other than the license steward has the right to modify or publish new versions of this License. Each version will be given a distinguishing version number. 10.2. Effect of New Versions You may distribute the Covered Software under the terms of the version of the License under which You originally received the Covered Software, or under the terms of any subsequent version published by the license steward. 10.3. Modified Versions If you create software not governed by this License, and you want to create a new license for such software, you may create and use a modified version of this License if you rename the license and remove any references to the name of the license steward (except to note that such modified license differs from this License). 10.4. Distributing Source Code Form that is Incompatible With Secondary Licenses If You choose to distribute Source Code Form that is Incompatible With Secondary Licenses under the terms of this version of the License, the notice described in Exhibit B of this License must be attached. Exhibit A - Source Code Form License Notice This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain one at http://mozilla.org/MPL/2.0/. If it is not possible or desirable to put the notice in a particular file, then You may include the notice in a location (such as a LICENSE file in a relevant directory) where a recipient would be likely to look for such a notice. You may add additional accurate notices of copyright ownership. Exhibit B - “Incompatible With Secondary Licenses” Notice This Source Code Form is “Incompatible With Secondary Licenses”, as defined by the Mozilla Public License, v. 2.0. github.com/hashicorp/serf/serf Mozilla Public License, version 2.0 1. Definitions 1.1. “Contributor” means each individual or legal entity that creates, contributes to the creation of, or owns Covered Software. 1.2. “Contributor Version” means the combination of the Contributions of others (if any) used by a Contributor and that particular Contributor’s Contribution. 1.3. “Contribution” means Covered Software of a particular Contributor. 1.4. “Covered Software” means Source Code Form to which the initial Contributor has attached the notice in Exhibit A, the Executable Form of such Source Code Form, and Modifications of such Source Code Form, in each case including portions thereof. 1.5. “Incompatible With Secondary Licenses” means a. that the initial Contributor has attached the notice described in Exhibit B to the Covered Software; or b. that the Covered Software was made available under the terms of version 1.1 or earlier of the License, but not also under the terms of a Secondary License. 1.6. “Executable Form” means any form of the work other than Source Code Form. 1.7. “Larger Work” means a work that combines Covered Software with other material, in a separate file or files, that is not Covered Software. 1.8. “License” means this document. 1.9. “Licensable” means having the right to grant, to the maximum extent possible, whether at the time of the initial grant or subsequently, any and all of the rights conveyed by this License. 1.10. “Modifications” means any of the following: a. any file in Source Code Form that results from an addition to, deletion from, or modification of the contents of Covered Software; or b. any new file in Source Code Form that contains any Covered Software. 1.11. “Patent Claims” of a Contributor means any patent claim(s), including without limitation, method, process, and apparatus claims, in any patent Licensable by such Contributor that would be infringed, but for the grant of the License, by the making, using, selling, offering for sale, having made, import, or transfer of either its Contributions or its Contributor Version. 1.12. “Secondary License” means either the GNU General Public License, Version 2.0, the GNU Lesser General Public License, Version 2.1, the GNU Affero General Public License, Version 3.0, or any later versions of those licenses. 1.13. “Source Code Form” means the form of the work preferred for making modifications. 1.14. “You” (or “Your”) means an individual or a legal entity exercising rights under this License. For legal entities, “You” includes any entity that controls, is controlled by, or is under common control with You. For purposes of this definition, “control” means (a) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (b) ownership of more than fifty percent (50%) of the outstanding shares or beneficial ownership of such entity. 2. License Grants and Conditions 2.1. Grants Each Contributor hereby grants You a world-wide, royalty-free, non-exclusive license: a. under intellectual property rights (other than patent or trademark) Licensable by such Contributor to use, reproduce, make available, modify, display, perform, distribute, and otherwise exploit its Contributions, either on an unmodified basis, with Modifications, or as part of a Larger Work; and b. under Patent Claims of such Contributor to make, use, sell, offer for sale, have made, import, and otherwise transfer either its Contributions or its Contributor Version. 2.2. Effective Date The licenses granted in Section 2.1 with respect to any Contribution become effective for each Contribution on the date the Contributor first distributes such Contribution. 2.3. Limitations on Grant Scope The licenses granted in this Section 2 are the only rights granted under this License. No additional rights or licenses will be implied from the distribution or licensing of Covered Software under this License. Notwithstanding Section 2.1(b) above, no patent license is granted by a Contributor: a. for any code that a Contributor has removed from Covered Software; or b. for infringements caused by: (i) Your and any other third party’s modifications of Covered Software, or (ii) the combination of its Contributions with other software (except as part of its Contributor Version); or c. under Patent Claims infringed by Covered Software in the absence of its Contributions. This License does not grant any rights in the trademarks, service marks, or logos of any Contributor (except as may be necessary to comply with the notice requirements in Section 3.4). 2.4. Subsequent Licenses No Contributor makes additional grants as a result of Your choice to distribute the Covered Software under a subsequent version of this License (see Section 10.2) or under the terms of a Secondary License (if permitted under the terms of Section 3.3). 2.5. Representation Each Contributor represents that the Contributor believes its Contributions are its original creation(s) or it has sufficient rights to grant the rights to its Contributions conveyed by this License. 2.6. Fair Use This License is not intended to limit any rights You have under applicable copyright doctrines of fair use, fair dealing, or other equivalents. 2.7. Conditions Sections 3.1, 3.2, 3.3, and 3.4 are conditions of the licenses granted in Section 2.1. 3. Responsibilities 3.1. Distribution of Source Form All distribution of Covered Software in Source Code Form, including any Modifications that You create or to which You contribute, must be under the terms of this License. You must inform recipients that the Source Code Form of the Covered Software is governed by the terms of this License, and how they can obtain a copy of this License. You may not attempt to alter or restrict the recipients’ rights in the Source Code Form. 3.2. Distribution of Executable Form If You distribute Covered Software in Executable Form then: a. such Covered Software must also be made available in Source Code Form, as described in Section 3.1, and You must inform recipients of the Executable Form how they can obtain a copy of such Source Code Form by reasonable means in a timely manner, at a charge no more than the cost of distribution to the recipient; and b. You may distribute such Executable Form under the terms of this License, or sublicense it under different terms, provided that the license for the Executable Form does not attempt to limit or alter the recipients’ rights in the Source Code Form under this License. 3.3. Distribution of a Larger Work You may create and distribute a Larger Work under terms of Your choice, provided that You also comply with the requirements of this License for the Covered Software. If the Larger Work is a combination of Covered Software with a work governed by one or more Secondary Licenses, and the Covered Software is not Incompatible With Secondary Licenses, this License permits You to additionally distribute such Covered Software under the terms of such Secondary License(s), so that the recipient of the Larger Work may, at their option, further distribute the Covered Software under the terms of either this License or such Secondary License(s). 3.4. Notices You may not remove or alter the substance of any license notices (including copyright notices, patent notices, disclaimers of warranty, or limitations of liability) contained within the Source Code Form of the Covered Software, except that You may alter any license notices to the extent required to remedy known factual inaccuracies. 3.5. Application of Additional Terms You may choose to offer, and to charge a fee for, warranty, support, indemnity or liability obligations to one or more recipients of Covered Software. However, You may do so only on Your own behalf, and not on behalf of any Contributor. You must make it absolutely clear that any such warranty, support, indemnity, or liability obligation is offered by You alone, and You hereby agree to indemnify every Contributor for any liability incurred by such Contributor as a result of warranty, support, indemnity or liability terms You offer. You may include additional disclaimers of warranty and limitations of liability specific to any jurisdiction. 4. Inability to Comply Due to Statute or Regulation If it is impossible for You to comply with any of the terms of this License with respect to some or all of the Covered Software due to statute, judicial order, or regulation then You must: (a) comply with the terms of this License to the maximum extent possible; and (b) describe the limitations and the code they affect. Such description must be placed in a text file included with all distributions of the Covered Software under this License. Except to the extent prohibited by statute or regulation, such description must be sufficiently detailed for a recipient of ordinary skill to be able to understand it. 5. Termination 5.1. The rights granted under this License will terminate automatically if You fail to comply with any of its terms. However, if You become compliant, then the rights granted under this License from a particular Contributor are reinstated (a) provisionally, unless and until such Contributor explicitly and finally terminates Your grants, and (b) on an ongoing basis, if such Contributor fails to notify You of the non-compliance by some reasonable means prior to 60 days after You have come back into compliance. Moreover, Your grants from a particular Contributor are reinstated on an ongoing basis if such Contributor notifies You of the non-compliance by some reasonable means, this is the first time You have received notice of non-compliance with this License from such Contributor, and You become compliant prior to 30 days after Your receipt of the notice. 5.2. If You initiate litigation against any entity by asserting a patent infringement claim (excluding declaratory judgment actions, counter-claims, and cross-claims) alleging that a Contributor Version directly or indirectly infringes any patent, then the rights granted to You by any and all Contributors for the Covered Software under Section 2.1 of this License shall terminate. 5.3. In the event of termination under Sections 5.1 or 5.2 above, all end user license agreements (excluding distributors and resellers) which have been validly granted by You or Your distributors under this License prior to termination shall survive termination. 6. Disclaimer of Warranty Covered Software is provided under this License on an “as is” basis, without warranty of any kind, either expressed, implied, or statutory, including, without limitation, warranties that the Covered Software is free of defects, merchantable, fit for a particular purpose or non-infringing. The entire risk as to the quality and performance of the Covered Software is with You. Should any Covered Software prove defective in any respect, You (not any Contributor) assume the cost of any necessary servicing, repair, or correction. This disclaimer of warranty constitutes an essential part of this License. No use of any Covered Software is authorized under this License except under this disclaimer. 7. Limitation of Liability Under no circumstances and under no legal theory, whether tort (including negligence), contract, or otherwise, shall any Contributor, or anyone who distributes Covered Software as permitted above, be liable to You for any direct, indirect, special, incidental, or consequential damages of any character including, without limitation, damages for lost profits, loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses, even if such party shall have been informed of the possibility of such damages. This limitation of liability shall not apply to liability for death or personal injury resulting from such party’s negligence to the extent applicable law prohibits such limitation. Some jurisdictions do not allow the exclusion or limitation of incidental or consequential damages, so this exclusion and limitation may not apply to You. 8. Litigation Any litigation relating to this License may be brought only in the courts of a jurisdiction where the defendant maintains its principal place of business and such litigation shall be governed by laws of that jurisdiction, without reference to its conflict-of-law provisions. Nothing in this Section shall prevent a party’s ability to bring cross-claims or counter-claims. 9. Miscellaneous This License represents the complete agreement concerning the subject matter hereof. If any provision of this License is held to be unenforceable, such provision shall be reformed only to the extent necessary to make it enforceable. Any law or regulation which provides that the language of a contract shall be construed against the drafter shall not be used to construe this License against a Contributor. 10. Versions of the License 10.1. New Versions Mozilla Foundation is the license steward. Except as provided in Section 10.3, no one other than the license steward has the right to modify or publish new versions of this License. Each version will be given a distinguishing version number. 10.2. Effect of New Versions You may distribute the Covered Software under the terms of the version of the License under which You originally received the Covered Software, or under the terms of any subsequent version published by the license steward. 10.3. Modified Versions If you create software not governed by this License, and you want to create a new license for such software, you may create and use a modified version of this License if you rename the license and remove any references to the name of the license steward (except to note that such modified license differs from this License). 10.4. Distributing Source Code Form that is Incompatible With Secondary Licenses If You choose to distribute Source Code Form that is Incompatible With Secondary Licenses under the terms of this version of the License, the notice described in Exhibit B of this License must be attached. Exhibit A - Source Code Form License Notice This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain one at http://mozilla.org/MPL/2.0/. If it is not possible or desirable to put the notice in a particular file, then You may include the notice in a location (such as a LICENSE file in a relevant directory) where a recipient would be likely to look for such a notice. You may add additional accurate notices of copyright ownership. Exhibit B - “Incompatible With Secondary Licenses” Notice This Source Code Form is “Incompatible With Secondary Licenses”, as defined by the Mozilla Public License, v. 2.0. github.com/kelseyhightower/envconfig Copyright (c) 2013 Kelsey Hightower Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \u0026quot;Software\u0026quot;), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \u0026quot;AS IS\u0026quot;, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. github.com/mattn/go-isatty Copyright (c) Yasuhiro MATSUMOTO \u0026lt;mattn.jp@gmail.com\u0026gt; MIT License (Expat) Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \u0026quot;Software\u0026quot;), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \u0026quot;AS IS\u0026quot;, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. github.com/miekg/dns Extensions of the original work are copyright (c) 2011 Miek Gieben As this is fork of the official Go code the same license applies: Copyright (c) 2009 The Go Authors. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of Google Inc. nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \u0026quot;AS IS\u0026quot; AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. github.com/mitchellh/cli Mozilla Public License, version 2.0 1. Definitions 1.1. “Contributor” means each individual or legal entity that creates, contributes to the creation of, or owns Covered Software. 1.2. “Contributor Version” means the combination of the Contributions of others (if any) used by a Contributor and that particular Contributor’s Contribution. 1.3. “Contribution” means Covered Software of a particular Contributor. 1.4. “Covered Software” means Source Code Form to which the initial Contributor has attached the notice in Exhibit A, the Executable Form of such Source Code Form, and Modifications of such Source Code Form, in each case including portions thereof. 1.5. “Incompatible With Secondary Licenses” means a. that the initial Contributor has attached the notice described in Exhibit B to the Covered Software; or b. that the Covered Software was made available under the terms of version 1.1 or earlier of the License, but not also under the terms of a Secondary License. 1.6. “Executable Form” means any form of the work other than Source Code Form. 1.7. “Larger Work” means a work that combines Covered Software with other material, in a separate file or files, that is not Covered Software. 1.8. “License” means this document. 1.9. “Licensable” means having the right to grant, to the maximum extent possible, whether at the time of the initial grant or subsequently, any and all of the rights conveyed by this License. 1.10. “Modifications” means any of the following: a. any file in Source Code Form that results from an addition to, deletion from, or modification of the contents of Covered Software; or b. any new file in Source Code Form that contains any Covered Software. 1.11. “Patent Claims” of a Contributor means any patent claim(s), including without limitation, method, process, and apparatus claims, in any patent Licensable by such Contributor that would be infringed, but for the grant of the License, by the making, using, selling, offering for sale, having made, import, or transfer of either its Contributions or its Contributor Version. 1.12. “Secondary License” means either the GNU General Public License, Version 2.0, the GNU Lesser General Public License, Version 2.1, the GNU Affero General Public License, Version 3.0, or any later versions of those licenses. 1.13. “Source Code Form” means the form of the work preferred for making modifications. 1.14. “You” (or “Your”) means an individual or a legal entity exercising rights under this License. For legal entities, “You” includes any entity that controls, is controlled by, or is under common control with You. For purposes of this definition, “control” means (a) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (b) ownership of more than fifty percent (50%) of the outstanding shares or beneficial ownership of such entity. 2. License Grants and Conditions 2.1. Grants Each Contributor hereby grants You a world-wide, royalty-free, non-exclusive license: a. under intellectual property rights (other than patent or trademark) Licensable by such Contributor to use, reproduce, make available, modify, display, perform, distribute, and otherwise exploit its Contributions, either on an unmodified basis, with Modifications, or as part of a Larger Work; and b. under Patent Claims of such Contributor to make, use, sell, offer for sale, have made, import, and otherwise transfer either its Contributions or its Contributor Version. 2.2. Effective Date The licenses granted in Section 2.1 with respect to any Contribution become effective for each Contribution on the date the Contributor first distributes such Contribution. 2.3. Limitations on Grant Scope The licenses granted in this Section 2 are the only rights granted under this License. No additional rights or licenses will be implied from the distribution or licensing of Covered Software under this License. Notwithstanding Section 2.1(b) above, no patent license is granted by a Contributor: a. for any code that a Contributor has removed from Covered Software; or b. for infringements caused by: (i) Your and any other third party’s modifications of Covered Software, or (ii) the combination of its Contributions with other software (except as part of its Contributor Version); or c. under Patent Claims infringed by Covered Software in the absence of its Contributions. This License does not grant any rights in the trademarks, service marks, or logos of any Contributor (except as may be necessary to comply with the notice requirements in Section 3.4). 2.4. Subsequent Licenses No Contributor makes additional grants as a result of Your choice to distribute the Covered Software under a subsequent version of this License (see Section 10.2) or under the terms of a Secondary License (if permitted under the terms of Section 3.3). 2.5. Representation Each Contributor represents that the Contributor believes its Contributions are its original creation(s) or it has sufficient rights to grant the rights to its Contributions conveyed by this License. 2.6. Fair Use This License is not intended to limit any rights You have under applicable copyright doctrines of fair use, fair dealing, or other equivalents. 2.7. Conditions Sections 3.1, 3.2, 3.3, and 3.4 are conditions of the licenses granted in Section 2.1. 3. Responsibilities 3.1. Distribution of Source Form All distribution of Covered Software in Source Code Form, including any Modifications that You create or to which You contribute, must be under the terms of this License. You must inform recipients that the Source Code Form of the Covered Software is governed by the terms of this License, and how they can obtain a copy of this License. You may not attempt to alter or restrict the recipients’ rights in the Source Code Form. 3.2. Distribution of Executable Form If You distribute Covered Software in Executable Form then: a. such Covered Software must also be made available in Source Code Form, as described in Section 3.1, and You must inform recipients of the Executable Form how they can obtain a copy of such Source Code Form by reasonable means in a timely manner, at a charge no more than the cost of distribution to the recipient; and b. You may distribute such Executable Form under the terms of this License, or sublicense it under different terms, provided that the license for the Executable Form does not attempt to limit or alter the recipients’ rights in the Source Code Form under this License. 3.3. Distribution of a Larger Work You may create and distribute a Larger Work under terms of Your choice, provided that You also comply with the requirements of this License for the Covered Software. If the Larger Work is a combination of Covered Software with a work governed by one or more Secondary Licenses, and the Covered Software is not Incompatible With Secondary Licenses, this License permits You to additionally distribute such Covered Software under the terms of such Secondary License(s), so that the recipient of the Larger Work may, at their option, further distribute the Covered Software under the terms of either this License or such Secondary License(s). 3.4. Notices You may not remove or alter the substance of any license notices (including copyright notices, patent notices, disclaimers of warranty, or limitations of liability) contained within the Source Code Form of the Covered Software, except that You may alter any license notices to the extent required to remedy known factual inaccuracies. 3.5. Application of Additional Terms You may choose to offer, and to charge a fee for, warranty, support, indemnity or liability obligations to one or more recipients of Covered Software. However, You may do so only on Your own behalf, and not on behalf of any Contributor. You must make it absolutely clear that any such warranty, support, indemnity, or liability obligation is offered by You alone, and You hereby agree to indemnify every Contributor for any liability incurred by such Contributor as a result of warranty, support, indemnity or liability terms You offer. You may include additional disclaimers of warranty and limitations of liability specific to any jurisdiction. 4. Inability to Comply Due to Statute or Regulation If it is impossible for You to comply with any of the terms of this License with respect to some or all of the Covered Software due to statute, judicial order, or regulation then You must: (a) comply with the terms of this License to the maximum extent possible; and (b) describe the limitations and the code they affect. Such description must be placed in a text file included with all distributions of the Covered Software under this License. Except to the extent prohibited by statute or regulation, such description must be sufficiently detailed for a recipient of ordinary skill to be able to understand it. 5. Termination 5.1. The rights granted under this License will terminate automatically if You fail to comply with any of its terms. However, if You become compliant, then the rights granted under this License from a particular Contributor are reinstated (a) provisionally, unless and until such Contributor explicitly and finally terminates Your grants, and (b) on an ongoing basis, if such Contributor fails to notify You of the non-compliance by some reasonable means prior to 60 days after You have come back into compliance. Moreover, Your grants from a particular Contributor are reinstated on an ongoing basis if such Contributor notifies You of the non-compliance by some reasonable means, this is the first time You have received notice of non-compliance with this License from such Contributor, and You become compliant prior to 30 days after Your receipt of the notice. 5.2. If You initiate litigation against any entity by asserting a patent infringement claim (excluding declaratory judgment actions, counter-claims, and cross-claims) alleging that a Contributor Version directly or indirectly infringes any patent, then the rights granted to You by any and all Contributors for the Covered Software under Section 2.1 of this License shall terminate. 5.3. In the event of termination under Sections 5.1 or 5.2 above, all end user license agreements (excluding distributors and resellers) which have been validly granted by You or Your distributors under this License prior to termination shall survive termination. 6. Disclaimer of Warranty Covered Software is provided under this License on an “as is” basis, without warranty of any kind, either expressed, implied, or statutory, including, without limitation, warranties that the Covered Software is free of defects, merchantable, fit for a particular purpose or non-infringing. The entire risk as to the quality and performance of the Covered Software is with You. Should any Covered Software prove defective in any respect, You (not any Contributor) assume the cost of any necessary servicing, repair, or correction. This disclaimer of warranty constitutes an essential part of this License. No use of any Covered Software is authorized under this License except under this disclaimer. 7. Limitation of Liability Under no circumstances and under no legal theory, whether tort (including negligence), contract, or otherwise, shall any Contributor, or anyone who distributes Covered Software as permitted above, be liable to You for any direct, indirect, special, incidental, or consequential damages of any character including, without limitation, damages for lost profits, loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses, even if such party shall have been informed of the possibility of such damages. This limitation of liability shall not apply to liability for death or personal injury resulting from such party’s negligence to the extent applicable law prohibits such limitation. Some jurisdictions do not allow the exclusion or limitation of incidental or consequential damages, so this exclusion and limitation may not apply to You. 8. Litigation Any litigation relating to this License may be brought only in the courts of a jurisdiction where the defendant maintains its principal place of business and such litigation shall be governed by laws of that jurisdiction, without reference to its conflict-of-law provisions. Nothing in this Section shall prevent a party’s ability to bring cross-claims or counter-claims. 9. Miscellaneous This License represents the complete agreement concerning the subject matter hereof. If any provision of this License is held to be unenforceable, such provision shall be reformed only to the extent necessary to make it enforceable. Any law or regulation which provides that the language of a contract shall be construed against the drafter shall not be used to construe this License against a Contributor. 10. Versions of the License 10.1. New Versions Mozilla Foundation is the license steward. Except as provided in Section 10.3, no one other than the license steward has the right to modify or publish new versions of this License. Each version will be given a distinguishing version number. 10.2. Effect of New Versions You may distribute the Covered Software under the terms of the version of the License under which You originally received the Covered Software, or under the terms of any subsequent version published by the license steward. 10.3. Modified Versions If you create software not governed by this License, and you want to create a new license for such software, you may create and use a modified version of this License if you rename the license and remove any references to the name of the license steward (except to note that such modified license differs from this License). 10.4. Distributing Source Code Form that is Incompatible With Secondary Licenses If You choose to distribute Source Code Form that is Incompatible With Secondary Licenses under the terms of this version of the License, the notice described in Exhibit B of this License must be attached. Exhibit A - Source Code Form License Notice This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain one at http://mozilla.org/MPL/2.0/. If it is not possible or desirable to put the notice in a particular file, then You may include the notice in a location (such as a LICENSE file in a relevant directory) where a recipient would be likely to look for such a notice. You may add additional accurate notices of copyright ownership. Exhibit B - “Incompatible With Secondary Licenses” Notice This Source Code Form is “Incompatible With Secondary Licenses”, as defined by the Mozilla Public License, v. 2.0. github.com/mitchellh/mapstructure The MIT License (MIT) Copyright (c) 2013 Mitchell Hashimoto Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \u0026quot;Software\u0026quot;), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \u0026quot;AS IS\u0026quot;, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. github.com/nats-io/gnatsd/conf The MIT License (MIT) Copyright (c) 2012-2016 Apcera Inc. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \u0026quot;Software\u0026quot;), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \u0026quot;AS IS\u0026quot;, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. github.com/nats-io/gnatsd/logger The MIT License (MIT) Copyright (c) 2012-2016 Apcera Inc. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \u0026quot;Software\u0026quot;), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \u0026quot;AS IS\u0026quot;, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. github.com/nats-io/gnatsd/server The MIT License (MIT) Copyright (c) 2012-2016 Apcera Inc. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \u0026quot;Software\u0026quot;), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \u0026quot;AS IS\u0026quot;, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. github.com/nats-io/gnatsd/server/pse The MIT License (MIT) Copyright (c) 2012-2016 Apcera Inc. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \u0026quot;Software\u0026quot;), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \u0026quot;AS IS\u0026quot;, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. github.com/nats-io/gnatsd/util The MIT License (MIT) Copyright (c) 2012-2016 Apcera Inc. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \u0026quot;Software\u0026quot;), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \u0026quot;AS IS\u0026quot;, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. github.com/nats-io/nats The MIT License (MIT) Copyright (c) 2012-2016 Apcera Inc. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \u0026quot;Software\u0026quot;), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \u0026quot;AS IS\u0026quot;, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. github.com/nats-io/nats/encoders/builtin The MIT License (MIT) Copyright (c) 2012-2016 Apcera Inc. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \u0026quot;Software\u0026quot;), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \u0026quot;AS IS\u0026quot;, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. github.com/nats-io/nuid The MIT License (MIT) Copyright (c) 2012-2016 Apcera Inc. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \u0026quot;Software\u0026quot;), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \u0026quot;AS IS\u0026quot;, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. github.com/pborman/uuid Copyright (c) 2009,2014 Google Inc. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of Google Inc. nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \u0026quot;AS IS\u0026quot; AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. github.com/Sirupsen/logrus The MIT License (MIT) Copyright (c) 2014 Simon Eskildsen Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \u0026quot;Software\u0026quot;), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \u0026quot;AS IS\u0026quot;, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. github.com/thejerf/suture Copyright (c) 2014-2015 Barracuda Networks, Inc. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \u0026quot;Software\u0026quot;), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \u0026quot;AS IS\u0026quot;, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. github.com/urfave/negroni The MIT License (MIT) Copyright (c) 2014 Jeremy Saenz Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \u0026quot;Software\u0026quot;), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \u0026quot;AS IS\u0026quot;, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. golang.org/x/crypto/bcrypt Copyright (c) 2009 The Go Authors. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of Google Inc. nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \u0026quot;AS IS\u0026quot; AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. golang.org/x/crypto/blowfish Copyright (c) 2009 The Go Authors. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of Google Inc. nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \u0026quot;AS IS\u0026quot; AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. golang.org/x/net/context Copyright (c) 2009 The Go Authors. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of Google Inc. nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \u0026quot;AS IS\u0026quot; AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. golang.org/x/sys/unix Copyright (c) 2009 The Go Authors. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of Google Inc. nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \u0026quot;AS IS\u0026quot; AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. golang.org/x/sys/windows Copyright (c) 2009 The Go Authors. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of Google Inc. nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \u0026quot;AS IS\u0026quot; AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. golang.org/x/sys/windows/registry Copyright (c) 2009 The Go Authors. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of Google Inc. nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \u0026quot;AS IS\u0026quot; AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. golang.org/x/sys/windows/svc/eventlog Copyright (c) 2009 The Go Authors. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of Google Inc. nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \u0026quot;AS IS\u0026quot; AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. golang.org/x/time/rate Copyright (c) 2009 The Go Authors. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of Google Inc. nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \u0026quot;AS IS\u0026quot; AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. gopkg.in/yaml.v2 Copyright 2011-2016 Canonical Ltd. Licensed under the Apache License, Version 2.0 (the \u0026quot;License\u0026quot;); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \u0026quot;AS IS\u0026quot; BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. ","excerpt":"The following document lists the open source software attributions in the StorageOS Control Plane, …","ref":"/docs/reference/open_source_attribution/","title":"StorageOS Open Source Software Attribution Notice"},{"body":"StorageOS V2 supports Openshift v4.\nOpenShift and StorageOS communicate with each other to perform actions such as creation, deletion and mounting of volumes through CSI. The CSI container running in the StorageOS Daemonset creates a Linux socket that allows the communication between OpenShift and StorageOS\nInstallation StorageOS v2 supports OpenShift 4.0, 4.1, 4.2, 4.3, 4.4 and 4.5.\nTo install StorageOS on OpenShift, please follow our installation instructions page.\n N.B. Openshift 4 uses the CRI-O container runtime that sets a default PID limit of 1024. StorageOS recommends that the limit be raised to 32768. Please see our prerequisites for more details.\n OpenShift Upgrades OpenShift provides an upgrade operator that automates the process of orchestrator version changes.\nThis procedure can cause StorageOS to malfunction due to sequential node restarts not taking the presence of stateful application data into consideration. To avoid this issue, make sure all stateful workloads using StorageOS Volumes are stopped - usually by scaling StatefulSets to 0. Please contact StorageOS support for further advice if required.\n OpenShift requires the internal registry to be available during the upgrade, however StorageOS volumes may not be available. Therefore using StorageOS for the internal registry is not recommended.\n CSI (Container Storage Interface) Note CSI is the standard that enables storage drivers to release on their own schedule. This allows storage vendors to upgrade, update, and enhance their drivers without the need to update Kubernetes source code, or follow Kubernetes release cycles.\nStorageOS v2 implements communication with the OpenShift controlplane with CSI only. The native driver has been deprecated\nStorageOS PersistentVolumeClaims The user can provide standard PVC definitions and StorageOS will dynamically provision them. StorageOS presents volumes to containers with standard POSIX mount targets. This enables the Kubelet to mount StorageOS volumes using standard linux device files. Checkout device presentation for more details.\n","excerpt":"StorageOS V2 supports Openshift v4.\nOpenShift and StorageOS communicate with each other to perform …","ref":"/docs/platforms/openshift/","title":"OpenShift"},{"body":"StorageOS policies are a way to control user and group access to StorageOS Namespaces. To grant a user or group access to a namespace, a policy needs to be created mapping the user or group to the namespace.\n Note: Users always have access to the default namespace\n For more information on how to use policies, see the Policies operations page.\n","excerpt":"StorageOS policies are a way to control user and group access to StorageOS Namespaces. To grant a …","ref":"/docs/concepts/policies/","title":"Policies"},{"body":"Policies control access to StorageOS namespaces. Policies can be configured at the group or user level so access can be controlled granularly.\nUsers can belong to one or more groups to control their namespace permissions. Additionally user specific policies can be created to grant a user access to a namespace. Users can belong to any number of groups and have any number of user level policies configured.\n Note: Users are created with access to the default namespace. Policies cannot be applied to the default namespace.\n Managing Policies To start creating policies, at least one custom namespace and user are required. For more information on how to create namespaces see our Namespace guide, for users see our Users CLI reference.\nIn order to create a policy navigate to \u0026ldquo;Policies\u0026rdquo; in the GUI and select \u0026ldquo;Create Policy\u0026rdquo;. A policy controls access to a variety of StorageOS resources and is applied to a user, by placing the user in the policies group.\nIn order to delete a policy, all users must be removed from the policy group before deletion of the policy can be completed.\n","excerpt":"Policies control access to StorageOS namespaces. Policies can be configured at the group or user …","ref":"/docs/operations/policies/","title":"Policies"},{"body":"PostgreSQL with StorageOS  PostgreSQL or \u0026ldquo;Postgres\u0026rdquo; is an open source object-relational database management system (ORDBMS).\nPostgres is deployed across a wide variety of platforms with a mix of workloads ranging from small, single-node use cases to large Internet-facing clusters with many concurrent users.\nBefore you start, ensure you have StorageOS installed and ready on a Kubernetes cluster. If you need to setup StorageOS on Kubernetes then please see our guide to [installing StorageOS on Kubernetes]({% link \u0026ldquo;docs/install/kubernetes.md\u0026rdquo; \u0026gt;}}).\nDeploying PostgreSQL on Kubernetes   You can find the latest files in the StorageOS use cases repository\ngit clone https://github.com/storageos/use-cases.git storageos-usecases PersistentVolumeClaim and Pod definition excerpts\nkind:PersistentVolumeClaimmetadata:name:pg-dataannotations:volume.beta.kubernetes.io/storage-class:fast...kind:Podmetadata:name:postgressspec:securityContext:fsGroup:26containers:- name:pgimage:crunchydata/crunchy-postgres:centos7-10.4-1.8.3volumeMounts:- mountPath:/pgdataname:data...volumes:- name:datapersistentVolumeClaim:claimName:pg-dataThis excerpt is from the PersistentVolumeClaim and Pod definition. The pod definition references the pg-data VolumeClaim so storage is dynamically provision storage, using the StorageOS storage class. Dynamic provisioning occurs as a volumeMount has been declared with the same name as a Volume Claim.\n  Move into the PostgreSQL examples folder and create the objects\ncd storageos-usecases kubectl create -f ./postgres   Confirm PostgreSQL is up and running.\n$ kubectl get pod postgres-0 -w NAME READY STATUS RESTARTS AGE postgres-0 1/1 Running 0 1m   Connect to the PostgreSQL client pod and connect to the PostgreSQL server through the service.\n$ kubectl exec -it postgres-0 -- psql -h postgres-0.postgres -U primaryuser postgres -c \u0026#34;\\l\u0026#34; Password for user primaryuser: password List of Databases Name | Owner | Encoding | Collate | Ctype | Access privileges +=========================================================================+ postgres | postgres | SQL_ASCII | C | C | template0 | postgres | SQL_ASCII | C | C | =c/postgres + | | | | | postgres=CTc/postgres template1 | postgres | SQL_ASCII | C | C | =c/postgres + | | | | | postgres=CTc/postgres userdb | postgres | SQL_ASCII | C | C | =Tc/postgres + | | | | | postgres=CTc/postgres+ | | | | | testuser=CTc/postgres (4 rows) The password for the primary user is password. You can see this is set in the ConfigMap file.\n  Configuration If you need custom startup options, you can edit the ConfigMap file 15-postgresd-configmap.yaml with your desired PostgreSQL configuration settings.\n","excerpt":"PostgreSQL with StorageOS  PostgreSQL or \u0026ldquo;Postgres\u0026rdquo; is an open source object-relational …","ref":"/docs/usecases/postgres/","title":""},{"body":"Prometheus with StorageOS  Prometheus is a popular application used for event monitoring and alerting in Kubernetes.\nBefore you start, ensure you have StorageOS installed and ready on a Kubernetes cluster. See our guide on how to install StorageOS on Kubernetes for more information.\nDeploying Prometheus on Kubernetes This is the Prometheus use case for StorageOS. Following are the steps for creating a Prometheus StatefulSet and using StorageOS to provide persistent storage.\n  You can find the latest files in the StorageOS use cases repository\ngit clone https://github.com/storageos/use-cases.git storageos-usecases Prometheus Custom Resource definition\napiVersion:monitoring.coreos.com/v1kind:Prometheusmetadata:name:prometheus-storageoslabels:app:prometheus-operatorspec:...storage:volumeClaimTemplate:metadata:name:datalabels:env:prodspec:accessModes:[\u0026#34;ReadWriteOnce\u0026#34;]storageClassName:storageos-replicatedresources:requests:storage:1GiThis excerpt is from the Prometheus Custom Resource definition. This file contains the VolumeClaimTemplate that will dynamically provision storage, using the StorageOS storage class. Dynamic provisioning occurs due to the VolumeClaimTemplate in the Prometheus StatefulSet. The Prometheus StatefulSet is created by the Prometheus Operator, triggered by the creation of the Prometheus resource.\n  Move into the Prometheus examples folder and install the Prometheus Operator and create a Prometheus resource.\n$ cd storageos-usecases/prometheus $ ./install-prometheus.sh   Confirm Prometheus is up and running.\n$ kubectl get pods -w -l app=prometheus NAME READY STATUS RESTARTS AGE prometheus-prometheus-storageos-0 3/3 READY 0 1m   You can see the created PVC using.\n$ kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE data-prometheus-prometheus-storageos-0 Bound pvc-b6c17c0a-e76b-4a0b-8fc6-46c0e1629210 1Gi RWO storageos-replicated 65m   In the Prometheus deployment script, a service monitor is created. A Service Monitor is a special object that the Prometheus operator uses to create configuration for Endpoints for Prometheus to scrape. Although the name implies that a Service Monitor defines Kubernetes services that will be scraped, Prometheus actually targets the Endpoints that the services point to. The new Prometheus instance will use the storageos-etcd service monitor to start scraping metrics from the storageos-etcd pods. Assuming the storageos cluster was setup using ETCD as pods. For more information about service monitors, have a look at the upstream documentation.\n$ kubectl get servicemonitor NAME AGE storageos-etcd 5d1h   The Prometheus web ui can be accessed by port-forwarding the Prometheus pods port to localhost.\n$ kubectl port-forward prometheus-prometheus-storageos-0 9090 Then launch a web browser and go to localhost:9090 to access the Prometheus web ui. You can confirm that the storageos-etcd target is configured there.\n  Configuration In the storageos-usecases/prometheus/manifests/prometheus directory there are other example Service Monitors. For more information about Prometheus, check out the prometheus documentation.\n","excerpt":"Prometheus with StorageOS  Prometheus is a popular application used for event monitoring and …","ref":"/docs/usecases/prometheus/","title":""},{"body":"To install StorageOS on Rancher, please follow our installation instructions page.\nStorageOS transparently supports Rancher deployments using CentOS, RHEL or Debian and can support other Linux distributions detailed in the systems supported page if the appropriate kernel modules are present.\n","excerpt":"To install StorageOS on Rancher, please follow our installation instructions page.\nStorageOS …","ref":"/docs/platforms/rancher/","title":"Rancher"},{"body":" Please note: StorageOS Project edition is required to create RWX Volumes.\n StorageOS supports ReadWriteMany (RWX) access mode Persistent Volumes. A RWX PVC can be used simultaneously by many Pods in the same Kubernetes namespace for read and write operations.\nStorageOS RWX Volumes are based on a shared filesystem - in the case of our implementation, this is NFS.\nArchitecture For each RWX Volume, the following components are involved:\nStorageOS ReadWriteOnly (RWO) Volume\nStorageOS provisions a standard Volume that provides a block device for the file system of the NFS server. This means that every RWX Volume has its own RWO Volume. This allows RWX Volumes to leverage the synchronous replication and automatic failover functionality of StorageOS, providing the NFS server with high availability.\nNFS-Ganesha server\nFor each RWX Volume, an NFS-Ganesha server is spawned by StorageOS. The NFS server runs in user space on the Node containing the primary Volume. Each NFS server uses its own RWO Volume to store data so the data of each Volume is isolated.\nStorageOS binds an ephemeral port to the host network interface for each NFS-Ganesha server. The NFS export is presented using NFS v4.2. Check the prerequisites page to see the range of ports needed for StorageOS RWX Volumes.\nStorageOS API Manager\nStorageOS fully integrates with Kubernetes. The StorageOS API Manager Pod monitors StorageOS RWX Volumes to create and maintain a Kubernetes Service that points towards each RWX Volume\u0026rsquo;s NFS export endpoint. The API Manager is responsible for updating the Service endpoint when a RWX Volume failover occurs.\nProvisioning and using RWX PVCs The sequence in which a RWX PVC is provisioned and used is as follows:\n A PersistentVolumeClaim (PVC) is created with RWX access mode using any StorageOS StorageClass. StorageOS dynamically provisions the PV. A new StorageOS RWO Volume is provisioned internally (not visible in Kubernetes). When the RWX PVC is consumed by a pod, an NFS-Ganesha server is instantiated on the same Node as the primary Volume. The NFS-Ganesha server thus uses the RWO StorageOS Volume as its back end disk. The StorageOS API Manager publishes the host IP and port for the NFS service endpoint, by creating a Kubernetes Service that points to the NFS-Ganesha server export endpoint. StorageOS issues a NFS mount on the Node where the Pod using the PVC is scheduled.  High availability RWX Volumes failover in the same way as standard RWO StorageOS Volumes. The replica Volume is promoted upon detection of Node failure and the NFS-Ganesha server is started on the Node containing the promoted replica. The StorageOS API Manager updates the endpoint of the Volume\u0026rsquo;s NFS service, causing traffic to be routed to the URL of the new NFS-Ganesha server. The NFS client in the application Node (where the user\u0026rsquo;s Pod is running) automatically reconnects.\nNotes  All feature labels that work on RWO Volumes will also work on RWX Volumes. A StorageOS RWX Volume is matched one-to-one with a PVC. Therefore the StorageOS RWX Volume can only be accessed by Pods in the same Kubernetes namespace. StorageOS RWX Volumes support volume resize. Refer to the resize documentation for more details.  ","excerpt":"Please note: StorageOS Project edition is required to create RWX Volumes.\n StorageOS supports …","ref":"/docs/concepts/rwx/","title":"ReadWriteMany"},{"body":" Please note: StorageOS Project edition is required to create RWX Volumes.\n StorageOS supports ReadWriteMany (RWX) access mode Persistent Volumes. A RWX PVC can be used simultaneously by many Pods in the same Kubernetes namespace for read and write operations.\nStorageOS RWX Volumes are based on a shared filesystem.\nTo create a ReadWriteMany (RWX) volume with StorageOS, create a Persistent Volume Claim (PVC) with an access mode of ReadWriteMany (see the First PVC documentation for examples of creating standard PVCs with StorageOS).\nThe following YAML manifest files provide an example:\nA 5Gi PVC with the StorageOS storageClassName of fast, with an accessMode of ReadWriteMany:\napiVersion:v1kind:PersistentVolumeClaimmetadata:name:pvc-rwxspec:storageClassName:fastaccessModes:- ReadWriteManyresources:requests:storage:5GiA Deployment of 3 Pods, each of which consume this PVC:\napiVersion:apps/v1kind:Deploymentmetadata:name:sharedlabels:app:sharedspec:replicas:3selector:matchLabels:app:sharedtemplate:metadata:labels:app:sharedspec:containers:- name:debianimage:debian:9-slimcommand:[\u0026#34;/bin/sleep\u0026#34;]args:[\u0026#34;3600\u0026#34;]volumeMounts:- mountPath:/mnt/name:v1volumes:- name:v1persistentVolumeClaim:claimName:pvc-rwxAfter creating the above resources, the PVC (here named pvc-rwx) should be bound and show an access mode of RWX:\n$ kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE pvc-rwx Bound pvc-59f7a152-8342-415c-a6ca-1cbb463410ab 5Gi RWX fast 60s Ensure that the Deployment Pods are running:\n$ kubectl get pods NAME READY STATUS RESTARTS AGE shared-1771418926-7o5ns 1/1 Running 0 1m shared-1771418926-r18az 1/1 Running 0 1m shared-1771418926-ds8f7 1/1 Running 0 1m The NFS-Ganesha service that exposes the StorageOS volume as RWX can now be viewed, showing the cluster IP of the service, and the default NFS port (2049).\n$ kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 \u0026lt;none\u0026gt; 443/TCP 120m pvc-59f7a152-8342-415c-a6ca-1cbb463410ab ClusterIP 10.107.10.0 \u0026lt;none\u0026gt; 2049/TCP 1m Features of the NFS volume can also be examined in the StorageOS UI. Under Volumes, the Attachment column shows an nfs tag. The Volume Details section provides information about the NFS Volume, such as the service endpoint and the node on which the underlying Volume is attached.\n","excerpt":"Please note: StorageOS Project edition is required to create RWX Volumes.\n StorageOS supports …","ref":"/docs/operations/rwx/","title":"ReadWriteMany"},{"body":"Redis with StorageOS  Redis is a popular networked, in-memory, key-value data store with optional durability to disk.\nBefore you start, ensure you have StorageOS installed and ready on a Kubernetes cluster. See our guide on how to install StorageOS on Kubernetes for more information.\nDeploying Redis on Kubernetes   You can find the latest files in the StorageOS use cases repository\ngit clone https://github.com/storageos/use-cases.git storageos-usecases StatefulSet defintion\nkind:StatefulSetmetadata:name:redisspec:selector:matchLabels:app:redisenv:prodserviceName:redisreplicas:1...spec:serviceAccountName:redis...volumeMounts:- name:datamountPath:/bitnami/redis/data...volumeClaimTemplates:- metadata:name:datalabels:env:prodspec:accessModes:[\u0026#34;ReadWriteOnce\u0026#34;]storageClassName:\u0026#34;fast\u0026#34;# StorageOS storageClass resources:requests:storage:5GiThis excerpt is from the StatefulSet definition. This file contains the VolumeClaim template that will dynamically provision storage, using the StorageOS storage class. Dynamic provisioning occurs as a volumeMount has been declared with the same name as a Volume Claim.\n  Move into the Redis examples folder and create the objects\ncd storageos-usecases kubectl create -f ./redis   Confirm Redis is up and running.\n$ kubectl get pods -w -l app=redis NAME READY STATUS RESTARTS AGE redis-0 1/1 Running 0 1m   Connect to the Redis client pod and connect to the Redis server through the service\n$ kubectl exec -it redis-0 -- redis-cli -a password Warning: Using a password with \u0026#39;-a\u0026#39; option on the command line interface may not be safe. 127.0.0.1:6379\u0026gt; CONFIG GET maxmemory 1) \u0026#34;maxmemory\u0026#34; 2) \u0026#34;0\u0026#34;   Configuration If you need custom startup options, you can edit the ConfigMap file 15-redis-configmap.yaml with your desired Redis configuration settings.\n","excerpt":"Redis with StorageOS  Redis is a popular networked, in-memory, key-value data store with optional …","ref":"/docs/usecases/redis/","title":""},{"body":"We recommend always using \u0026ldquo;tagged\u0026rdquo; versions of StorageOS rather than \u0026ldquo;latest\u0026rdquo;, and to perform upgrades only after reading the release notes.\nThe latest tagged release is v2.3.2. For installation instructions please see our Install page.\nThe latest CLI release is v2.3.0, available from Github.\nUpgrading To upgrade from version 1.x to 2.x, please contact support for assistance.\nv2.3.2 - Released 2020-11-25 Fixed  (controlplane/rejoin) Failure to delete data causes re-advertise loop. (controlplane/rejoin) Handle timeout waiting for progress report. (dataplane/log) Change buffering of symmetra output to prevent stalls.  v2.3.1 - Released 2020-11-16  Allows access to ReadWriteMany shared volumes when running containers as a non-root user.  Fixed  nfs: root squash to uid=0 is now configured on all shared volumes.  v2.3.0 - Released 2020-10-31 This release adds production-grade shared file support to v2, previously a technology preview in v1.\nBreaking  The v2.3.0 operator is no longer able to run StorageOS v1.  New  Adds support for ReadWriteMany shared volumes. See ReadWriteMany. Adds api-manager deployment to support shared volumes. See the api manager Github repository for more information. Kubernetes 1.19 support.  Improved  dataplane: Reduce replication thread usage by having the replication processes share the main thread pool. This helps ensure that there isn\u0026rsquo;t a spike in thread usage when a node recovers and begins re-syncing its volumes. This is particularly relevant on CRIO-based orchestrators such as Openshift where the default maximum allowed PID limit (which also governs the thread limit) is low. dataplane: Detect and log the effective maximum PID limit on startup. dataplane: Internal device presentation mappings are now ephemeral and are not persisted across reboots. dataplane: Disabled default verbose logging for fdatasync/flushWAL timers. dataplane: Log both volume inode and UUID in replication error messages for easier correlation. dataplane: On startup, ensure any remnant devices that may have been left after an unclean shutdown have been properly cleared. dataplane: Signal when all startup tasks complete. This ensures no IO can be initiated before this time. ha: Implement a backoff when attempting to repoint an attached volume after the master has failed. ha: Replicas can now rejoin after an asymmetric partition. This can occur when the master has not lost communication to the replica, but the replica can\u0026rsquo;t communicate with the master. Previously the replica would not be able to rejoin until the master determined it had failed. ha: A master that was partitioned can now re-join to the new master as a replica. api: node label changes update target node prior to committing new state. api: Validation errors now include more information on the failure and how to resolve. csi: Volume resize error messages (e.g. capacity exceeded) now passed through in CSI response. csi: Volume attachment is now verified prior to mount for more instructive error message. csi: Returns RESOURCE EXHAUSTED error when attempting to exceed maximum of 250 StorageOS volume attachments per node. diagnostics: Multiple improvements to bundle collection and collected data. ui: Allow collection of partial diagnostics bundles. ui: Tolerate clock skew when authenticating via the UI. licensing: Read-through cache added. Licence updates will take up to 60s to propagate to all nodes. cli: Set replicas output formatting. init: Checks the effective maximum PID limit and warns if less than the StorageOS recommended PID limit (32,768). CRIO-based distributions such as Openshift have a much lower default value (1024). Consult prerequisites for more information.  Fixed  dataplane: Fixes transport endpoint is not connected on startup after an unclean shutdown. csi: Volume unmount requests now succeed when the mountpoint has already been removed by the orchestrator. csi: Volume detach requests now succeed when the volume has already been deleted. Previously the volume would be stuck in Terminating status.  v2.2.0 - Released 2020-08-18 This release focuses on performance. We analysed existing performance characteristics across a variety of real-world use cases and ended up with improvements across the board. Of particular note:\n Sequential reads have improved by up to 130% Sequential writes have improved by up to 737% Random reads have improved by up to 45% Random writes have improved by up to 135% I/O for large block sizes (128K) has improved by up to 353%  We are extremely proud of our performance and we love to talk about it. Have a look at the Benchmarking section of the self-evaluation guide and consider sharing your results. Our PRE engineers are available to discuss in our slack channel.\nNew   Data engine revamp focused on provable consistency and performance. Key characteristics:\n Metadata is stored in an optimised index, lowering I/O latency and improving performance for all workloads. Large block reads/writes are now be handled in a single operation. Applications like Kafka will go much faster.    On-disk compression is now disabled by default as in most scenarios this offers better performance. To enable on-disk compression for a specific workload, see compression.\n  Improved  dataplane: The number of I/O threads are now determined by the number of processing cores available. This improves scalability and performance on larger servers. ha: Improve partition tolerance behaviour when a volume master that has lost its connection to etcd rejoins. ha: Allow replicas in unhealthy states to be remediated and re-used while maintaining partition tolerance. ha: When a master fails and the new master is not yet available, introduce a back-off to the redirection logic to avoid spamming the logs with connection failure errors. ha: Ignore health advertisements for local node. Local nodes are handled directly. node delete: Only refuse to delete a node if the node health can be authoritatively verified to be in use. api: Increase HTTP server write timeout. cli/ui: Allow partial diagnostic bundle downloads. ui: Namespace dropdown can now be scrolled. ui: Add \u0026ldquo;Job title\u0026rdquo; to UI licence form. logging: Log version at startup at INFO level. logging: Lower verbosity of SCSI warnings that do not apply to StorageOS. diagnostics: Include logs that have been rotated. diagnostics: Bundle collection across providers is now done in parallel. build: Update base image to RHEL 8.2. operator: Removed DB migration utility required for v1.3 -\u0026gt; v1.4 upgrades. operator: Automatically refreshes StorageOS API token without failing requests when the token expires. operator: Updated CSI attacher and provisioner to latest upstream. operator: Remove cluster.local suffix on Pod scheduler service address. This allows the scheduler to work in clusters with custom DNS configuration. operator: Defaults are now set for most CSI configuration options in the StorageOSCluster custom resource.  Fixed  csi: When unmount request is received for a volume that has already been unmounted, return success. csi: Verify volume is attached on the node before mounting it. xfs: Support older RHEL kernels which have an XFS library that does not allow reflinks/dedupe. dataplane: Reserve 1GiB of capacity on the target disk to allow manual cleanup operations, rather than filling target disk to capacity. operator: In some cases /var/lib/storageos could fail to unmount cleanly after a restart. This resulted in multiple entries in /proc/mounts.  v2.1.0 - Released 2020-06-26 New  csi: Volume expansion now supported in offline mode. To expand a volume, stop any workloads accessing the volume, then edit the PVC to increase the capacity. For more information, see our Volume Resize operations page and the CSI Volume Expansion page. api: Volume configuration including replica count can now be updated while the volume is in use. Other updateable fields include labels and description. failover: Before determining that a node is offline and performing recovery operations, the I/O path is also verified. This provides more robust failure detection and ensures that nodes that are still responding to I/O do not get replaced. This I/O path verification is in addition to the gossip-based failure detection. operator: Default tolerations are now set for the StorageOS node container. This helps ensure that the StorageOS node container does not get evicted when the node is running low on resources.  Improved  api: Added checks to prevent deletion of a node with active volumes, or if it is the master of at least one volume. This helps prevent orphaned volumes. cli: Add an --offline-delete flag to allow removal of volumes whose master and replica nodes are offline. This allows cleanup of orphaned volumes. ui: Add an offline volume delete option. ui: Volumes can now be detached from the UI. cli: Labels are no longer truncated. api: When a new node is added to the cluster, its capacity is available to use immediately.  Fixed  ui: Favicon was missing. ui: Duplicate volumes could be shown on the node details page. operator: During uninstall a ClusterRoleBinding was not removed.  v2.0.0 - Released 2020-05-05 New  operator: StorageOS containers now run in the kube-system namespace by default to allow the system-node-critical priority class to be set. This instructs Kubernetes to start StorageOS before application Pods, and to evict StorageOS only after application Pods have finished. This setting was previously recommended in documentation; it is now the default. operator: StorageOS CSI helper containers now run as privileged. This ensures that the CSI endpoint can be seen on systems with SELinux enabled. ui: replication progress for new or re-joining replicas is now displayed. ui: show warning for unlicensed clusters. cli: new commands:  licence management get policy create namespace create policy describe user describe namespace describe policy delete user delete namespace delete policy   licence: removed the default licence expiry date added for v2.0.0-rc.1.  Improved  dataplane: improved retry behaviour for network I/O. cli: \u0026ldquo;get volumes\u0026rdquo; for all namespaces should be done in parallel. cli: help text document config file ui: link node name and get to node details on the volume details page. ui: node details add available capacity spinner. ui: node list remove capacity values / address port. ui: node list show master/replica counts. ui: node list remove edit action. ui: format entity labels. ui: node details link volumes. ui: align buttons for licences. ui: k8s warning in \u0026ldquo;create volume\u0026rdquo; modal. ui: node list remove \u0026ldquo;API\u0026rdquo; from \u0026ldquo;API Address\u0026rdquo; ui: add some details about the Licence on the licence page. api: include valid for duration in login response. licence: restrict nodes which are unregistered after 24 hours. scheduler: return error for namespace/volume not found dataplane: start gRPC threads separately from rest of the supervisor.  Fixed  ui: centre licence types. ui: capacity in ui is per namespace. cli: fail gracefully if missing some output details (i.e. no node exists for id).  v2.0.0-rc.1 - Released 2020-03-31 Initial release of version 2.x. See StorageOS v2.0 Release Blog for details.\n","excerpt":"We recommend always using \u0026ldquo;tagged\u0026rdquo; versions of StorageOS rather than …","ref":"/docs/reference/release_notes/","title":"Release notes"},{"body":"StorageOS replicates volumes across nodes for data protection and high availability. Synchronous replication ensures strong consistency for applications such as databases and Elasticsearch, incurring one network round trip on writes.\nThe basic model for StorageOS replication is of a master volume with distributed replicas. Each volume can be replicated between 0 and 5 times, which are provisioned to 0 to 5 nodes, up to the number of remaining nodes in the cluster.\nIn this diagram, the master volume D was created on node 1, and two replicas, D2 and D3 on nodes 3 and 5.\nWrites that come into D (step 1) are written in parallel to D2 and D3 (step 2). When both replicas and the master acknowledge that the data has been written (step 3), the write operation return successfully to the application (step 4).\nFor most applications, one replica is sufficient (storageos.com/replicas=1).\nAll replication traffic on the wire is compressed using the lz4 algorithm, then streamed over tcp/ip to target port tcp/5703.\nIf the master volume is lost, a replica is promoted to master (D2 or D3 above) and a new replica is created and synced on an available node (Node 2 or 4). This is transparent to the application and does not cause downtime.\nIf a replica volume is lost and there are enough remaining nodes, a new replica is created and synced on an available node. While a new replica is created and being synced, the volume\u0026rsquo;s health will be marked as degraded.\nIf the lost replica comes back online before the new replica has finished synchronizing, then StorageOS will calculate which of the two synchronizing replicas has the smallest difference compared to the master volume and keep that replica. The same holds true if a master volume is lost and a replica is promoted to be the new master. If possible, a new replica will be created and begin to sync. Should the former master come back online it will be demoted to a replica and the replica will the smallest difference to the current master will be kept.\nWhile the replica count is controllable on a per-volume basis, some environments may prefer to set default labels on the StorageClass.\nNumber of nodes StorageOS replicas are distributed across available nodes. When a node fails, a new replica is provisioned and synced as described above. Failure to provision a new replica will eventually cause the volume to become read-only. StorageOS will re-attempt creation of the replica for 90 seconds. After that period, if the old replica is not available and a new replica cannot be provisioned, StorageOS cannot guarantee that the data is safe and stored on multiple nodes as requested by the user. It will therefore force the volume to be set to read-only.\nTo ensure that a new replica can always be created, an additional node should be available. To guarantee high availability, clusters using volumes with 1 replica must have at least 3 storage nodes. When using volumes with 2 replicas, at least 4 storage nodes, 3 replicas, 5 nodes, etc.\nMinimum number of storage nodes = 1 (primary) + N (replicas) + 1\nFor an application to be able to use the volume normally, the volume needs to recover from a degraded state. This is achieved by having enough nodes on which to place the data, and triggering a remount of the volumes - i.e. deleting the Pod mounting the volume once it is in a healthy state.\nDelta Sync StorageOS implements a delta sync between a volume master and its replicas. This means that if a replica for a volume goes offline, that when the replica comes back online only the regions with changed blocks need to be synchronized. This optimization reduces the time it takes for replicas to catch up, improving volume resilience. Additionally, it reduces network and IO bandwidth which can reduce costs when running in public clouds.\n","excerpt":"StorageOS replicates volumes across nodes for data protection and high availability. Synchronous …","ref":"/docs/concepts/replication/","title":"Replication"},{"body":"Managing Resources for StorageOS containers Kubernetes resource requests and limits are two optional Pod properties that allow you to specify how much of a resource a container in a Pod needs or can use. They are two main resources that you can specify requests and limits for, CPU and Memory.\nAs StorageOS is an infrastructure component, the health of other applications depends on being able to write to the StorageOS volumes. As such it is of paramount importance to avoid restarts of the StorageOS DaemonSet Pods. Restarting a StorageOS Pod results in the volumes of the node the StorageOS Pod is running on being marked as Read Only, and causes the failover of primary volumes on that node to their replicas. After a StorageOS Pod restart, once the StorageOS DaemonSet Pod is \u0026ldquo;READY\u0026rdquo;, the application Pods running on the node need to be restarted in order to trigger a mount of the filesystem hosted on the StorageOS volume and resume normal operations. To avoid restarts of the StorageOS main container by Kubernetes due to resource limits being reached, it is recommended to not set resource limits on the StorageOS DaemonSet. In addition to avoiding resource limits, StorageOS uses a high priority class when the DaemonSet is installed in the \u0026lsquo;kube-system\u0026rsquo; namespace. That avoids the DaemonSet Pods of being evicted.\nFor more information about managing resources for containers please see the Kubernetes documentation\nDefining Pod resource requests and reservations To add resource requests and reservations to the StorageOS DaemonSet configure them in the StorageOSCluster resource.\n","excerpt":"Managing Resources for StorageOS containers Kubernetes resource requests and limits are two optional …","ref":"/docs/reference/resource_requests_and_limits/","title":"Resource Requests and Limits"},{"body":"","excerpt":"","ref":"/search/","title":"Search Results"},{"body":"Backing up files from StorageOS volumes In this example use case we provide three different strategies for accessing files that have been written to a StorageOS persistent volume.\nIn the following examples the \u0026ldquo;application\u0026rdquo; container is the container main, which has a rsync, Nginx or sftp sidecar container. The StorageOS volume that the application is writing to will be mounted into the sidecar container so files written by the application are available for export. Files can be exported using Nginx as a web file server, transferred using rsync or accessed via SFTP.\nThe files create a stateful set that can be used AFTER a StorageOS cluster has been created. See our guide on how to install StorageOS on Kubernetes for more information.\nClone Repository In order to deploy the examples, clone this repository and use kubectl to create the Kubernetes objects.\n$ git clone https://github.com/storageos/use-cases.git storageos-usecases $ cd storageos-usecases/backup  Before deploying the backup-example stateful set we recommend looking through the examples to understand how the different containers are configured\n Exfiltrating files through HTTP   Deploy the Nginx example\n$ kubectl create -f nginx/ service/backup-example created configmap/nginx-config created statefulset.apps/backup-example created pod/busybox created   Check that a backup-example pod is running\n$ kubectl get pods -w -l app=backup-example-nginx NAME READY STATUS RESTARTS AGE backup-example-0 1/1 Running 0 1m   Exec into the main container and write some data to a file\n$ kubectl exec -it backup-example-nginx-0 -c main bash root@backup-example-0:/# echo $(date) \u0026gt; /data/date.txt   Check that the service exists\n$ kubectl get svc backup-example-nginx NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE backup-example-nginx ClusterIP 100.65.18.199 \u0026lt;none\u0026gt; 80/TCP 46s   Use wget to access the files served by Nginx. Nginx is sharing files from the same volume that the main application container is writing to. The connection to the Nginx container is made via the backup-example service.\n$ kubectl exec -it busybox -- /bin/wget -q -O- http://backup-example-nginx \u0026lt;html\u0026gt; \u0026lt;head\u0026gt;\u0026lt;title\u0026gt;Index of /\u0026lt;/title\u0026gt;\u0026lt;/head\u0026gt; \u0026lt;body\u0026gt; \u0026lt;h1\u0026gt;Index of /\u0026lt;/h1\u0026gt;\u0026lt;hr\u0026gt;\u0026lt;pre\u0026gt;\u0026lt;a href=\u0026#34;../\u0026#34;\u0026gt;../\u0026lt;/a\u0026gt; \u0026lt;a href=\u0026#34;lost%2Bfound/\u0026#34;\u0026gt;lost+found/\u0026lt;/a\u0026gt; 12-Feb-2019 12:32 - \u0026lt;a href=\u0026#34;date.txt\u0026#34;\u0026gt;date.txt\u0026lt;/a\u0026gt; 12-Feb-2019 12:49 29 \u0026lt;/pre\u0026gt;\u0026lt;hr\u0026gt;\u0026lt;/body\u0026gt; \u0026lt;/html\u0026gt; $ kubectl exec -it busybox -- /bin/wget -q -O- http://backup-example-nginx/date.txt Tue Feb 12 12:49:15 UTC 2019   Depending on what files have been written to the StorageOS volume the output of the index file will be different. In the example the date.txt file we created in Step 2 is present on the volume.\nExfiltrating files through Rsync   Deploy the rsync example\n$ kubectl create -f rsync/ service/backup-example created configmap/rsync-config created secret/rsync-credentials created statefulset.apps/backup-example created pod/rsync created   Check that a backup-example pod is running\n$ kubectl get pods -w -l app=backup-example-rsync NAME READY STATUS RESTARTS AGE backup-example-0 1/1 Running 0 1m   Exec into the main container and write some data to a file\n$ kubectl exec -it backup-example-rsync-0 -c main bash root@backup-example-0:/# echo $(date) \u0026gt; /data/date.txt   Check that the service exists\n$ kubectl get svc backup-example-rsync NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE backup-example-rsync ClusterIP 100.65.18.199 \u0026lt;none\u0026gt; 873/TCP 46s   Use rsync to access the files shared by the rsync daemon. rsync is sharing files from the same volume that the main container is writing to. A username and password that are set in the rsync-credentials secret. The secret supplied in the example has the username and password set to username and password.\n$ kubectl exec -it rsync sh / # rsync --list-only rsync://username@backup-example-rsync/share Password: drwxr-xr-x 4,096 2019/02/12 12:49:15 . -rw-r--r-- 29 2019/02/12 12:49:15 date.txt drwx------ 16,384 2019/02/12 12:32:40 lost+found / # rsync -chavzP rsync://username@backup-example-rsync/share/date.txt . Password: receiving incremental file list date.txt 29 100% 28.32kB/s 0:00:00 (xfr#1, to-chk=0/1) sent 43 bytes received 135 bytes 50.86 bytes/sec total size is 29 speedup is 0.16 / # cat date.txt Tue Feb 12 12:49:15 UTC 2019   In the example above the list of available files was shown and a file called date.txt was synchronized to the rsync container.\nExfiltrating files through SFTP   Deploy the sftp example\n$ kubectl create -f sftp/   Exec into the main container and write some data to a file\n$ kubectl exec -it backup-example-sftp-0 -c main bash root@backup-example-0:/# echo $(date) \u0026gt; /data/date.txt   Check that the service exists\n$ kubectl get svc backup-example-sftp NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE backup-example-sftp ClusterIP 100.70.50.56 \u0026lt;none\u0026gt; 22/TCP 2h   Use SFTP to access the files shared by the SFTP container. If you have made no changes to the sftp-config secret the password is password.\n$ kubectl exec -it sftp -- bash root@sftp:/# sftp alex@backup-example-sftp alex@backup-example-sftp\u0026#39;s password: Connected to backup-example-sftp. sftp\u0026gt; ls date.txt lost+found sftp\u0026gt; get date.txt Fetching /date.txt to date.txt /date.txt 100% 29 15.9KB/s 00:00 sftp\u0026gt; bye root@sftp:/# cat date.txt Tue Feb 12 17:51:32 UTC 2019   In order to do this a SFTP user needs to be configured. The details for the user are stored in the sftp-config secret (see sftp/17-secret.yaml). The secret consists of base64 encoded username:password:uid:guid and the user is chroot\u0026rsquo;ed inside their home directory so the mount point for the StorageOS volume in the SFTP container in sftp/20-backup-pod.yaml needs to be configured.\nUsing custom SSH Keys The ConfigMap ssh-key-pub (see sftp/15-configmap.yaml) needs to be populated with a public key. The corresponding private key needs to be base64 encoded and put into the ssh-key-private secret (see sftp/17-secret.yaml). The user to connect as is determined by the user that is configured in the sftp-config configMap. To restrict logins to the SSH key edit the sftp-config secret so it contains no password (user::uid:guid).\n  Connect to the sftp pod and connect through the service to the SFTP container running inside the backup-example pod.\n$ kubectl exec -it sftp -- bash root@sftp:/# sftp -i /home/alex/.ssh/id_rsa alex@backup-example-sftp Connected to backup-example-sftp. sftp\u0026gt; ls date.txt lost+found   ","excerpt":"Backing up files from StorageOS volumes In this example use case we provide three different …","ref":"/docs/usecases/sidecar-backups/","title":""},{"body":"StorageClassess in Kubernetes are used to link PVCs with a backend storage provisioner - for instance, StorageOS. A StorageClass defines parameters to pass to the provisioner, which in the case of StorageOS can be translated into behaviour applied to the Volumes. Many StorageClasses can be provisioned to apply different feature labels to the StorageOS Volumes.\nBy default the StorageOS Cluster Operator installs the fast StorageClass at bootstrap of StorageOS. You can define its name in the StorageOS Cluster Resource.\napiVersion:storage.k8s.io/v1kind:StorageClassmetadata:name:fastlabels:app:storageosprovisioner:csi.storageos.com# CSI DriverallowVolumeExpansion:trueparameters:pool:defaultcsi.storage.k8s.io/fstype:ext4# Change the NameSpace below if StorageOS doesn\u0026#39;t run in kube-systemcsi.storage.k8s.io/controller-expand-secret-name:csi-controller-expand-secretcsi.storage.k8s.io/controller-publish-secret-name:csi-controller-publish-secretcsi.storage.k8s.io/node-publish-secret-name:csi-node-publish-secretcsi.storage.k8s.io/provisioner-secret-name:csi-provisioner-secretcsi.storage.k8s.io/controller-expand-secret-namespace:kube-system# NameSpace that runs StorageOS Daemonsetcsi.storage.k8s.io/controller-publish-secret-namespace:kube-system# NameSpace that runs StorageOS Daemonsetcsi.storage.k8s.io/node-publish-secret-namespace:kube-system# NameSpace that runs StorageOS Daemonsetcsi.storage.k8s.io/provisioner-secret-namespace:kube-system# NameSpace that runs StorageOS DaemonsetStorageClasses can be created to define default labels for StorageOS volumes, but also to map to any semantic aggregation of volumes that suits your use case, whether there are different roles (dev, staging, prod), or a StorageClass maps to a team or customer using the cluster.\nExamples You can find the basic examples in the StorageOS use-cases repository, in the 00-basic directory.\ngit clone https://github.com/storageos/use-cases.git storageos-usecases cd storageos-usecases/00-basic StorageClass definition in v2-storageclass-replicated.yaml.\napiVersion:storage.k8s.io/v1kind:StorageClassmetadata:name:storageos-replicatedprovisioner:csi.storageos.com# CSI DriverallowVolumeExpansion:trueparameters:fsType:ext4pool:defaultstorageos.com/replicas:\u0026#34;1\u0026#34;# Enforces 1 replica for the Volume# Change the NameSpace below if StorageOS doesn\u0026#39;t run in kube-systemcsi.storage.k8s.io/controller-expand-secret-name:csi-controller-expand-secretcsi.storage.k8s.io/controller-publish-secret-name:csi-controller-publish-secretcsi.storage.k8s.io/node-publish-secret-name:csi-node-publish-secretcsi.storage.k8s.io/provisioner-secret-name:csi-provisioner-secretcsi.storage.k8s.io/controller-expand-secret-namespace:kube-system# NameSpace that runs StorageOS Daemonsetcsi.storage.k8s.io/controller-publish-secret-namespace:kube-system# NameSpace that runs StorageOS Daemonsetcsi.storage.k8s.io/node-publish-secret-namespace:kube-system# NameSpace that runs StorageOS Daemonsetcsi.storage.k8s.io/provisioner-secret-namespace:kube-system# NameSpace that runs StorageOS DaemonsetThat StorageClass can be used by a PVC:\napiVersion:v1kind:PersistentVolumeClaimmetadata:name:my-vol-1spec:storageClassName:\u0026#34;storageos-replicated\u0026#34;# StorageOS StorageClassaccessModes:- ReadWriteOnceresources:requests:storage:5GiThe above StorageClass has the storageos.com/replicas label set. This label tells StorageOS to create a volume with a replica. Adding StorageOS feature labels to the StorageClass ensures all volumes created with the StorageClass have the same labels.\nYou can also choose to add the label in the PVC definition rather than the StorageClass. The PVC definition takes precedence over the SC.\n","excerpt":"StorageClassess in Kubernetes are used to link PVCs with a backend storage provisioner - for …","ref":"/docs/operations/storageclasses/","title":"Kubernetes StorageClasses"},{"body":"","excerpt":"","ref":"/","title":"StorageOS"},{"body":"StorageOS collects telemetry and error reports from StorageOS clusters via two different methods for two different purposes.\nTelemetry  Telemetry is made up a DNS version check query and a once per day report. Error reporting is the sentry.io crash dump reporting.  sentry.io StorageOS sends crash reports to sentry.io. This information helps our developers monitor and fix crashes. Information is sent to sentry.io when a process inside the StorageOS container crashes.\n The crash report contains the signal that triggered the shutdown (e.g. SIGSEGV), the exit code and whether or not the crash generated a core dump.  All StorageOS clusters with a routable connection to the internet will send crash reports to sentry.io over tcp/443. StorageOS respects environment variables that ProxyFromEnvironment uses.\nAn exhaustive list of information included in the crash report is below:\n StorageOS version Crash description string Anonymized Cluster ID Anonymized Node ID  DNS Query StorageOS will perform a \u0026ldquo;latest version check\u0026rdquo; using a DNS query in order to inform administrators that a new version is available. StorageOS will also send anonymized node ids, cluster id and StorageOS version information to StorageOS using a DNS query. The information that we send in the query is encoded as well as being anonymized. This query allows us to inform Cluster admins when StorageOS upgrades are available in the StorageOS GUI and in the logs.\nThe DNS query includes:\n Anonymized StorageOS Cluster ID Anonymized StorageOS node ID StorageOS version number  Once Per Day Report The once per day report contains information about the StorageOS cluster and Kubernetes versions to help StorageOS focus our development efforts on the most popular platforms. The once per day data is encrypted and sent to a StorageOS telemetry server so it is never processed outside of StorageOS assets.\nAn exhaustive list of information included in the once per day report is below:\n api_call_metrics cluster_disable_crash_reporting cluster_disable_version_check cluster_log_format cluster_log_level cluster_tls_provided k8s_distribution k8s_in_k8s k8s_scheduler_extender_enabled k8s_version node_available_bytes node_capacity node_crash_files_on_disk node_created_at_time node_etcd_config node_etcd_namespacing_enabled node_etcd_tls_enabled node_free_bytes node_health node_http_tls_enabled node_id node_labels node_status node_storageos_version node_system_clock_time node_total_bytes node_version volume_fs_type volume_id volume_labels volume_master_attach volume_master_delete_deployment volume_master_detach volume_master_failover_deployment volume_master_promote volume_master_provision volume_master_recover_replica volume_master_trigger_rejoin volume_metrics volume_placement_strategy volume_provision_source_user_agent volume_replica_delete_deployment volume_replica_failover_deployment volume_replica_promote volume_replica_provision volume_replicas volume_replica_trigger_rejoin volume_size_bytes  Disable Telemetry It is possible to disable telemetry using the GUI, CLI, API, environment variables or the StorageOS Cluster Spec.\nStorageOS Cluster Spec Disable telemetry explicitly through the configurable spec parameters of the StorageOSCluster custom resource.\nEnvironment Variables You can use the following environmental variables to disable or enable telemetry.\nDISABLE_VERSION_CHECK # Disable the DNS version check DISABLE_TELEMETRY # Disable the once per day reporting DISABLE_ERROR_REPORTING # Disable sentry.io crash reports ","excerpt":"StorageOS collects telemetry and error reports from StorageOS clusters via two different methods for …","ref":"/docs/reference/telemetry/","title":"StorageOS Telemetry"},{"body":"The support bundle is a tool that gathers information both about StorageOS and the environment in which it is operating. It provides information about the state and configuration of the cluster, nodes and other Kubernetes objects, as well as system performance metrics.\nThe support bundle is an addition rather than a replacement to the diagnostic bundle.\nInstall Kubectl StorageOS plugin The support bundle is generated with the StorageOS kubectl plugin, which can be installed as follows:\n$ curl -sL https://docs.storageos.com/sh/storageos-support-bundle-install.sh | bash If you prefer to run the installation commands individually, we provide them here:\n# Create a temporary directory in which to store the plugin binaries TMP_DIR=$(mktemp -d /tmp/storageos-kubectl-plugin-XXXXX) # Download the plugin binaries and extract to the temporary directory TARGET=\u0026#34;kubectl-storageos_linux_amd64.tar.gz\u0026#34; curl -sSL -o kubectl-storageos.tar.gz https://github.com/storageos/storageos.github.io/raw/master/sh/$TARGET tar -xf kubectl-storageos.tar.gz -C $TMP_DIR/ # Clean up the tar file rm -f kubectl-storageos.tar.gz # Add executable permissions for the plugin binaries and move into system path # For details of the bundle-generation tool\u0026#39;s functionality, visit: # https://docs.storageos.com/docs/bundles/support-bundle/ chmod +x $TMP_DIR/bin/kubectl-storageos-bundle sudo mv $TMP_DIR/bin/kubectl-storageos-bundle /usr/local/bin/  For MacOS build, use TARGET=kubectl-storageos_darwin_amd64.tar.gz\n Generate Bundle To generate a bundle, use the following command, the specification of which can be viewed and edited by obtaining the bundle-configuration.yaml file, which is publicly available.\n$ kubectl storageos bundle https://docs.storageos.com/yaml/bundle-configuration.yaml\n The tool expects there to be a StorageOS CLI running in kube-system as a Pod with the label run=cli. The tool will exec into this pod to get information from the StorageOS API. If the StorageOS CLI Pod does not match this criteria, you can either pull the YAML file and change the selector in the file, or add the label to the Pod. You can run the cli container following these instructions.\n Data collected in the bundle The data collected covers both the state of the StorageOS cluster, and in particular information regarding the infrastructure on which StorageOS is operating. It includes:\n StorageOS CLI information StorageOS Operator logs StorageOS logs Cluster metadata Cluster resources (DaemonSets, Events, Services, Pods, etc.) Backend disk configuration and performance statistics Load average Network checks across StorageOS nodes and ports Running processes  Privacy StorageOS can only obtain the bundle if it is downloaded by the user and given to our engineering team, or uploaded for analysis. The data received by StorageOS is private and never leaves nor will leave StorageOS Inc.\nThe data contained in the support bundle has the sole purpose of helping customers troubleshoot their issues.\n","excerpt":"The support bundle is a tool that gathers information both about StorageOS and the environment in …","ref":"/docs/reference/bundles/support_bundle/","title":"Support bundle"},{"body":"Host Filesystems StorageOS will automatically use /var/lib/storageos on each host as a base directory for storing configuration and blob files. Recommended host filesystem types are ext4 or xfs. If you require a specific filesystem please contact StorageOS.\nPersistent Volume Filesystems StorageOS provides a block device on which a file system can be created. The creation of the filesystem is either handled by StorageOS or by Kubernetes which affects what filesystems can be created.\nCSI Driver When using StorageOS with the CSI driver, StorageOS is responsible for running mkfs against the block device that pods mount. StorageOS is able to create ext2, ext3, ext4 and xfs file systems.\n","excerpt":"Host Filesystems StorageOS will automatically use /var/lib/storageos on each host as a base …","ref":"/docs/reference/filesystems/","title":"Supported File systems"},{"body":"Kubernetes Tolerations Tolerations are a Kubernetes pod property that allow pods to tolerate certain node taints. Taints can be thought of as the opposite of Node affinity in that taints repel pods. Node taints are automatically applied by Kubernetes in response to node resources coming under contention. As StorageOS provides storage to pods it should not be evicted during periods of resource contention, as any pods using StorageOS volumes on the same node would need to be restarted.\nAs StorageOS runs as a daemonset some Kubernetes tolerations are added by Kubernetes while others are automatically added by the StorageOS operator.\nFor more information about tolerations please see the Kubernetes documentation.\ntolerations:# The unreachable and not-ready tolerations are added by Kubernetes to daemonsets automatically- key:\u0026#34;node.kubernetes.io/unreachable\u0026#34;operator:\u0026#34;Exists\u0026#34;effect:\u0026#34;NoExecute\u0026#34;- key:\u0026#34;node.kubernetes.io/not-ready\u0026#34;operator:\u0026#34;Exists\u0026#34;effect:\u0026#34;NoExecute\u0026#34;# The following tolerations are added to the StorageOS daemonset by the StorageOS operator- key:node.kubernetes.io/disk-pressureoperator:Exists- key:node.kubernetes.io/memory-pressureoperator:Exists- key:node.kubernetes.io/network-unavailableoperator:Exists- key:node.kubernetes.io/out-of-diskoperator:Exists- key:node.kubernetes.io/pid-pressureoperator:Exists- key:node.kubernetes.io/unschedulableoperator:ExistsAdding Custom Tolerations To add custom tolerations to the StorageOS daemonset configure them in the StorageOSCluster resource.\n","excerpt":"Kubernetes Tolerations Tolerations are a Kubernetes pod property that allow pods to tolerate certain …","ref":"/docs/reference/tolerations/","title":"Tolerations"},{"body":"This section is aimed to help you troubleshoot issues in your cluster, whether they are related to the StorageOS installation, integration with orchestrators or common misconfigurations.\nTools To be able to troubleshoot issues the StorageOS cli is required.\nPod in pending because of mount error Issue: The output of kubectl describe pod $POD_ID contains no such file or directory and references the StorageOS volume device file.\nroot@node1:~# kubectl -n kube-system describe $POD_ID (...) Events: (...) Normal Scheduled 11s default-scheduler Successfully assigned default/d1 to node3 Warning FailedMount 4s (x4 over 9s) kubelet, node3 MountVolume.SetUp failed for volume \u0026#34;pvc-f2a49198-c00c-11e8-ba01-0800278dc04d\u0026#34; : stat /var/lib/storageos/volumes/d9df3549-26c0-4cfc-62b4-724b443069a1: no such file or directory Reason: There are two main reasons this issue may arise:\n The StorageOS DEVICE_DIR location is wrongly configured when using Kubelet as a container Mount Propagation is not enabled  (Option 1) Misconfiguration of the DeviceDir/SharedDir\nSome Kubernetes distributions such as Rancher, DockerEE or some installations of OpenShift deploy the Kubelet as a container, because of this, the device files that StorageOS creates to mount into the containers need to be visible to the kubelet. StorageOS can be configured to share the device directory.\nModern installations use CSI, which handles the complexity internally.\nAssert: root@node1:~# kubectl -n default describe stos | grep \u0026#34;Shared Dir\u0026#34; Shared Dir: # \u0026amp;lt;-- Shouldn\u0026#39;t be blank Solution: The Cluster Operator Custom Definition should specify the SharedDir option as follows.\nspec: sharedDir: \u0026#39;/var/lib/kubelet/plugins/kubernetes.io~storageos\u0026#39; # Needed when Kubelet as a container ...  See example on how to configure the StorageOS Custom Resource.\n  (Option 2) Mount propagation is not enabled.\n Applies only if Option 1 is configured properly.\n Assert: If not using the Kubelet as a container, SSH into one of the nodes and check if /var/lib/storageos/volumes is empty. If so, exec into any StorageOS pod and check the same directory.\nroot@node1:~# ls /var/lib/storageos/volumes/ root@node1:~# # \u0026amp;lt;-- Shouldn\u0026#39;t be blank root@node1:~# kubectl exec $POD_ID -c storageos -- ls -l /var/lib/storageos/volumes bst-196004 d529b340-0189-15c7-f8f3-33bfc4cf03fa ff537c5b-e295-e518-a340-0b6308b69f74 If the directory inside the container and the device files are visible, disabled mount propagation is the cause.\nIf using the Kubelet as a container, SSH into one of the nodes and check if /var/lib/kubelet/plugins/kubernetes.io~storageos/devices is empty. If so, exec into any StorageOS pod and check the same directory.\nroot@node1:~# ls /var/lib/kubelet/plugins/kubernetes.io~storageos/devices root@node1:~# # \u0026amp;lt;-- Shouldn\u0026#39;t be blank root@node1:~# kubectl exec $POD_ID -c storageos -- ls -l /var/lib/kubelet/plugins/kubernetes.io~storageos/devices bst-196004 d529b340-0189-15c7-f8f3-33bfc4cf03fa ff537c5b-e295-e518-a340-0b6308b69f74 If the directory inside the container and the device files are visible, disabled mount propagation is the cause.\nSolution: Older versions of Kubernetes need to enable mount propagation as it is not enabled by default. Most Kubernetes distributions allow MountPropagation to be enabled using FeatureGates. Rancher specifically, needs to enable it in the \u0026ldquo;View in API\u0026rdquo; section of your cluster. You need to edit the section \u0026ldquo;rancherKubernetesEngineConfig\u0026rdquo; to enable the Kubelet feature gate.\nPVC pending state - Failed to dial StorageOS A created PVC remains in pending state making pods that need to mount that PVC unable to start.\nIssue: root@node1:~/# kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE vol-1 Pending fast 7s kubectl describe pvc $PVC (...) Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning ProvisioningFailed 7s (x2 over 18s) persistentvolume-controller Failed to provision volume with StorageClass \u0026#34;fast\u0026#34;: Get http://storageos-cluster/version: failed to dial all known cluster members, (10.233.59.206:5705) Reason: For non CSI installations of StorageOS, Kubernetes uses the StorageOS API endpoint to communicate. If that communication fails, relevant actions such as create or mount volume can\u0026rsquo;t be transmitted to StorageOS, hence the PVC will remain in pending state. StorageOS never received the action to perform, so it never sent back an acknowledgement.\nIn this case, the Event message indicates that StorageOS API is not responding, implying that StorageOS is not running. For Kubernetes to define StorageOS pods ready, the health check must pass.\nAssert: Check the status of StorageOS pods.\nroot@node1:~/# kubectl -n kube-system get pod --selector app=storageos # for CSI add --selector kind=daemonset NAME READY STATUS RESTARTS AGE storageos-qrqkj 0/1 Running 0 1m storageos-s4bfv 0/1 Running 0 1m storageos-vcpfx 0/1 Running 0 1m storageos-w98f5 0/1 Running 0 1m If the pods are not READY, the service will not forward traffic to the API they serve hence PVC will remain in pending state until StorageOS pods are available.\n Kubernetes keeps trying to execute the action until it succeeds. If a PVC is created before StorageOS finish starting, the PVC will be created eventually.\n Solution:  StorageOS health check takes 60 seconds of grace before reporting as READY. If StorageOS is starting properly after that period, the volume will be created when StorageOS finishes its bootstrap. If StorageOS is not running or is not starting properly, the solution would be to troubleshoot the installation.  PVC pending state - Secret Missing A created PVC remains in pending state making pods that need to mount that PVC unable to start.\nIssue: kubectl describe pvc $PVC (...) Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning ProvisioningFailed 13s (x2 over 28s) persistentvolume-controller Failed to provision volume with StorageClass \u0026#34;fast\u0026#34;: failed to get secret from [\u0026#34;storageos\u0026#34;/\u0026#34;storageos-api\u0026#34;] Reason: For non CSI installations of StorageOS, Kubernetes uses the StorageOS API endpoint to communicate. If that communication fails, relevant actions such as create or mount a volume can\u0026rsquo;t be transmitted to StorageOS, and the PVC will remain in pending state. StorageOS never received the action to perform, so it never sent back an acknowledgement.\nThe StorageClass provisioned for StorageOS references a Secret from where it retrieves the API endpoint and the authentication parameters. If that secret is incorrect or missing, the connections won\u0026rsquo;t be established. It is common to see that the Secret has been deployed in a different namespace where the StorageClass expects it or that is has been deployed with a different name.\nAssert:   Check the StorageClass parameters to know where the Secret is expected to be found.\nkubectl get storageclass fast -o yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: creationTimestamp: 2018-09-25T08:44:57Z labels: app: storageos name: fast resourceVersion: \u0026#34;108853\u0026#34; selfLink: /apis/storage.k8s.io/v1/storageclasses/fast uid: 48490a9b-c09f-11e8-ba01-0800278dc04d parameters: adminSecretName: storageos-api adminSecretNamespace: storageos description: Kubernetes volume fsType: ext4 pool: default provisioner: kubernetes.io/storageos reclaimPolicy: Delete  Note that the parameters specify adminSecretName and adminSecretNamespace.\n   Check if the secret exists according to those parameters\nkubectl -n kube-system get secret storageos-api No resources found. Error from server (NotFound): secrets \u0026#34;storageos-api\u0026#34; not found If no resources are found, it is clear that the Secret doesn\u0026rsquo;t exist or it is not deployed in the right location.\n  Solution: Deploy StorageOS following the installation procedures. If you are using the manifests provided for Kubernetes to deploy StorageOS rather than using automated provisioners, make sure that the StorageClass parameters and the Secret reference match.\nNode name different from Hostname Issue: StorageOS nodes can\u0026rsquo;t join the cluster showing the following log entries.\ntime=\u0026#34;2018-09-24T13:47:02Z\u0026#34; level=error msg=\u0026#34;failed to start api\u0026#34; error=\u0026#34;error verifying UUID: UUID aed3275f-846b-1f75-43a1-adbfec8bf974 has already been registered and has hostname \u0026#39;debian-4\u0026#39;, not \u0026#39;node4\u0026#39;\u0026#34; module=command Reason: The StorageOS registration process to start the cluster uses the hostname of the node where the StorageOS container is running, provided by the Kubelet. However, StorageOS verifies the network hostname of the OS as a prestart check to make sure it can communicate with other nodes. If those names don\u0026rsquo;t match, StorageOS will be unable to start.\nSolution: Make sure the hostnames match with the Kubernetes advertised names. If you have changed the hostname of your nodes, make sure that you restart the nodes to apply the change.\nPeer discovery - Networking Issue: StorageOS nodes can\u0026rsquo;t join the cluster showing the following logs after one minute of container uptime.\ntime=\u0026#34;2018-09-24T13:40:20Z\u0026#34; level=info msg=\u0026#34;not first cluster node, joining first node\u0026#34; action=create address=172.28.128.5 category=etcd host=node3 module=cp target=172.28.128.6 time=\u0026#34;2018-09-24T13:40:20Z\u0026#34; level=error msg=\u0026#34;could not retrieve cluster config from api\u0026#34; status_code=503 time=\u0026#34;2018-09-24T13:40:20Z\u0026#34; level=error msg=\u0026#34;failed to join existing cluster\u0026#34; action=create category=etcd endpoint=\u0026#34;172.28.128.3,172.28.128.4,172.28.128.5,172.28.128.6\u0026#34; error=\u0026#34;503 Service Unavailable\u0026#34; module=cp time=\u0026#34;2018-09-24T13:40:20Z\u0026#34; level=info msg=\u0026#34;retrying cluster join in 5 seconds...\u0026#34; action=create category=etcd module=cp Reason: StorageOS uses a gossip protocol to discover nodes in the cluster. When StorageOS starts, one or more nodes can be referenced so new nodes can query existing nodes for the list of members. This error indicates that the node can\u0026rsquo;t connect to any of the nodes in the known list. The known list is defined in the JOIN variable.\nAssert: It is likely that ports are block by a firewall.\nSSH into one of your nodes and check connectivity to the rest of the nodes.\n# Successfull execution: [root@node06 ~]# nc -zv node04 5705 Ncat: Version 7.50 ( https://nmap.org/ncat ) Ncat: Connected to 10.0.1.166:5705. Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds. StorageOS exposes network diagnostics in its API, viewable from the CLI. To use this feature, the CLI must query the API of a running node. The diagnostics show information from all known cluster members. If all the ports are blocked during the first bootstrap of the cluster, the diagnostics won\u0026rsquo;t show any data as nodes couldn\u0026rsquo;t register.\n StorageOS networks diagnostics are available for storageos-rc5 and storageos-cli-rc3 and above.\n # Example: root@node1:~# storageos cluster connectivity SOURCE NAME ADDRESS LATENCY STATUS MESSAGE node4 node2.nats 172.28.128.4:5708 1.949275ms OK node4 node3.api 172.28.128.5:5705 3.070574ms OK node4 node3.nats 172.28.128.5:5708 2.989238ms OK node4 node2.directfs 172.28.128.4:5703 2.925707ms OK node4 node3.etcd 172.28.128.5:5707 2.854726ms OK node4 node3.directfs 172.28.128.5:5703 2.833371ms OK node4 node1.api 172.28.128.3:5705 2.714467ms OK node4 node1.nats 172.28.128.3:5708 2.613752ms OK node4 node1.etcd 172.28.128.3:5707 2.594159ms OK node4 node1.directfs 172.28.128.3:5703 2.601834ms OK node4 node2.api 172.28.128.4:5705 2.598236ms OK node4 node2.etcd 172.28.128.4:5707 16.650625ms OK node3 node4.nats 172.28.128.6:5708 1.304126ms OK node3 node4.api 172.28.128.6:5705 1.515218ms OK node3 node2.directfs 172.28.128.4:5703 1.359827ms OK node3 node1.api 172.28.128.3:5705 1.185535ms OK node3 node4.directfs 172.28.128.6:5703 1.379765ms OK node3 node1.etcd 172.28.128.3:5707 1.221176ms OK node3 node1.nats 172.28.128.3:5708 1.330122ms OK node3 node2.api 172.28.128.4:5705 1.238541ms OK node3 node1.directfs 172.28.128.3:5703 1.413574ms OK node3 node2.etcd 172.28.128.4:5707 1.214273ms OK node3 node2.nats 172.28.128.4:5708 1.321145ms OK node1 node4.directfs 172.28.128.6:5703 1.140797ms OK node1 node3.api 172.28.128.5:5705 1.089252ms OK node1 node4.api 172.28.128.6:5705 1.178439ms OK node1 node4.nats 172.28.128.6:5708 1.176648ms OK node1 node2.directfs 172.28.128.4:5703 1.529612ms OK node1 node2.etcd 172.28.128.4:5707 1.165681ms OK node1 node2.api 172.28.128.4:5705 1.29602ms OK node1 node2.nats 172.28.128.4:5708 1.267454ms OK node1 node3.nats 172.28.128.5:5708 1.485657ms OK node1 node3.etcd 172.28.128.5:5707 1.469429ms OK node1 node3.directfs 172.28.128.5:5703 1.503015ms OK node2 node4.directfs 172.28.128.6:5703 1.484ms OK node2 node1.directfs 172.28.128.3:5703 1.275304ms OK node2 node4.nats 172.28.128.6:5708 1.261422ms OK node2 node4.api 172.28.128.6:5705 1.465532ms OK node2 node3.api 172.28.128.5:5705 1.252768ms OK node2 node3.nats 172.28.128.5:5708 1.212332ms OK node2 node3.directfs 172.28.128.5:5703 1.192792ms OK node2 node3.etcd 172.28.128.5:5707 1.270076ms OK node2 node1.etcd 172.28.128.3:5707 1.218522ms OK node2 node1.api 172.28.128.3:5705 1.363071ms OK node2 node1.nats 172.28.128.3:5708 1.349383ms OK Solution: Open ports following the prerequisites page.\nPeer discovery - Pod allocation Issue: StorageOS nodes can\u0026rsquo;t join the cluster and show the following log entries.\ntime=\u0026#34;2018-09-24T13:40:20Z\u0026#34; level=info msg=\u0026#34;not first cluster node, joining first node\u0026#34; action=create address=172.28.128.5 category=etcd host=node3 module=cp target=172.28.128.6 time=\u0026#34;2018-09-24T13:40:20Z\u0026#34; level=error msg=\u0026#34;could not retrieve cluster config from api\u0026#34; status_code=503 time=\u0026#34;2018-09-24T13:40:20Z\u0026#34; level=error msg=\u0026#34;failed to join existing cluster\u0026#34; action=create category=etcd endpoint=\u0026#34;172.28.128.3,172.28.128.4,172.28.128.5,172.28.128.6\u0026#34; error=\u0026#34;503 Service Unavailable\u0026#34; module=cp time=\u0026#34;2018-09-24T13:40:20Z\u0026#34; level=info msg=\u0026#34;retrying cluster join in 5 seconds...\u0026#34; action=create category=etcd module=cp Reason: StorageOS uses a gossip protocol to discover the nodes in the cluster. When StorageOS starts, one or more active nodes must be referenced so new nodes can query existing nodes for the list of members. This error indicates that the node can\u0026rsquo;t connect to any of the nodes in the known list. The known list is defined in the JOIN variable.\nIf there are no active StorageOS nodes, the bootstrap process will elect the first node in the JOIN variable as master, and the rest will try to discover from it. In case of that node not starting, the whole cluster will remain unable to bootstrap.\nInstallations of StorageOS use a DaemonSet, and by default do not schedule StorageOS pods to master nodes, due to the presence of the node-role.kubernetes.io/master:NoSchedule taint that is typically present. In such cases the JOIN variable must not contain master nodes or the StorageOS cluster will remain unable to start.\nAssert: Check that the first node of the JOIN variable started properly.\nroot@node1:~/# kubectl -n kube-system describe ds/storageos | grep JOIN JOIN: 172.28.128.3,172.28.128.4,172.28.128.5 root@node1:~/# kubectl -n kube-system get pod -o wide | grep 172.28.128.3 storageos-8zqxl 1/1 Running 0 2m 172.28.128.3 node1 Solution: Make sure that the JOIN variable doesn\u0026rsquo;t specify the master nodes. In case you are using the discovery service, it is necessary to ensure that the DaemonSet won\u0026rsquo;t allocate Pods on the masters. This can be achieved with taints, node selectors or labels.\nFor installations with the StorageOS operator you can specify which nodes to deploy StorageOS on using nodeSelectors. See examples in the Cluster Operator Examples page.\nFor more advanced installations using compute-only and storage nodes, check the storageos.com/deployment=computeonly label that can be added to the nodes through Kubernetes node labels, or StorageOS in the Labels page.\nLIO Init:Error Issue: StorageOS pods not starting with Init:Error\nkubectl -n kube-system get pod NAME READY STATUS RESTARTS AGE storageos-2kwqx 0/3 Init:Err 0 6s storageos-cffcr 0/3 Init:Err 0 6s storageos-d4f69 0/3 Init:Err 0 6s storageos-nhq7m 0/3 Init:Err 0 6s Reason: This indicates that since the Linux open source SCSI drivers are not enabled, StorageOS cannot start. The StorageOS DaemonSet enables the required kernel modules on the host system. If you are seeing these errors it is because that container couldn\u0026rsquo;t load the modules.\nAssert Check the logs of the init container.\nkubectl -n kube-system logs $ANY_STORAGEOS_POD -c storageos-init In case of failure, it will show the following output, indicating which kernel modules couldn\u0026rsquo;t be loaded or that they are not properly configured:\nChecking configfs configfs mounted on sys/kernel/config Module target_core_mod is not running executing modprobe -b target_core_mod Module tcm_loop is not running executing modprobe -b tcm_loop modprobe: FATAL: Module tcm_loop not found. Solution: Install the required kernel modules (usually found in the linux-image-extra-$(uname -r) package of your distribution) on your nodes following this prerequisites page and delete StorageOS pods, allowing the DaemonSet to create the pods again.\nLIO not enabled Issue: StorageOS node can\u0026rsquo;t start and shows the following log entries.\ntime=\u0026#34;2018-09-24T14:34:40Z\u0026#34; level=error msg=\u0026#34;liocheck returned error\u0026#34; category=liocheck error=\u0026#34;exit status 1\u0026#34; module=dataplane stderr=\u0026#34;Sysfs root \u0026#39;/sys/kernel/config/target\u0026#39; is missing, is kernel configfs present and target_core_mod loaded? category=fslio level=warn\\nRuntime error checking stage \u0026#39;target_core_mod\u0026#39;: SysFs root missing category=fslio level=warn\\nliocheck: FAIL (lio_capable_system() returns failure) category=fslio level=fatal\\n\u0026#34; stdout= time=\u0026#34;2018-09-24T14:34:40Z\u0026#34; level=error msg=\u0026#34;failed to start dataplane services\u0026#34; error=\u0026#34;system dependency check failed: exit status 1\u0026#34; module=command Reason: This indicates that one or more kernel modules required for StorageOS are not loaded.\nAssert The following kernel modules must be enabled in the host.\nlsmod | egrep \u0026#34;^tcm_loop|^target_core_mod|^target_core_file|^configfs\u0026#34; Solution: Install the required kernel modules (usually found in the linux-image-extra-$(uname -r) package of your distribution) on your nodes following this prerequisites page and restart the container.\n(OpenShift) StorageOS pods missing \u0026ndash; DaemonSet error StorageOS DaemonSet doesn\u0026rsquo;t have any pod replicas. The DaemonSet couldn\u0026rsquo;t allocate any Pod due to security issues.\nIssue: [root@master02 standard]# oc get pod No resources found. [root@master02 standard]# oc describe daemonset storageos (...) Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedCreate 0s (x12 over 10s) daemonset-controller Error creating: pods \u0026#34;storageos-\u0026#34; is forbidden: unable to validate against any security context constraint: [provider restricted: .spec.securityContext.hostNetwork: Invalid value: true: Host network is not allowed to be used provider restricted: .spec.securityContext.hostPID: Invalid value: true: Host PID is not allowed to be used spec.volumes[0]: Invalid value: \u0026#34;hostPath\u0026#34;: hostPath volumes are not allowed to be used spec.volumes[1]: Invalid value: \u0026#34;hostPath\u0026#34;: hostPath volumes are not allowed to be used spec.volumes[2]: Invalid value: \u0026#34;hostPath\u0026#34;: hostPath volumes are not allowed to be used spec.volumes[3]: Invalid value: \u0026#34;hostPath\u0026#34;: hostPath volumes are not allowed to be used spec.initContainers[0].securityContext.privileged: Invalid value: true: Privileged containers are not allowed capabilities.add: Invalid value: \u0026#34;SYS_ADMIN\u0026#34;: capability may not be added spec.initContainers[0].securityContext.hostNetwork: Invalid value: true: Host network is not allowed to be used spec.initContainers[0].securityContext.containers[0].hostPort: Invalid value: 5705: Host ports are not allowed to be used spec.initContainers[0].securityContext.hostPID: Invalid value: true: Host PID is not allowed to be used spec.containers[0].securityContext.privileged: Invalid value: true: Privileged containers are not allowed capabilities.add: Invalid value: \u0026#34;SYS_ADMIN\u0026#34;: capability may not be added spec.containers[0].securityContext.hostNetwork: Invalid value: true: Host network is not allowed to be used spec.containers[0].securityContext.containers[0].hostPort: Invalid value: 5705: Host ports are not allowed to be used spec.containers[0].securityContext.hostPID: Invalid value: true: Host PID is not allowed to be used] Reason: The OpenShift cluster has security context constraint policies enabled that forbid any pod, without an explicitly set policy for the service account, to be allocated.\nAssert: Check if the StorageOS ServiceAccount can create pods with enough permissions\noc get scc privileged -o yaml # Or custom scc with enough privileges (...) users: - system:admin - system:serviceaccount:openshift-infra:build-controller - system:serviceaccount:management-infra:management-admin - system:serviceaccount:management-infra:inspector-admin - system:serviceaccount:storageos:storageos \u0026amp;lt;-- - system:serviceaccount:tiller:tiller If the StorageOS sa system:serviceaccount:storageos:storageos is in the privileged scc it will be able to create pods.\nSolution: Add the ServiceAccount system:serviceaccount:storageos:storageos to a scc with enough privileges.\noc adm policy add-scc-to-user privileged system:serviceaccount:storageos:storageos Getting Help If our troubleshooting guides do not help resolve your issue, please see our support section for details on how to get in touch with us.\n","excerpt":"This section is aimed to help you troubleshoot issues in your cluster, whether they are related to …","ref":"/docs/operations/troubleshooting/","title":"Troubleshooting"},{"body":"This document details a step-by-step procedure to remove StorageOS from a Kubernetes cluster.\nRemember that StorageOS enables the stateful applications within your cluster. It is very important to remove any applications that rely on StorageOS before you remove StorageOS itself, or those applications will suffer unrecoverable errors.\nRemove Stateful Workloads and Data   Delete any resources using StorageOS volumes\nDelete any statefulsets, deployments or pods that are using StorageOS Volumes.\n  Delete PVCs using StorageOS\nDelete any Persistent Volume Claims that are using StorageOS.\n$ kubectl -n $NS delete pvc $PVC  This will delete data held by StorageOS and won\u0026rsquo;t be recoverable.\n   Remove StorageOS Cluster   Delete StorageOS Cluster\n$ kubectl get storageoscluster --all-namespaces # Find the namespace where the Custom Resource runs $ kubectl -n $NS delete storageoscluster --all # Usually to be found in storageos-operator   Wait until the StorageOS resources are gone\n$ kubectl -n kube-system get pod # NS: Namespace where StorageOS Daemonset is running, usually \u0026#39;kube-system\u0026#39;   Uninstall the StorageOS Operator  Delete the Cluster Operator once the StorageOS Pods are terminated\n   Delete the StorageOS Operator deployment\n$ kubectl delete -f https://github.com/storageos/cluster-operator/releases/download/v2.3.1/storageos-operator.yaml   Procedure is finished. StorageOS is now uninstalled.\nRemove StorageOS contents and metadata (unrecoverable) The steps up until now have been recoverable - as long as the etcd backing StorageOS and the contents of /var/lib/storageos on your nodes are safe then StorageOS can be reinstalled. For complete removal and recovery of disk space, you can use the following procedure.\n The following steps will delete all data held by StorageOS and won\u0026rsquo;t be recoverable.\n   Remove the StorageOS data directory\nThere are two ways to remove the StorageOS data directory:\n  (Option 1) Login in to the hosts and execute the following commands\n$ sudo rm -rf /var/lib/storageos $ sudo umount /var/lib/kubelet/plugins_registry/storageos   (Option 2) Execute the following command to deploy a DaemonSet that removes the StorageOS data directory.\n N.B This step is irreversible and once the data is removed it cannot be recovered.\n  Run the following command where kubectl is installed and with the context set for your Kubernetes cluster.\n $ curl -s https://docs.storageos.com/sh/permanently-delete-storageos-data.sh | bash     Flush Etcd Data\nThis will remove any keys written by StorageOS.\n$ export ETCDCTL_API=3 $ etcdctl --endpoints=http://$ETCD_IP:2379 del --prefix \u0026#34;storageos\u0026#34; If running Etcd with mTLS, you can set the certificates location with the following command.\n$ export ETCDCTL_API=3 $ etcdctl --endpoints=https://$ETCD_IP:2379 \\  --cacert=/path/to/ca.pem \\  --cert=/path/to/client-cert.pem \\  --key=/path/to/client-key.pem \\  del --prefix \u0026#34;storageos\u0026#34;   ","excerpt":"This document details a step-by-step procedure to remove StorageOS from a Kubernetes cluster. …","ref":"/docs/operations/uninstall/","title":"Uninstall StorageOS"},{"body":"This document details a step-by-step procedure to upgrade a StorageOS v2 cluster.\nKeep in mind that upgrading a cluster will require minor downtime of applications using StorageOS volumes. However we will take steps to minimize the required downtime as much as possible.\nUpgrade StorageOS  Ensure that you have read the PIDs prerequisite introduced in StorageOS v2.3 and that you check the init container logs to ensure your environments PID limits are set correctly.\n  Warning: To reduce downtime, it is recommended to docker pull the new StorageOS container image storageos/node:v2.3.2 on the nodes beforehand so that the cluster spins up faster!\n   First make sure you keep a backup of all the StorageOS yaml files. You can reuse the StorageOSCluster configuration file to easily upgrade your cluster. You can also backup the Statefulset yaml files to keep track of the replicas.\nkubectl get pod -n storageos-operator -o yaml \u0026gt; storageos_operator.yaml kubectl get storageoscluster -n storageos-operator -o yaml \u0026gt; storageos_cr.yaml kubectl get statefulset --all-namespaces \u0026gt; statefulset-sizes.yaml   Scale all stateful applications that use StorageOS volumes to 0.\n  Delete the StorageOSCluster CR.\nkubectl delete storageoscluster cluster-storageos -n storageos-operator If you have renamed the StorageOSCluster resource you can find it by using the command below.\nkubectl get storageoscluster --all-namespaces   Deploy the new operator.\nkubectl apply -f https://github.com/storageos/cluster-operator/releases/download/v2.3.1/storageos-operator.yaml  Warning: If you have made changes to the CRDs, Service Account or Cluster Role, make sure you migrate those changes in the StorageOS operator yaml.\n   Edit the StorageOSCluster Custom Resource (storageos_cr.yaml) with the new node image version.\nimages:nodeContainer:\u0026#34;storageos/node:v2.3.2\u0026#34;If you\u0026rsquo;re not using any of the hardcoded container image versions make sure to delete them as well and only leave the nodeContainer image as shown above.\nMake sure to also delete all metadata that you\u0026rsquo;re not using. The final metadata stanza should be something similar to that:\nmetadata:name:\u0026#34;cluster-storageos\u0026#34;namespace:\u0026#34;storageos-operator\u0026#34;  Re-create the StorageOSCluster\nkubectlcreate-fstorageos_cr.yaml  Wait for all the StorageOS pods to enter the RUNNING state\nkubectl get pods -l app=storageos -A -w   Scale your stateful applications back up.\n  Congratulations, you now have the latest version StorageOS!\n","excerpt":"This document details a step-by-step procedure to upgrade a StorageOS v2 cluster.\nKeep in mind that …","ref":"/docs/operations/upgrade/","title":"Upgrade StorageOS"},{"body":"A StorageOS cluster admin can create users and restrict their access rights to StorageOS namespaces using policies.\n Note: Users are created with access to the default namespace. This access is only revoked when a policy is created for the user or their group.\n Managing users Users can be created and deleted by navigating to the \u0026ldquo;Users\u0026rdquo; section of the GUI.\nAlternatively users can also be managed using the CLI.\nTo create a user with the CLI, run:\n$ storageos user create jim The above command will create a user named jim. The command will also prompt you to enter a password for the newly created user.\n","excerpt":"A StorageOS cluster admin can create users and restrict their access rights to StorageOS namespaces …","ref":"/docs/operations/users/","title":"User Management"},{"body":"Velero Backups Using Velero to backup your StorageOS volumes Velero is an open source tool to safely backup and restore, perform disaster recovery, and migrate Kubernetes cluster resources and persistent volumes.\nVelero consists of the following components:\n A Velero Server in the cluster A CLI client A restic daemonset in the cluster  Here\u0026rsquo;s a diagram on how Velero backups work:\nAnd here\u0026rsquo;s how Velero restores work:\nIn our case, we will use MinIO in the Kubernetes cluster as the \u0026ldquo;cloud provider\u0026rdquo; pictured in the backup diagram. MinIO is an object store that uses an S3 compatible API and as such can be used to store our backed up resources.\nWe will set up MinIO through a StatefulSet with a 50GiB StorageOS volume and a ClusterIP service. The UI can be accessed through the browser by port-forwarding the MinIO service.\nVelero uses Restic to backup Kubernetes volumes. Restic is a fast and secure backup program for filesystems whose documentation can be found here. The way it works is that it scans the volume directory for its files and then splits those files into blobs which are then sent to MinIO. Here\u0026rsquo;s how it integrates with Velero.\nPrerequisites Here are the prerequisites for running Velero in your Kubernetes cluster:\n Kubernetes cluster version 1.13+ with DNS Kubectl installed Velero cli installed https://Velero.io/docs/main/basic-install/   Velero can also be installed from a helm chart\n Install MinIO with a StorageOS volume   First, make sure to clone the StorageOS use cases repository and navigate to the Velero directory:\ngit clone https://github.com/storageos/use-cases.git cd use-cases/Velero   Installing MinIO is really simple, just deploy it using the minio-deploy.yaml manifest file:\nkubectl apply -f ./minio   Confirm that MinIO was deployed successfully:\n$ kubectl get pods -n velero NAME READY STATUS RESTARTS AGE minio-0 1/1 Running 0 3m48s minio-setup-zvcdg 0/1 Completed 1 3m47s You can access the web UI of MinIO by port-forwarding the MinIO service with this command:\nkubectl port-forward service/minio -n velero 9000   Install Velero Use the following command to install Velero via the Velero CLI or alternatively use the helm chart. To install it using the Velero cli, just run this command:\n The AWS plugin is being used because MinIO implements the S3 API. This is required even if you\u0026rsquo;re not using AWS.\n velero install \\ --provider aws \\ --plugins velero/velero-plugin-for-aws:v1.0.0 \\ --bucket velero \\ --secret-file ./credentials-Velero \\ --use-volume-snapshots=false \\ --backup-location-config region=minio,s3ForcePathStyle=\u0026#34;true\u0026#34;,s3Url=http://minio.velero.svc:9000 \\ --use-restic Make sure that Velero is installed correctly:\n$ kubectl logs deployment/velero -n velero ... time=\u0026#34;2020-08-25T15:33:09Z\u0026#34; level=info msg=\u0026#34;Server started successfully\u0026#34; logSource=\u0026#34;pkg/cmd/server/server.go:881\u0026#34; time=\u0026#34;2020-08-25T15:33:09Z\u0026#34; level=info msg=\u0026#34;Starting controller\u0026#34; controller=restic-repository logSource=\u0026#34;pkg/controller/generic_controller.go:76\u0026#34; time=\u0026#34;2020-08-25T15:33:09Z\u0026#34; level=info msg=\u0026#34;Starting controller\u0026#34; controller=restore logSource=\u0026#34;pkg/controller/generic_controller.go:76\u0026#34; time=\u0026#34;2020-08-25T15:33:09Z\u0026#34; level=info msg=\u0026#34;Starting controller\u0026#34; controller=backup-sync logSource=\u0026#34;pkg/controller/generic_controller.go:76\u0026#34; time=\u0026#34;2020-08-25T15:33:09Z\u0026#34; level=info msg=\u0026#34;Starting controller\u0026#34; controller=backup logSource=\u0026#34;pkg/controller/generic_controller.go:76\u0026#34; time=\u0026#34;2020-08-25T15:33:09Z\u0026#34; level=info msg=\u0026#34;Starting controller\u0026#34; controller=backup-deletion logSource=\u0026#34;pkg/controller/generic_controller.go:76\u0026#34; time=\u0026#34;2020-08-25T15:33:09Z\u0026#34; level=info msg=\u0026#34;Checking for expired DeleteBackupRequests\u0026#34; controller=backup-deletion logSource=\u0026#34;pkg/controller/backup_deletion_controller.go:551\u0026#34; time=\u0026#34;2020-08-25T15:33:09Z\u0026#34; level=info msg=\u0026#34;Done checking for expired DeleteBackupRequests\u0026#34; controller=backup-deletion logSource=\u0026#34;pkg/controller/backup_deletion_controller.go:579\u0026#34; time=\u0026#34;2020-08-25T15:33:09Z\u0026#34; level=info msg=\u0026#34;Starting controller\u0026#34; controller=schedule logSource=\u0026#34;pkg/controller/generic_controller.go:76\u0026#34; time=\u0026#34;2020-08-25T15:33:09Z\u0026#34; level=info msg=\u0026#34;Starting controller\u0026#34; controller=downloadrequest logSource=\u0026#34;pkg/controller/generic_controller.go:76\u0026#34; time=\u0026#34;2020-08-25T15:33:09Z\u0026#34; level=info msg=\u0026#34;Starting controller\u0026#34; controller=gc-controller logSource=\u0026#34;pkg/controller/generic_controller.go:76\u0026#34; time=\u0026#34;2020-08-25T15:33:09Z\u0026#34; level=info msg=\u0026#34;Starting controller\u0026#34; controller=serverstatusrequest logSource=\u0026#34;pkg/controller/generic_controller.go:76\u0026#34; Quiesce Before moving on to the MySQL use case, it is important to talk about quiescence. To quiesce is to pause or alter a device or application to achieve a consistent state, usually in preparation for a backup or other maintenance. Quiescence is very important when backing up real world applications because in flight or cached write operations of the filesystem during the backup process can cause corruption of the backed up data. It is a common mistake for engineers to back up their data without quiescing the filesystem and noticing corruption when they need to restore from that backup. A simple form of quiescence is the unix command fsfreeze, but in most cases, application specific quiescence should be used instead.\nMySQL use case For the our use case, we\u0026rsquo;ll use Velero with a real world application, MySQL.\n  First deploy MySQL, based on the MySQL use case:\nkubectl apply -f ./mysql Notice that the Statefulset also includes 5 annotations:\nannotations:backup.velero.io/backup-volumes:datapre.hook.backup.velero.io/command:\u0026#39;[\u0026#34;/sbin/fsfreeze\u0026#34;, \u0026#34;--freeze\u0026#34;, \u0026#34;/var/lib/mysql\u0026#34;]\u0026#39;pre.hook.backup.velero.io/container:fsfreezepost.hook.backup.velero.io/command:\u0026#39;[\u0026#34;/sbin/fsfreeze\u0026#34;, \u0026#34;--unfreeze\u0026#34;, \u0026#34;/var/lib/mysql\u0026#34;]\u0026#39;post.hook.backup.velero.io/container:fsfreezeThe first annotation specifies which volume to backup using restic. The other annotations are used to perform an fsfreeze on the volume mount point using pre and post backup hooks, for more details about Velero pre/post backup hooks please see their documentation here.\nWe have to specify to use the fsfreeze ubuntu container since the MySQL container doesn\u0026rsquo;t support fsfreeze\n- name:fsfreezeimage:ubuntu:bionicsecurityContext:privileged:truevolumeMounts:- name:datamountPath:/var/lib/mysqlcommand:- \u0026#34;/bin/bash\u0026#34;- \u0026#34;-c\u0026#34;- \u0026#34;sleep infinity\u0026#34;  Wait for the pod to spin up:\n$ kubectl get pods -n mysql NAME READY STATUS RESTARTS AGE client 1/1 Running 0 24m mysql-0 2/2 Running 0 24m   Exec into the MySQL pod and populate it with data using the commands below.\n$ kubectl exec mysql-0 -n mysql -ti -c mysql -- mysql mysql\u0026gt; create database shop; mysql\u0026gt; use shop; mysql\u0026gt; create table books (title VARCHAR(256), price decimal(4,2)); mysql\u0026gt; insert into books value (\u0026#39;Gates of Fire\u0026#39;, 13.99); mysql\u0026gt; select * from books; +---------------+-------+ | title | price | +---------------+-------+ | Gates of Fire | 13.99 | +---------------+-------+ 1 row in set (0.00 sec) mysql\u0026gt; exit   Create the Velero backup:\nvelero backup create mysql-backup --include-namespaces mysql --wait   Confirm that all the Kubernetes objects are there and the restic backups completed successfully:\nvelero backup describe mysql-backup --details   After the backup is finished, delete the StatefulSet and PVC.\n N.B. It\u0026rsquo;s important to make sure that the StatefulSet is deleted because the restore would be unable to complete if a StatefulSet pod is recreated during the restore process.\n kubectl delete statefulset mysql -n mysql kubectl delete pvc data-mysql-0 -n mysql   Make sure that the pod is fully terminated:\n$ kubectl get pods -n mysql NAME READY STATUS RESTARTS AGE client 1/1 Running 0 25m   Restore MySQL using Velero:\nvelero restore create --from-backup mysql-backup --wait   Wait for the MySQL pod to spin up and see if the data is backed up:\n$ kubectl exec mysql-0 -n mysql -ti -c mysql -- mysql mysql\u0026gt; use shop; mysql\u0026gt; select * from books; +---------------+-------+ | title | price | +---------------+-------+ | Gates of Fire | 13.99 | +---------------+-------+ 1 row in set (0.00 sec) mysql\u0026gt; exit   The data should be there and the backup was restored successfully!\n","excerpt":"Velero Backups Using Velero to backup your StorageOS volumes Velero is an open source tool to safely …","ref":"/docs/usecases/velero-backups/","title":""},{"body":"$ storageos version --help View version information for the StorageOS CLI Usage: storageos version [flags] Flags: -h, --help help for version Global Flags: --endpoints stringArray set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705]) -n, --namespace string specifies the namespace to operate within for commands that require one (default \u0026#34;default\u0026#34;) -o, --output string specifies the output format (one of [json yaml text]) (default \u0026#34;text\u0026#34;) --password string set the StorageOS account password to authenticate with (default \u0026#34;storageos\u0026#34;) --timeout duration set the timeout duration to use for execution of the command (default 5s) --use-ids specify existing StorageOS resources by their unique identifiers instead of by their names --username string set the StorageOS account username to authenticate as (default \u0026#34;storageos\u0026#34;) ","excerpt":"$ storageos version --help View version information for the StorageOS CLI Usage: storageos version …","ref":"/docs/reference/cli/version/","title":"Version"},{"body":"StorageOS supports offline resize of volumes, either through editing a PVC storage request, or by updating the volume config via the CLI or UI. A volume cannot be resized while it is in use. To ensure that it is not, for a resize operation to take place the volume must not be attached to a node. This requires that any pods using a volume be scaled down for the resize to take place.\nPlease note that StorageOS only supports increasing volume size. For more information about how the resize works please see our Resize Concepts page.\nResizing a Volume In order to resize a PVC the storage request field must be updated.\napiVersion: v1 kind: PersistentVolumeClaim metadata: name: pvc-1 spec: storageClassName: fast accessModes: - ReadWriteOnce resources: requests: storage: 5Gi The Volume must be offline for this to take effect. Therefore, before performing the operation, scale down any pods using the volume. This will ensure that it is not in use.\nIn order to edit a PVC you can use kubectl edit or kubectl apply to make changes.\n N.B. Resizing a volume without updating the PVC directly will NOT result in the PVC being updated. The methods below are included for completeness. In Kubernetes environments editing the PVC is the preferred method for resizing a volume.\n To resize a volume using the StorageOS CLI use the volume update command\n$ storageos update volume size pvc-a47cfa03-cc92-4ec9-84ab-00e5516c64fa 10GiB Name: pvc-a47cfa03-cc92-4ec9-84ab-00e5516c64fa ID: 925e667f-91d3-465a-9391-8fdb56d0c9ff Size: 11 GB Description: AttachedOn: Replicas: 1x ready Labels: - csi.storage.k8s.io/pv/name pvc-a47cfa03-cc92-4ec9-84ab-00e5516c64fa - csi.storage.k8s.io/pvc/name pvc-1 - csi.storage.k8s.io/pvc/namespace default - foo bar - pool default - storageos.com/replicas 1 Volume pvc-a47cfa03-cc92-4ec9-84ab-00e5516c64fa (925e667f-91d3-465a-9391-8fdb56d0c9ff) updated. Size changed. To resize a volume using the StorageOS UI, navigate to the volumes section and click the edit pencil in order to update the volume config.\n","excerpt":"StorageOS supports offline resize of volumes, either through editing a PVC storage request, or by …","ref":"/docs/operations/resize/","title":"Volume Resize"},{"body":"StorageOS volumes are a logical construct which represent a writeable volume and exhibit standard POSIX semantics. StorageOS presents volumes as mounts into containers via the Linux LIO subsystem.\nConceptually, StorageOS volumes have a frontend presentation, which is what the application sees, and a backend presentation, which is the actual on-disk format. Depending on the configuration, frontend and backend components may be on the same or different hosts.\nVolumes are formatted using the linux standard ext4 filesystem by default. Kubernetes users may change the default filesystem type to ext2, ext3, ext4, or xfs by setting the fsType parameter in their StorageClass (see Supported Filesystems for more information). Different filesystems may be supported in the future.\nStorageOS volumes are represented on disk in two parts. Actual volume data is written to blob files in /var/lib/storageos/data/dev[\\d+]. Inside these directories, each StorageOS block device gets two blob files of the form vol.xxxxxx.y.blob, where x is the inode number for the device, and y is an index between 0 and 1. We provide two blob files in order to ensure that certain operations which require locking do not impede in-flight writes to the volume.\nIn systems which have multiple /var/lib/storageos/data/dev[\\d+] directories, two blob files are created per block device. This allows us to load-balance writes across multiple devices. In cases where dev directories are added after a period of run time, later directories are favoured for writes until the data is distributed evenly across the blob files.\nMetadata is kept in directories named /var/lib/storageos/data/db[\\d+]. We maintain an index of all blocks written to the blob file inside the metadata store, including checksums. These checksums allow us to detect bitrot, and return errors on reads, rather than serve bad data. In future versions we may implement recovery from replicas for volumes with one or more replicas defined.\nStorageOS metadata requires approximately 2.7GB of storage per 1TB of allocated blocks in the associated volume. This size is consistent irrespective of data compression defined on the volume.\nTo ensure deterministic performance, individual StorageOS volumes must fit on a single node.\nVolume Resize StorageOS v2.1 supports offline resize of volumes. This means that a volume cannot be resized while it is in use. Furthermore, in order for a resize operation to take place the volume must not be attached to a node. This is to ensure that the volume is not in use.\nThis means that if a Kubernetes pod is currently consuming a volume that a resize request has been issued for, the resize will not be actioned until the pod is terminated and the volume is detached from the node. The StorageOS controlplane will then attach the volume to the node that holds the master deployment and resize the underlying block device and then run resize2fs to expand the filesystem.\nFor a walk through of how to resize a volume please see the Volume Resize operations page.\n","excerpt":"StorageOS volumes are a logical construct which represent a writeable volume and exhibit standard …","ref":"/docs/concepts/volumes/","title":"Volumes"},{"body":"Zookeeper with StorageOS  ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.\nUsing StorageOS persistent volumes with Apache Zookeeper means that if a pod fails, the cluster is only in a degraded state for as long as it takes Kubernetes to restart the pod. When the pod comes back up, the pod data is immediately available. Should Kubernetes schedule the Zookeeper pod on a new node, StorageOS allows for the data to be available to the pod, irrespective of whether or not the original StorageOS master volume is located on the same node.\nAs Zookeeper has features to allow it to handle replication, and as such careful consideration of whether to allow StorageOS or Zookeeper to handle replication is required.\nBefore you start, ensure you have StorageOS installed and ready on a Kubernetes cluster. See our guide on how to install StorageOS on Kubernetes for more information.\nDeploying Zookeeper on Kubernetes Pre-requisites  StorageOS is assumed to have been installed; please check for the latest available version here.  Helm To simplify the deployment of Zookeeper, we\u0026rsquo;ve used this Zookeeper helm chart (incubator) (version 1.2.2, app version 3.4.10) and rendered it into the example deployment files you can find in our GitHub repo.\nDeployment Clone the use cases repo You can find the latest files in the StorageOS use cases repository in /zookeeper/\ngit clone https://github.com/storageos/use-cases.git storageos-usecases StatefulSet defintion ---apiVersion:apps/v1beta1kind:StatefulSetmetadata:name:zookeeper...spec:replicas:3# \u0026lt;--- number of zookeeper pods...containers:- name:zookeeperimage:\u0026#34;gcr.io/google_samples/k8szk:v3\u0026#34;imagePullPolicy:IfNotPresent...volumeMounts:- name:datamountPath:/var/lib/zookeepervolumeClaimTemplates:- metadata:name:dataspec:accessModes:- \u0026#34;ReadWriteOnce\u0026#34;storageClassName:\u0026#34;fast\u0026#34;# \u0026lt;--- the StorageClass to useresources:requests:storage:\u0026#34;5Gi\u0026#34;# \u0026lt;--- storage requested per podThis excerpt is from the StatefulSet definition (10-statefulset.yaml). The file contains the PersistentVolumeClaim template that will dynamically provision the necessary storage, using the StorageOS storage class. Dynamic provisioning occurs as a volumeMount has been declared with the same name as a VolumeClaimTemplate.\nCreate the kubernetes objects cd storageos-usecases kubectl apply -f ./zookeeper/ Confirm Zookeeper is up and running $ kubectl get pods NAME READY STATUS RESTARTS AGE zookeeper-0 1/1 Running 0 2m30s zookeeper-1 1/1 Running 0 112s zookeeper-2 1/1 Running 0 56s zookeeper-test-client 1/1 Running 0 2m30s Connect to Zookeeper Connect to the zookeeper client pod and list existing topics using the service endpoint\nkubectl exec -it zookeeper-test-client /bin/bash and issue a command to the zookeeper service\nzkCli.sh -server zookeeper ls /zookeeper ","excerpt":"Zookeeper with StorageOS  ZooKeeper is a centralized service for maintaining configuration …","ref":"/docs/usecases/zookeeper/","title":""}]