Skip to content

Latest commit

 

History

History
209 lines (166 loc) · 6.63 KB

container-storage-interface.md

File metadata and controls

209 lines (166 loc) · 6.63 KB
Raw

Container storage interface

This project implements a csi plugin for Kubernetes clusters running on Oracle Cloud Infrastructure (OCI). It enables provisioning, attach, detach, mount and unmount of OCI block storage volumes to Kubernetes Pods via the CSI plugin interface.

Install / Setup

We publish two binaries oci-csi-controller-driver which runs on the master nodes, and oci-csi-node-controller which runs on each of the worker nodes.

Submit configuration as a Kubernetes secret

Create a config.yaml file with contents similar to the following. This file will contain authentication information necessary to authenticate with the OCI APIs and provision block storage volumes. An example configuration file can be found here

Submit this as a Kubernetes Secret.

kubectl create secret generic oci-volume-provisioner \
    -n kube-system \
    --from-file=config.yaml=provider-config-example.yaml

Installer

Create the associated RBAC rules if your cluster is configured to use RBAC

Before applying the below yaml configs make sure to set the version you want to install

$ export RELEASE=?
$ kubectl apply -f https://github.com/oracle/oci-cloud-controller-manager/releases/download/${RELEASE}/oci-csi-node-rbac.yaml

The CSI Volume Snapshot-Restore feature requires additional CRDs

$ kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/client/config/crd/snapshot.storage.k8s.io_volumesnapshotclasses.yaml
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/client/config/crd/snapshot.storage.k8s.io_volumesnapshotcontents.yaml
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/master/client/config/crd/snapshot.storage.k8s.io_volumesnapshots.yaml

Deploy the csi-controller-driver: It is provided as a deployment and it has five containers -

  1. csi-provisioner external-provisioner
  2. csi-attacher external-attacher
  3. snapshot-controller snapshot-controller
  4. csi-snapshotter external-snapshotter
  5. oci-csi-controller-driver
$ kubectl apply -f https://github.com/oracle/oci-cloud-controller-manager/releases/download/${RELEASE}/oci-csi-controller-driver.yaml

Deploy the node-driver: It is provided as a daemon set and it has two containers -

  1. node-driver-registrar node-driver-registrar
  2. oci-csi-node-driver
$ kubectl apply -f https://github.com/oracle/oci-cloud-controller-manager/releases/download/${RELEASE}/oci-csi-node-driver.yaml

Deploy the csi storage class:

$ kubectl apply -f https://raw.githubusercontent.com/oracle/oci-cloud-controller-manager/master/manifests/container-storage-interface/storage-class.yaml

Lastly, verify that the oci-csi-controller-driver and oci-csi-node-controller is running in your cluster. By default it runs in the 'kube-system' namespace.

$ kubectl -n kube-system get po | grep csi-oci-controller
$ kubectl -n kube-system get po | grep csi-oci-node

Tutorial

Create a claim:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: oci-bv-claim
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: oci-bv
  resources:
    requests:
      storage: 50Gi

Check PVC is in pending state:

$ kubectl describe pvc/oci-bv-claim

Create pod:

apiVersion: v1
kind: Pod
metadata:
  name: app1
spec:
  containers:
    - name: app1
      image: centos
      command: ["/bin/sh"]
      args: ["-c", "while true; do echo $(date -u) >> /data/out.txt; sleep 5; done"]
      volumeMounts:
        - name: persistent-storage
          mountPath: /data
  volumes:
    - name: persistent-storage
      persistentVolumeClaim:
        claimName: oci-bv-claim

Check if PVC is now in bound state:

$ kubectl describe pvc/oci-bv-claim

Troubleshoot

FsGroup policy not propagated from pod security context

If your fsGroup is not being applied on the files in your volume.

Read more about fsGroup Policy.

Ex.

apiVersion: v1
kind: Pod
metadata:
  name: security-context-demo
spec:
  securityContext:
    fsGroup: 2000
  containers:
  - name: sec-ctx-demo
    image: busybox:1.28
    command: [ "sh", "-c", "sleep 1h" ]
    volumeMounts:
    - name: sec-ctx-vol
      mountPath: /data/demo
kubectl exec -it security-context-demo -- sh -c "cd /data/demo && echo hello > testfile"
kubectl exec -it security-context-demo -- sh -c "ls -l /data/demo/testfile"

The output you would expect is that the /data/demo/testfile file has group ID 2000, which is the value of fsGroup

-rw-r--r-- 1 root 2000 6 Jun  6 20:08 testfile

But the same does not reflect on your volume, i.e. the permissions on your files/folders are not what you would expect. Ex:

-rw-r--r-- 1 root root 6 Jun  6 20:08 testfile

Solution:

Create a CSI Driver object with spec: fsGroupPolicy: File. Ex:

apiVersion: storage.k8s.io/v1
kind: CSIDriver
metadata:
  name: blockvolume.csi.oraclecloud.com
spec:
  fsGroupPolicy: File

File - Indicates that the CSI volume driver supports volume ownership and permission change via fsGroup, and Kubernetes may use fsGroup to change permissions and ownership of the volume to match user requested fsGroup in the pod's SecurityPolicy regardless of fstype or access mode.

Calico-node pods fail readiness health checks when oci-fss-utils is installed on node

If you are using Calico CNI, the following error is thrown by the Calico daemonset pods that are scheduled on the node which has oci-fss-utils package installed:

Readiness probe failed: 2023-10-12 08:51:08.172 [INFO][346] confd/health.go 180: Number of node(s) with BGP peering established = 0 calico/node is not ready: BIRD is not ready: BGP not established with 10.0.0.5,10.0.76.88,10.0.66.239

Solution:

Add the following env setting to the calico-node container in the calico-node daemonset manifest.

- name: IP_AUTODETECTION_METHOD
  value: 'skip-interface=v-eth.*'