can't deploy pod with pvc

[simonebenati]

Hello,

I have deployed correctly every topolvm component. Everything is up and running but when I try to deploy a pod with a pv.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc
spec:
  storageClassName: topolvm
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 15Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
        - name: my-container
          image: ubuntu:latest
          ports:
            - containerPort: 80
          volumeMounts:
            - name: my-volume
              mountPath: /data
      volumes:
        - name: my-volume
          persistentVolumeClaim:
            claimName: my-pvc
      tolerations:
      - key: "database"
        operator: "Equal"
        value: "true"
        effect: "NoSchedule"

The cluster is not able to provision a pod because:

default-scheduler 0/7 nodes are available: 3 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }, 4 Insufficient topolvm.io/capacity. preemption: 0/7 nodes are available: 3 Preemption is not helpful for scheduling, 4 No preemption victims found for incoming pod.

Is anyone able to provide help on this matter? I don't understand what Insufficient topolvm.io/capacity is.. I checked all the documentation but couldn't find much on it

[toshipp]

This means no VGs where the PV will be created has more than 15GB of free space.
Could you check VG free space on your nodes using vgs command?

[simonebenati]

Hello @toshipp

Thank you for your answer,
Sorry it took me a while to answer but I had to recreate the cluster and redeploy the test-pod.
I still have the issue:

And here's what you asked me to check:

vgs vg-data01
  VG        #PV #LV #SN Attr   VSize    VFree
  vg-data01   1   0   0 wz--n- <116.50g <116.50g

Furthermore I even tried to manually create a lv under vg-data01 and it was created successfully:

lvcreate -l 5%FREE -n lv-test vg-data01
  Logical volume "lv-test" created.
root@worker04-cortex:~# lsblk
NAME                  MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
loop0                   7:0    0  67.8M  1 loop /snap/lxd/22753
loop1                   7:1    0    62M  1 loop /snap/core20/1611
loop2                   7:2    0    47M  1 loop /snap/snapd/16292
loop100                 7:100  0 116.5G  0 loop
└─vg--data01-lv--test 253:0    0   5.8G  0 lvm
sr0                    11:0    1   508K  0 rom
vda                   252:0    0   240G  0 disk
├─vda1                252:1    0 239.9G  0 part /
├─vda14               252:14   0     4M  0 part
└─vda15               252:15   0   106M  0 part /boot/efi
vdb                   252:16   0   100G  0 disk

Hence:

  VG        #PV #LV #SN Attr   VSize    VFree
  vg-data01   1   1   0 wz--n- <116.50g 110.67g

I also noticed that in my worker node there are two annotations:

capacity.topolvm.io/00default: 108095602688
capacity.topolvm.io/ssd: 108095602688

But the pod is referring to this other annotation:
4 Insufficient topolvm.io/capacity.
And I therefore tried to manually add the latter annotation to my worker node but without any luck as it is still not scheduled.

[toshipp]

@simonebenati
Thank you for your information.
To solve your problem, could you show me your topolvm install procedure such as the using version, helm values.yaml, or anything else?

[simonebenati]

in rke2, in machine_global_config file we added the following:

  kube-scheduler-arg:
    - config=/etc/kubernetes/scheduler/scheduler-config.yaml

Then for each controller

create folder: /etc/kubernetes/scheduler
download file: https://raw.githubusercontent.com/topolvm/topolvm/<$TOPOLVM_VERSION>/deploy/scheduler-config/scheduler-config.yaml in the folder /etc/kubernetes/scheduler

Then we modify the just downloaded file from just

kubeconfig: /etc/kubernetes/scheduler.conf in:
kubeconfig: /var/lib/rancher/rke2/server/cred/scheduler.kubeconfig

[toshipp]

Thank you.
TopoLVM has many options and kubernetes has too.
Please provide your information as much as possible.
What topolvm and kubernetes versions do you use? Could you provide your helm values.yaml?
Logs from topolvm components are also helpful.

[simonebenati]

latest topolvm version, and kubernetes v1.25.10+rke2r1

Default helm values

[simonebenati]

Sorry but i was on vacation for the past two weeks, this issue is still in place!
Thank you for your help @toshipp

[toshipp]

@simonebenati
I doubt you failed to specify the StorageClass name in the PVC.
The default StorageClass name is topolvm-provisioner, but you specified topolvm.
Could you check the StorageClass name by using kubectl get sc and your PVC?

[simonebenati]

My SC name is "topolvm" we decided to quit this approach but continue with Storage Capacity Tracking always found in deprecated rancher guide.

If possible please add documentation on how to make topolvm work on rke2.. with scheduler.. everything changed and it's quite complicated

[toshipp]

We have no plan to update the rancher guide because we don't have enough knowledge of it.