Error attaching/detaching volumes after the upgrade from 1.6 to 2.3.2

[d3473r]

Hello,

i did an upgrade from kubernetes 1.20 to 1.24 and the respective upgrade of the csi driver from 1.6.0 to 2.3.2 following the instructions from here: https://github.com/hetznercloud/csi-driver/blob/main/docs/kubernetes/README.md#from-v1-to-v2

The upgrade went smooth, all commands executed successfully, the fix-volumeattachments.sh script patched all volume attachments.

But if i scale down a deployment to 0 with an attached volume, the volume is no longer detached from the node.
I can detach it manually via the console.hetzner.cloud ui.
The same error is happening when starting a deployment, the volume is not getting attached to the node and the pod is failing to start with: MountVolume.SetUp failed for volume "pvc-xxx" : rpc error: code = Internal desc = failed to publish volume: exit status 1 mke2fs 1.46.6 (1-Feb-2023) /dev/disk/by-id/scsi-0HC_Volume_123 is mounted; will not make a filesystem here!

This workaround fixes the error: #278 (comment)
But only once, if the pod is stopped and started again the same error is occuring

Did i miss some migrations steps to upgrade from 1.6.0 to 2.3.2?

Regards

[Privatecoder]

same here

[toabi]

I just upgraded my k3s cluster from v1.25.9 to v1.26.7 and during rotation of storage nodes (running rook ceph) the OSDs couldn't launch because the PVC was not unmounted properly (even after 10+ minutes).

The workaround with deleting the volumeattachment seems to fix this issue…

Update: Rotation of other nodes somehow worked without manual intervention, so this is not easy to reproduce.

[github-actions]

This issue has been marked as stale because it has not had recent activity. The bot will close the issue if no further action occurs.

[github-actions]

This issue has been marked as stale because it has not had recent activity. The bot will close the issue if no further action occurs.