Re-provisioning a node, Volumes lost?

[guettli]

We have bare metal servers.

They have two disks.

We plan to use one disk for the operating system, the second for topolvm.

When we re-provision a node the first disk will get erased and a new node will get created (we use cluster API).

If the topolvm daemon set gets installed on the new node, will topolvm find the existing volumes on the second disk?

Is topolvm able to make these volumes available again (with the original content, some Terrabyte of data)?

[satoru-takeuchi]

If the topolvm daemon set gets installed on the new node, will topolvm find the existing volumes on the second disk?

Is topolvm able to make these volumes available again (with the original content, some Terrabyte of data)?

No, topolvm doesn't migrate the PVC and the corresponding data in the second disk to the new node. It's necessary for users to migrate data by themselves. In your case, the following steps are necessary.

1. There is a topolvm PVC, created on top of the second disk, in the node 0.
2. node 1 is provisioned.
3. Create another topolvm PVC in node 1.
4. Copy the data in the first PVC to the second one.

[guettli]

Wait a second. For this question I am looking at one bare-metal server. The server has two disks. An upgrade erases the data on the first disk (Linux OS and Kubernetes services like kubelet), but the second data disk is untouched. After this upgrade it looks like the old node is gone, and a new node came up. But it is still the same hardware.

We would like to avoid that the PVC gets lost, when the name of the node changes.

Copying the data would take too long.

I see two solutions:

• We change the code which creates the new node, so that the node name does not change. I hope topolvm would re-connect to the data on the second disk in this case
• We provide a patch to topolvm, so that constant identifier (not the node name) gets used in the PV.

What do you think?

[satoru-takeuchi]

After this upgrade it looks like the old node is gone, and a new node came up. But it is still the same hardware.

I misunderstood your situation. Please wait for a while.

[satoru-takeuchi]

I guess you are be able to do what you want if you do the following two things,

• Change your code to create a new node with the same name as old one
• Specify --skip-node-finalize

However, topolvm only tests in the in-place node upgrade environment (node resource is not re-created during K8s upgrade). Please verify whether this solution works carefully by yourself.

[guettli]

@satoru-takeuchi thank you very much for your answer.

Would you accept a PR which makes topolvm build the name of the PVC differently?

For example the string ProviderID which is part of the official NodeSpec could be used to construct a PV name which does not change, if a bare-metal machine gets deleted and created again.

Here are some examples of that ProviderID string:

aws:///us-west-2/i-1234567890abcdef0
gce://my-gcp-project/us-central1-a/my-instance-id
azure:///subscriptions/12345678-1234-1234-1234-123456789abc/resourceGroups/myResourceGroup/providers/Microsoft.Compute/virtualMachines/myVMName
vsphere://42309a5a-fc49-4953-8077-05184b6d8915
baremetal://node-01

The string ProviderID allows more characters than a PV-name, but we could use a hash, so that the PV name stays constant and is a valid PV-name.

Of course this would be not active by default. Only if the user provides --pv-from-providerid, then our PR could construct the PV-name via a hash from the ProviderID.

What are your thoughts about that?

[satoru-takeuchi]

Could you tell me why such a change is necessary? In my understanding, current PVC naming convention is enough to accomplish what you want to do.

[batistein]

@satoru-takeuchi The background is that the Kubernetes lifecycle system Cluster API creates new "servers" when the cluster is upgraded. The baremetal server stores the data on a separate disk, so only the OS disk is re-provisioned. Cluster API creates a new node name. After a rollout, we have the same physical server with the data on the separate disk, but with a different new name. Now we want to reassign the PV to this node. Basically, we want to "fix" the relationship. The ProviderID is always the same because it depends on the physical node.

I hope this helps to better understand the use case we're working on.

FYI: @guettli and me work on the same company

[satoru-takeuchi]

Would you accept a PR which makes topolvm build the name of the PVC differently?

I misunderstood what @guettli said because the above description say PVC names but actually you want to set PV names(and another paragraphs is about "PV name" correctly).

Now we want to reassign the PV to this node. Basically, we want to "fix" the relationship. The ProviderID is always the same because it depends on the physical node.

PV name is not assigned by topolvm, but by csi sidecars. So this kind of change should be done in csi sidecars. In addition, if my understand correctly, both PVC and PV won't be deleted on node deletion, you can use the same PVC and PVC as expected if the new node name is the same as the old one.

[guettli]

@satoru-takeuchi thank you very much for your answers. We updated our environment, so that node-names don't change if they get deleted and re-created via cluster-api. Our (Syself) cluster api provider for Hetzner now supports constant hostnames. See Release notes for beta.34