Design for dynamic provisioning of both LVs and raw devices

[travisn]

Current State

TopoLVM today provisions LVs from VGs that have been configured outside of TopoLVM. The configuration to add a device to the VG must be made in advance of using TopoLVM. Adding a device to the VG either requires tools to be run on the host, or a tool such as the TopoLVM operator to provide that configuration.

Design Question

What if the TopoLVM driver could expand its scope for working with local storage to include raw devices as well? Some scenarios prefer to have a whole dedicated disk, instead of using an LV. For example, a Ceph OSD prefers a raw disk, while most applications may work perfectly fine with an LV.

If the support for raw devices is added to topolvm, the big advantage is that the decision to provision raw devices or LVs can be delayed until an application requests the storage. No advance decision about raw devices and VGs would be needed.

Without adding support for raw devices to topolvm, this would imply:

• A separate solution would be needed for raw devices
• For clusters where both solutions are needed (raw devices and dynamic LVs), a solution would be needed to assign which devices are for raw usage, and which are for LVs. This would require a static division of the devices where the admin would need to decide how many devices for each type.

Design Suggestion

We are imagining a topolvm design such as the following:

1. The admin could create two storage classes, one for each type.
   • A storage class for dynamic provisioned LVs
   • A storage class for dynamic provisioned raw devices

Each storage class would specify a list of raw devices to be considered for provisioning. The list could be based on a device name filter, or device type (ssd/hdd), or some other selection criteria.

2. When a PVC is created and the PV is provisioned, TopoLVM would do the following:

• If the request is for a raw device, bind the next available raw device to the PV.
• If the request is for an LV:
  • If there is existing space in an existing VG, create a new LV from that VG
  • If there is not enough space in an existing VG, add the next available raw device to the VG, then create a new LV

Conclusion

This would be a very big design change for topolvm to support raw devices. We would like to confirm if this could fit into the roadmap, if we were to contribute the feature.

Do you see any blockers for implementing this design change? Thanks!

[satoru-takeuchi]

My team originally tended to share a VG with a K8s cluster and a local host. Just expanding lvmd will be very intrusive change. To minimize the effect of this feature, it's better to add an additional layer to TopoLVM.

The current brief software stack is as follows.

---

topolvm csi driver: provide an LV from the VG managed by lvmd

^
|
v

lvmd: manage an VG created in host

The new software stack might be as follows.

---

topolvm

new volume provisioner layer

^ ^
| | * We can specify the plugin (listed below) in storageClass and it's lvmd by default.
v v

---

lvmd new daemon: manage raw devices specified in storageClass

---

[travisn]

Understood, this would be an intrusive architecture change. I assume the following implications:

• The existing scenarios would continue to work with topolvm-node connecting directly to lvmd
• The new scenarios would specify a new provider (perhaps call it "rawd" for raw devices). The new component would configure raw devices or connect to lvmd depending on the request.

A modification of the design diagram for the node might look like the attached image. Perhaps "rawd" is just part of topolvm-node, or perhaps it is a new container, not sure yet. We can work through the details later.

@liewegas Other thoughts on the scenarios to consider?

[toshipp]

This project has a tight scope that topolvm creates PVs backed by LV from a given VG. This makes topolvm simple and composable.
Introducing a feature to treat raw devices extends the scope, so it is needed to consider carefully.
I think it is better to separate the feature from topolvm because users may need only one of these.
BTW, I agree on the difficulty to share raw devices for raw PV and topolvm.
Here is my just idea, not thinking deeply, but for further discussion.
How do you think about introducing a PV/PVC-like layer to lvmd?
(I'm going to call it VG/VGC.)
New lvmd will request more space for a VG by VGC if the VG is exhausted. And another component extends the VG according to VGC.
This mechanism allows other components to manipulate raw devices.

[travisn]

There are really two scenarios that we hope to address with a single csi driver but with a separate storage class:

1. Provision volumes with an LV. If out of space, automatically add spare raw device(s) to the VGs so more space is available.
2. Provision volumes from raw disks.
   If a single provider can dynamically provide both these scenarios, there is no need to decide in advance how to allocate the devices between raw and VGs.

I'm not sure how to implement it yet in the details of topolvm components. Are you saying that with a PV/PVC-like layer in lvmd, then lvmd would decide when to add more raw devices? This means we would need a new csi driver for provisioning raw devices, right? And we would still need a common component that discovers the available raw devices?

[toshipp]

@travisn

Are you saying that with a PV/PVC-like layer in lvmd, then lvmd would decide when to add more raw devices? This means we would need a new csi driver for provisioning raw devices, right?

Yes.

And we would still need a common component that discovers the available raw devices?

I didn't consider how to allocate raw devices between a csi plugin for raw devices and topolvm, we would need to add something to do that.
So, yes, as you say.

[toshipp]

@jmolmo
Very interesting.

I am not familiar with topolvm-operator, but if it is needed to modify topolvm for VG requests, I'll be willing to accommodate that.

[travisn]

To clarify my earlier comment, I don't believe it is a requirement to have a single driver to satisfy both scenarios. I believe the requirements are:

1. TopoLVM needs the ability to dynamically add more raw disk(s) to a VG when the VG does not have sufficient space to satisfy the PVC request.
2. TopoLVM and a raw device provisioner would need to be based on the same device discovery implementation, otherwise they may conflict with provisioning raw devices.

For the first item, it is an interesting thought that lvmd could request more storage from the topolvm operator. I'm not sure how well this will fit together with the following considerations:

• If there are no more devices available, lvmd should fail the request immediately instead of trying to wait for a VG expansion request to be fulfilled by the topolvm operator.
• If lvmd is already aware of the available raw devices from discovery, it seems simple to just add it to the VG instead of having the topolvm operator in the middle of the request.

[toshipp]

Understood.

If there are no more devices available, lvmd should fail the request immediately instead of trying to wait for a VG expansion request to be fulfilled by the topolvm operator.

Currently, to avoid such a situation, we use capacity tracking or topolvm-scheduler to filter out empty nodes. But this scheme might be impossible because topolvm can not know how much disk remains in dynamic allocation nature.
If the scheduler does not help, we could use the response for a VG expansion to notify disk is empty. In the k8s world, this is done via the status field.

If lvmd is already aware of the available raw devices from discovery, it seems simple to just add it to the VG instead of having the topolvm operator in the middle of the request.

Unfortunately, lvmd does not have discovery logic, it only uses VGs specified by a config file.

[travisn]

Thanks for the clarifications on how big a change this would be to the architecture.

[little-guy-lxr]

topolvm operator now support dynamic add device to vg by edit TopolvmCluster config. if user edit topolvmcluster CR to expand vg, topolvm-operator will start a job that will add device to vg.

[jmolmo]

The idea will be more like to include in the topolvm cluster from the very beginning a set of devices, filters and rules to determine available devices for use i storage context. The operator would use these devices to make and automatic expansion of the VGs if more space is requested, without any further user intervention. If there is no need for space the devices would not be used.

[liewegas]

One thought: if the request for/claim of raw storage is done by topolvm then it can happen on demand, i.e., when a PVC comes along that can't be satisfied from the existing space in the VG. It would also keep the operator simple and out of the capacity management game.