[cinder-csi-plugin] Multi region/clouds support for controllerServer

[MatthieuFin]

Affected binaries:

• cinder-csi-plugin

What this PR does / why we need it:
Enable multi region / multi clouds support on cinder-csi-plugin controllerServer.

Which issue this PR fixes(if applicable):
fixes #2035

Special notes for reviewers:
I have a kubernetes cluster stretch on multiples openstack clusters (3 OVH and 1 onPremise linked by a private dark fiber).
That why my approche is more "multi clouds" than only "multi region"

I don't touch nodeServer behavior (I simply deploy a Daemonset with a nodeSelector on a label whitch select nodes based on their hypervisor, and mount a dedicated secret with openstack creds associated)

The purpose is on controllerServer which handle all kube requests to manage pvc, he had be able to CRUD volumes on all managed openstack cluster.

I propose to use gcfg subsection feature to be abble to handle multiple "sections" Global in config files. This permit to be backaward compatible with configfiles syntax.

I choose to use StorageClass parameters to deal with cloud selection (i add an optionnal field cloud which containt config Global subsection name). In this way when a createVolumeRequest income controllerServer could identify OSInstance to used based on match between SC field parametes.cloud and his config Global subsections.

Issue
This PR is currently a MVP, which permit to create volumes and attach them to correct VM / OpenStack cluster, but i have a bit issue to troubleshoot, indeed all csi spec message don't contains volume_context or parameters, especially for DeleteVolumeRequest which only contain volumeID.

I have 2 ideas to troubleshoot this issue:

• query each managed OS clouds to find one who had a volume with searched ID (That's linear complexity)
• implement a kubernetes client and as kubernetes API to find (bad idea, I just realized that volume_id is openstack id not pv name)
• maybe use secrets field as it seems suggested in external-provisioner discussion

Release note:

feat add multi regions/clouds support.

[k8s-ci-robot]

Welcome @MatthieuFin!

It looks like this is your first PR to kubernetes/cloud-provider-openstack 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/cloud-provider-openstack has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[k8s-ci-robot]

Hi @MatthieuFin. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign jichenjc for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[MatthieuFin]

As mentioned here and after some local tests it seems to be the proper implementation way.

I propose to avoid usage of storageClass parameter.cloud and volume context to propagate config cloud name between gRPC calls, and instead use a secret with a key cloud which contain the Global config subsection name, which permit to controller to retrieve proper credentials from his configuration.

[dulek]

/ok-to-test

Sorry, I won't have time to take a closer look before my vacations, will be back in 2 weeks. @kayrus might be the proper person to check this PR out.

[dulek]

Looks like the tests have to be fixed here.

[MatthieuFin]

/retest

[MatthieuFin]

/retest

[MatthieuFin]

Hi,
After a few days of use on my cluster, I encountered an issue, regard allowedTopologies set on StorageClass (first of all don't forget to enable feature gate Topology add argument --feature-gates Topology=true on container csi-provisioner) which permit to push allowedTopologies constraints on pv .spec.nodeAffinity and keep affinity on pod reschedule.

My issue concern topology keys, currently only available is topology.cinder.csi.openstack.org/zone and obviously my provider OVH use same zone name (nova) on each of their openstack cluster, so I need to add possibility to manage another key to differentiate them.

[sebastienmusso]

Hi, After a few days of use on my cluster, I encountered an issue, regard allowedTopologies set on StorageClass (first of all don't forget to enable feature gate Topology add argument --feature-gates Topology=true on container csi-provisioner) which permit to push allowedTopologies constraints on pv .spec.nodeAffinity and keep affinity on pod reschedule.

My issue concern topology keys, currently only available is topology.cinder.csi.openstack.org/zone and obviously my provider OVH use same zone name (nova) on each of their openstack cluster, so I need to add possibility to manage another key to differentiate them.

we have exactly the same problem (several openstack with same availability zone name)

I'm interesseted by this feature let me know if you need some test feedback

regards

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: MatthieuFin (102fd87, 69e1f5b, 5861f43, 8b36da6, b46e3d8, 10b8667)

[MatthieuFin]

Thanks to @sebastienmusso, @fllaap and @Krast76 who offered to help write documentation.

@dulek, @jichenjc, @kayrus, @mdbooth how can we advance this PR ?

[sergelogvinov]

Thank you for this PR, very interesting idea. 👍

So the complexity of this PR is that node-pluging needs openstack api credentials for ephemeral storage which is deprecated. Perhaps we should ask about the removal process for it?

The killer feature is to use one name of storageClass in stetefulset deployment. Base on nodAffinity/podAntiAffinity kubernetes scheduler spread the pods across the regions/zones.

I quit to using allowedTopologies in storageClass they are immutable. It is very hard to maintain. It's better to use nodeSelector/nodeAffinity instead; you can change these parameters during the lifetime.

To help kubernetes scheduler to choose the best region/zone (if not set) we need to expose free disk capacity.

[sergelogvinov]

I've added a capacity capabilities - #2597

[sebastienmusso]

@dulek, @jichenjc, @kayrus, @mdbooth how can we advance this PR ?
And if this capabilities is not available within openstack cinder-csi-plugin, how can we handle
the stateful storage with cinder in a kubernetes cluster spread over multiple openstack cluster ?

[jichenjc]

And if this capabilities is not available within openstack cinder-csi-plugin, how can we handle
the stateful storage with cinder in a kubernetes cluster spread over multiple openstack cluster ?

I don't know how other cloud are handling this kind of cross region (e.g AWS cloud provider handle this?) maybe someone can help comment

and I knew openstack itself handling multiple cloud (region) is that not good
especialyly I don't think we knew the free size of the storage cinder backend IIRC
and neutron might give us additional issue as cross region usually means no connection (not sure the impact to our network related things implementation)

so overall I think we may easily focus on compute (VM) provisioning and claim others as gap/follow up?

[MatthieuFin]

Hi @jichenjc ,
Other cloud provider (I'll take example of GCP) offer a unique storage class which is able to create pods across multiple "regions".
To do that they implements topology spec in their csi implementation with help of PluginCapability VOLUME_ACCESSIBILITY_CONSTRAINTS

Technically our openstack implementation doesn't support PluginCapability VOLUME_ACCESSIBILITY_CONSTRAINTS so GRPc calls createVolume doesn't have field accessibility_requirements fill.

We can implement support of PluginCapability VOLUME_ACCESSIBILITY_CONSTRAINTS to get informations in field accessibility_requirements and retreive information from scheduler or git topology choose by csi plugin to scheduler (depending of volumeBindingMode). anyway this approach doesn't help us here.

Indeed, GCP or AWS could implement that because all regions are uniformly managed by cloud API, so when you ask for a volume_id in DeleteVolumeRequest for example (same for others calls) to your cloud API, you have got response never mind in which region is the volume.

In our case, manage multi openstack region is the same approach than manage multiple openstack clusters. So I have many n clouds api to n regions, where in GCP you have n region for only 1 cloud API.

The "GCP/AWS like" behavior is probably to manage "regions" with AZ zone notion in openstack. But that is not my use case, I need to be able to support multiple openstack clusters not multiple AZ in one openstack cluster. And I don't know public cloud provider based on openstack which properly implement openstack availability zones...

Moreover if we wanna handle multi clouds (openstack regions) with the help of PluginCapability VOLUME_ACCESSIBILITY_CONSTRAINTS we have 2 possibility:

• modify csi spec to add some additional data which permit to trace cloud volume owner in GRPc calls (I don't think that will be possible/accepted)
• Implement an "openstack api like" software which expose a unique API to csi plugin and manage multiple clouds in backend and maintain an internal database to know which cloud has each volume_id, and ovoid possible collision in volume_id between 2 differents clouds.

Personally these 2 solution really to complex to implement and maintain.

This PR permit to do it easier way to understand and maintain imo.

I have a storageClass per regions/clouds. and if i wanna deploy 1 sts cross 3 regions I create 3 pvc one per regions (each with a different SC: data-0, data-1, data-2) and I use sts:

spec.volumeClaimTemplates:
  - apiVersion: v1
    kind: 
      name: data

In that way I'm able to dispatch pods from same sts cross different sc and different openstack cluster.

If you wanna a "multi-region" implementation like GCP or AWS

• Currently I couldn't implement it because I haven't openstack cluster environment with multi AZ.
• You miss a feature that GCP/AWS doesn't offer which is storage support cross multi openstack cluster.

From my limited experience as a cloud consumer, openstack providers offer openstacks with multiple regions and not with multiple AZs. It's sad I know, but it seems to be the public offerings.

Concerning network cross region indeed it shouldn't have to be managed by neutron, personally i managed it by myself out of openstack with some BGP routing and physical interconnections L2 (dark fiber) cross my DCs and ipsec tunnels for dev/poc links cross different DCs not physically linked (I shared my firsts reflections about it here but that's out of this PR scope). I know that @sebastienmusso run k8s clusters spread on multiples openstack clusters too, I don't know how you handle this point ?

Implement capacity capabilities as proposed by @sergelogvinov could probably help in multi AZ environment but in this PR context it should probably not have impact.