[occm] Improve route controller reconciling to ensure the cluster's nodes can access each other #2484

jeffyjf · 2023-11-28T10:05:30Z

What this PR does / why we need it:

In order to the nodes of one cluster can access each other, route controller need to check and set three things:

The openstack router's route rules, so that the packets can be forwarded to correct nodes.
The node port's AllowAddressPair, to ensure the node permit the packets that access the node's pods/services pass through.
The openstack security group's rules, so that the nodes that bind the security group permit the packets from other nodes enter into.

The current codes just check router's route rule when controller call ListRoutes and just set Route and AllowAddressPair when controller call CreateRoute. This PR complements all of the other works.

Which issue this PR fixes(if applicable):
fixes #2482

Special notes for reviewers:

This PR lack of unit tests, because of I found occm lack of a mechanism to mock openstack client. Further more the whole occm lack of lots of unit tests due to the reason. I plan to dive deeper into gophercloud in the next several days try to study whether it is possible to mock it. If it is possible I'll commit anohter PR to add the unit tests.

Release note:

NONE

k8s-ci-robot · 2023-11-28T10:05:40Z

Hi @jeffyjf. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

dulek · 2023-11-28T16:50:44Z

Why did this ended up in Routes controller? The issue description feels like this is a general issue in cases CAPO is missing SG rules to allow interconnectivity of Node.

pkg/openstack/routes.go

jeffyjf · 2023-11-29T00:44:15Z

Why did this ended up in Routes controller? The issue description feels like this is a general issue in cases CAPO is missing SG rules to allow interconnectivity of Node.

According to the offical document:

The route controller is responsible for configuring routes in the cloud appropriately so that containers on different nodes in your Kubernetes cluster can communicate with each other.

I think that this is route controller's duty. I user don't activate route controller, the interconnectivity of pods should be ensured by other mechanisms.

jichenjc · 2023-11-29T01:39:20Z

/ok-to-test

jeffyjf · 2023-12-13T02:12:59Z

@dulek @jichenjc @kayrus @zetaab asking for review

kayrus · 2023-12-13T12:01:01Z

The PR needs a rebase. However the #2499 is on the way, and I adopted your getSecurityGroupRules change there as well.

jeffyjf · 2023-12-14T01:14:07Z

/hold

jeffyjf · 2023-12-14T03:15:03Z

/remove hold

jeffyjf · 2023-12-14T03:16:30Z

/remove-hold

pkg/openstack/routes.go

dulek

Some remarks inline. In general I'm not totally against it, but I think the code should introduce a new option that will control the new behavior of creating and updating the SG. I don't think this works for every user here.

pkg/openstack/routes.go

dulek · 2024-01-10T19:09:54Z

pkg/openstack/routes.go

+	origSgs := port.SecurityGroups
+	mc := metrics.NewMetricContext("port", "update")
+	_, err := ports.Update(network, port.ID, ports.UpdateOpts{
+		SecurityGroups: &sgs,
+	}).Extract()
+	if mc.ObserveRequest(err) != nil {


This is prone to a lost update race condition, but it's not a big deal IMO.

pkg/openstack/routes.go

k8s-ci-robot · 2024-01-15T05:29:40Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign kayrus for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

…onnectivity In oreder to ensure the connectivity between different nodes, route controller need to do the following things: 1. Check and set openstack router's route rules, so that the packets can be forwarded to correct nodes. 2. Check and set the node port's AllowAddressPair, to permit the packets from the pods pass through the node's port and leave the node. 3. Check and set openstack security group's rules, so that the nodes that bind the security group permit the packets from other nodes enter into. But, the previous occm codes just check router's route rules, just set router's route and AllowAddressPair, this patch completed the other steps.

jeffyjf · 2024-01-15T07:26:54Z

/retest pull-cloud-provider-openstack-test

k8s-ci-robot · 2024-01-15T07:27:08Z

@jeffyjf: The /retest command does not accept any targets.
The following commands are available to trigger required jobs:

/test openstack-cloud-controller-manager-e2e-test
/test openstack-cloud-controller-manager-ovn-e2e-test
/test openstack-cloud-csi-cinder-e2e-test
/test openstack-cloud-csi-cinder-sanity-test
/test openstack-cloud-csi-manila-e2e-test
/test openstack-cloud-csi-manila-sanity-test
/test openstack-cloud-keystone-authentication-authorization-test
/test pull-cloud-provider-openstack-check
/test pull-cloud-provider-openstack-test

Use /test all to run the following jobs that were automatically triggered:

openstack-cloud-controller-manager-e2e-test
openstack-cloud-controller-manager-ovn-e2e-test
pull-cloud-provider-openstack-check
pull-cloud-provider-openstack-test

In response to this:

/retest pull-cloud-provider-openstack-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

jeffyjf · 2024-01-15T07:28:08Z

/test pull-cloud-provider-openstack-test

k8s-triage-robot · 2024-04-14T07:37:13Z

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

Mark this PR as fresh with /remove-lifecycle stale
Close this PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot · 2024-05-14T07:43:33Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

Mark this PR as fresh with /remove-lifecycle rotten
Close this PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

dulek · 2024-05-23T09:24:55Z

/remove-lifecycle rotten
/lgtm

kayrus

thanks for the PR. Could you take a look at comments I just left?

kayrus · 2024-05-23T09:27:45Z

pkg/openstack/openstack.go

@@ -483,7 +485,7 @@ func (os *OpenStack) Routes() (cloudprovider.Routes, bool) {
 		return nil, false
 	}

-	r, err := NewRoutes(os, network, netExts["extraroute-atomic"], netExts["allowed-address-pairs"])
+	r, err := NewRoutes(os, network, netExts["extraroute-atomic"], netExts["allowed-address-pairs"], os.routeOpts.AutoConfigSecurityGroup)


the os is already passed to the NewRoutes func. There is no need to pass the os.routeOpts.AutoConfigSecurityGroup.

kayrus · 2024-05-23T09:30:36Z

pkg/openstack/routes.go

+	if err != nil {
+		if errors.IsNotFound(err) {
+			mc := metrics.NewMetricContext("security_group", "create")
+			klog.V(2).InfoS("Create node security group", "name", sgName)
+			group, err := groups.Create(r.network, groups.CreateOpts{Name: sgName}).Extract()
+			if mc.ObserveRequest(err) != nil {
+				return err
+			}
+			sgId = group.ID
+		} else {
+			return err
+		}
+	}


Suggested change

if err != nil {

if errors.IsNotFound(err) {

mc := metrics.NewMetricContext("security_group", "create")

klog.V(2).InfoS("Create node security group", "name", sgName)

group, err := groups.Create(r.network, groups.CreateOpts{Name: sgName}).Extract()

if mc.ObserveRequest(err) != nil {

return err

}

sgId = group.ID

} else {

return err

}

}

if err != nil {

if !errors.IsNotFound(err) {

return err

}

mc := metrics.NewMetricContext("security_group", "create")

klog.V(2).InfoS("Create node security group", "name", sgName)

group, err := groups.Create(r.network, groups.CreateOpts{Name: sgName}).Extract()

if mc.ObserveRequest(err) != nil {

return err

}

r.nodeSecurityGroupId = group.ID

return nil

}

kayrus · 2024-05-23T09:32:33Z

pkg/openstack/routes.go

+	if r.allowedAddressPairs {
+		// check whether the related AllowAddressPair is existing
+		for _, addrPair := range port.AllowedAddressPairs {
+			if addrPair.IPAddress == route.DestinationCIDR {
+				return true
+			}
+		}
+		return false
+	}
+	return true


Suggested change

if r.allowedAddressPairs {

// check whether the related AllowAddressPair is existing

for _, addrPair := range port.AllowedAddressPairs {

if addrPair.IPAddress == route.DestinationCIDR {

return true

}

}

return false

}

return true

if !r.allowedAddressPairs {

return true

}

// check whether the related AllowAddressPair is existing

for _, addrPair := range port.AllowedAddressPairs {

if addrPair.IPAddress == route.DestinationCIDR {

return true

}

}

return false

k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Nov 28, 2023

k8s-ci-robot requested review from anguslees and FengyunPan2 November 28, 2023 10:05

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Nov 28, 2023

k8s-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Nov 28, 2023

dulek reviewed Nov 28, 2023

View reviewed changes

pkg/openstack/routes.go Outdated Show resolved Hide resolved

jeffyjf mentioned this pull request Nov 29, 2023

[occm] The pods cannot access each other when across nodes #2482

Closed

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Nov 29, 2023

jeffyjf force-pushed the route-controller-security-group branch from 6b498a6 to f9ab6ef Compare December 6, 2023 03:35

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Dec 6, 2023

jeffyjf force-pushed the route-controller-security-group branch 3 times, most recently from 3c6af74 to d639248 Compare December 13, 2023 01:48

jeffyjf changed the title ~~[occm][WIP] Add SG to node ensure the different node's pods can access each other~~ [occm] Add SG to node ensure the different node's pods can access each other Dec 13, 2023

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 13, 2023

jeffyjf force-pushed the route-controller-security-group branch from d639248 to eaaea60 Compare December 14, 2023 01:12

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 14, 2023

jeffyjf force-pushed the route-controller-security-group branch from eaaea60 to 35f8dca Compare December 14, 2023 03:00

jeffyjf changed the title ~~[occm] Add SG to node ensure the different node's pods can access each other~~ [occm] Improve route controller reconciling to ensure the cluster's nodes can access each other Dec 14, 2023

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 14, 2023

jeffyjf force-pushed the route-controller-security-group branch from 35f8dca to abe4633 Compare December 14, 2023 03:26

kayrus requested changes Dec 14, 2023

View reviewed changes

pkg/openstack/routes.go Show resolved Hide resolved

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 14, 2023

jeffyjf force-pushed the route-controller-security-group branch from abe4633 to 1a93fd9 Compare December 15, 2023 09:27

dulek requested changes Jan 10, 2024

View reviewed changes

jeffyjf force-pushed the route-controller-security-group branch from 1a93fd9 to d1da026 Compare January 15, 2024 05:29

jeffyjf force-pushed the route-controller-security-group branch 4 times, most recently from 87dea84 to 81236c6 Compare January 15, 2024 06:03

jeffyjf force-pushed the route-controller-security-group branch from 81236c6 to ee22e72 Compare January 15, 2024 07:14

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Apr 14, 2024

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels May 14, 2024

k8s-ci-robot removed the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label May 23, 2024

k8s-ci-robot assigned dulek May 23, 2024

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 23, 2024

kayrus requested changes May 23, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[occm] Improve route controller reconciling to ensure the cluster's nodes can access each other #2484

[occm] Improve route controller reconciling to ensure the cluster's nodes can access each other #2484

jeffyjf commented Nov 28, 2023 •

edited

k8s-ci-robot commented Nov 28, 2023

dulek commented Nov 28, 2023

jeffyjf commented Nov 29, 2023

jichenjc commented Nov 29, 2023

jeffyjf commented Dec 13, 2023 •

edited

kayrus commented Dec 13, 2023

jeffyjf commented Dec 14, 2023

jeffyjf commented Dec 14, 2023

jeffyjf commented Dec 14, 2023

dulek left a comment

dulek Jan 10, 2024

k8s-ci-robot commented Jan 15, 2024

jeffyjf commented Jan 15, 2024

k8s-ci-robot commented Jan 15, 2024

jeffyjf commented Jan 15, 2024

k8s-triage-robot commented Apr 14, 2024

k8s-triage-robot commented May 14, 2024

dulek commented May 23, 2024

kayrus left a comment

kayrus May 23, 2024

kayrus May 23, 2024

kayrus May 23, 2024

[occm] Improve route controller reconciling to ensure the cluster's nodes can access each other #2484

Are you sure you want to change the base?

[occm] Improve route controller reconciling to ensure the cluster's nodes can access each other #2484

Conversation

jeffyjf commented Nov 28, 2023 • edited

k8s-ci-robot commented Nov 28, 2023

dulek commented Nov 28, 2023

jeffyjf commented Nov 29, 2023

jichenjc commented Nov 29, 2023

jeffyjf commented Dec 13, 2023 • edited

kayrus commented Dec 13, 2023

jeffyjf commented Dec 14, 2023

jeffyjf commented Dec 14, 2023

jeffyjf commented Dec 14, 2023

dulek left a comment

Choose a reason for hiding this comment

dulek Jan 10, 2024

Choose a reason for hiding this comment

k8s-ci-robot commented Jan 15, 2024

jeffyjf commented Jan 15, 2024

k8s-ci-robot commented Jan 15, 2024

jeffyjf commented Jan 15, 2024

k8s-triage-robot commented Apr 14, 2024

k8s-triage-robot commented May 14, 2024

dulek commented May 23, 2024

kayrus left a comment

Choose a reason for hiding this comment

kayrus May 23, 2024

Choose a reason for hiding this comment

kayrus May 23, 2024

Choose a reason for hiding this comment

kayrus May 23, 2024

Choose a reason for hiding this comment

jeffyjf commented Nov 28, 2023 •

edited

jeffyjf commented Dec 13, 2023 •

edited