[occm] feat : add load balancer listener tag using service annotation #2439

yorubad-dev · 2023-10-18T14:40:57Z

openstack-cloud-controller-manager (occm)

/kind feat

What this PR does / why we need it:
this PR solves the add annotation for custom octavia listener tags problem

this is currently a draft PR that needs review, also i am a bit confuse on where to plug the method 😢

Which issue this PR fixes(if applicable):
fixes #2327

k8s-ci-robot · 2023-10-18T14:41:00Z

Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our release note process to remove it.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot · 2023-10-18T14:41:07Z

Hi @KingDaemonX. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

yorubad-dev · 2023-10-18T14:42:06Z

@pierreprinetti can you take a look at this ??

kayrus

thanks for the PR. Is it still WIP? If so, please add documentation and tests and mark this PR as WIP. See also some major findings

pkg/openstack/loadbalancer.go

jichenjc · 2023-10-19T03:06:05Z

/ok-to-test

pkg/openstack/loadbalancer.go

pierreprinetti · 2023-10-19T09:12:30Z

Also: please don't forget to add tests! At a minimum, a unit test to exercise tag splitting (with various configurations of spaces in it) would probably be a good idea

yorubad-dev · 2023-10-19T12:50:01Z

thanks for the PR. Is it still WIP? If so, please add documentation and tests and mark this PR as WIP. See also some major findings

do you mean the comments on the issue ??

pkg/openstack/loadbalancer.go

k8s-ci-robot · 2023-10-23T19:21:41Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign zetaab for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

yorubad-dev · 2023-10-23T19:27:37Z

@pierreprinetti @kayrus please re-review

i am working currently working on the test case, hence the WIP

dulek

This still needs a bunch of improvements.

docs/openstack-cloud-controller-manager/expose-applications-using-loadbalancer-type-service.md

pkg/openstack/loadbalancer.go

dulek · 2023-11-08T16:01:51Z

docs/openstack-cloud-controller-manager/expose-applications-using-loadbalancer-type-service.md

+- `loadbalancer.openstack.org/custom-tags`
+
+  Allows customable loadbalancer tag configurable added during initial stage of loadbalancer creation.
+  Tags are abitrary strings that can me added to loadbalancer and using this annotation allows adding of one or more custom tag to the `LoadBalancer`, `Listener` and `Pool`


I specifically asked to not do that, I do not think we should end up with 10 more annotations supported for each of the resource we create in CPO.

pkg/openstack/loadbalancer.go

jeffyjf · 2023-11-09T11:14:17Z

Hi @KingDaemonX

If you addressed something, you can resolve the related comments.

dulek

I realized we're missing tagging of floating IPs here. You should be able to add it here:

cloud-provider-openstack/pkg/openstack/loadbalancer.go

Lines 918 to 927 in 03c525d

    
           func (lbaas *LbaasV2) createFloatingIP(msg string, floatIPOpts floatingips.CreateOpts) (*floatingips.FloatingIP, error) { 
        
           	klog.V(4).Infof("%s floating ip with opts %+v", msg, floatIPOpts) 
        
           	mc := metrics.NewMetricContext("floating_ip", "create") 
        
           	floatIP, err := floatingips.Create(lbaas.network, floatIPOpts).Extract() 
        
           	err = PreserveGopherError(err) 
        
           	if mc.ObserveRequest(err) != nil { 
        
           		return floatIP, fmt.Errorf("error creating LB floatingip: %s", err) 
        
           	} 
        
           	return floatIP, err 
        
           }

Please note it's a Neutron resource again, so the tagging is done the same way as in case of the security groups (the string should be "floatingips", I check that in the API reference).

I'm trying to build this and run a bit of tests, might get back with more comments.

pkg/openstack/loadbalancer.go

dulek · 2023-11-09T16:44:29Z

I'm trying to build this and run a bit of tests, might get back with more comments.

Okay, nevermind, after changing "securitygroups" to "security-groups it just works.

dulek · 2023-11-09T16:48:58Z

Ah, one more thing, we should add these Neutron tag operations to metrics. Here's how you do it when tagging: https://github.com/kubernetes/cloud-provider-openstack/blob/release-1.27/pkg/openstack/loadbalancer.go#L769-L774

Let's name the context "security_group_tag", "replace" and "floating_ip_tag", "replace".

yorubad-dev · 2023-11-09T19:33:37Z

Unrelated change, please remove.

yeah i get you
i will do that

yorubad-dev · 2023-11-13T14:52:30Z

i have made all the required changes to the PR @dulek

dulek

Seems like you've used security-group instead of security-groups. I'm fairly sure only the latter works, but please prove me wrong if you've tested it and it works.

Also seems like my remarks from this comment about metrics are not implemented.

pkg/openstack/loadbalancer.go

yorubad-dev · 2023-11-13T19:37:42Z

Seems like you've used security-group instead of security-groups. I'm fairly sure only the latter works, but please prove me wrong if you've tested it and it works.

Also seems like my remarks from this comment about metrics are not implemented.

i totally miss the remark on metric until now that you pointed it again
i will fix them now

kayrus · 2023-11-14T10:33:16Z

Hm, I also don't like an idea that we need to tag all the resources at once. Neutron is modular and there is a possibility that it doesn't have tags module enabled, while octavia can have this module. If tags in neutron are not enabled, the OCCM will fail the reconciliation with an error.
Though, cascade tagging of the octavia resources seems fine with me.

Also security groups and especially FIPs can have their own set of tags, e.g. FIP can be preallocated in advance and a special tag can be assigned on it. Once OCCM starts to manage this FIP, it will remove the tags assigned manually. This should not happen.

k8s-ci-robot · 2023-11-15T03:05:57Z

@KingDaemonX: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
openstack-cloud-csi-cinder-sanity-test	`d726570`	link	true	`/test openstack-cloud-csi-cinder-sanity-test`
openstack-cloud-csi-manila-e2e-test	`d726570`	link	true	`/test openstack-cloud-csi-manila-e2e-test`
openstack-cloud-csi-cinder-e2e-test	`d726570`	link	true	`/test openstack-cloud-csi-cinder-e2e-test`
openstack-cloud-csi-manila-sanity-test	`d726570`	link	true	`/test openstack-cloud-csi-manila-sanity-test`
openstack-cloud-keystone-authentication-authorization-test	`d726570`	link	true	`/test openstack-cloud-keystone-authentication-authorization-test`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

…ing service annotation

dulek

This version adds the metrics in a wrong place and using wrong IDs. I've pointed to that codeplace as an example where we did some tagging and metrics for another purpose, but the actual tagging should happen just after the resource is created.

dulek · 2023-11-20T17:38:14Z

pkg/openstack/loadbalancer.go

+			mc = metrics.NewMetricContext("floating_ip_tag", "replace")
+			_, err = neutrontags.ReplaceAll(network, "floatingips", port.ID, neutrontags.ReplaceAllOpts{Tags: tags}).Extract()
+			if mc.ObserveRequest(err) != nil {
+				return fmt.Errorf("failed to add tag %s to port %s of floating_ips: %v", tags, port.ID, err)
+			}


This shouldn't be here, this function doesn't have anything to do with floating IPs.

Also this won't ever work, you specify port.ID which will not match a floating IP.

dulek · 2023-11-20T17:39:32Z

pkg/openstack/loadbalancer.go

+			mc = metrics.NewMetricContext("security_group_tag", "replace")
+			_, err := neutrontags.ReplaceAll(network, "security_groups", port.ID, neutrontags.ReplaceAllOpts{Tags: tags}).Extract()
+			if mc.ObserveRequest(err) != nil {
+				return fmt.Errorf("failed to add tag %s to port %s: %v", tags, port.ID, err)
+			}


I think this is wrong placement, this method applies the security group to a certain port. For now we should limit the SG tagging to the place we're creating it.

Also this won't ever work, you specify port.ID which will not match a security group.

dulek · 2023-11-20T17:40:22Z

pkg/openstack/loadbalancer.go

+	if _, err := neutrontags.ReplaceAll(lbaas.network, "floatingips", floatIP.ID, neutrontags.ReplaceAllOpts{Tags: tags}).Extract(); err != nil {
+		return nil, fmt.Errorf("failed to add custom tags %s to floatingIPs %s with a projectID (%s)", tags, floatIP.ID, floatIP.ProjectID)
+	}


This is missing the metrics addition. This is where you should add mc = metrics.NewMetricContext("floating_ip_tag", "replace") and all the other stuff.

I also realized we should only call this code when len(tags) > 0.

dulek · 2023-11-20T17:41:29Z

pkg/openstack/loadbalancer.go

+		if _, err := neutrontags.ReplaceAll(lbaas.network, "security-groups", lbSecGroupID, neutrontags.ReplaceAllOpts{Tags: tags}).Extract(); err != nil {
+			return fmt.Errorf("failed to add custom tags %s to security group %s (%s)", tags, lbSecGroupID, lbSecGroupName)
+		}


This is missing the mc = metrics.NewMetricContext("security_group_tag", "replace") and then the subsequent ObserveRequest().

Same here should only do tagging when len(tags) > 0.

dulek · 2023-11-20T17:47:50Z

pkg/openstack/loadbalancer.go

+	if _, err := neutrontags.ReplaceAll(lbaas.network, "floatingips", floatIP.ID, neutrontags.ReplaceAllOpts{Tags: tags}).Extract(); err != nil {
+		return nil, fmt.Errorf("failed to add custom tags %s to floatingIPs %s with a projectID (%s)", tags, floatIP.ID, floatIP.ProjectID)
+	}


I also realized we should only call this code when len(tags) > 0.

dulek · 2023-11-20T17:48:09Z

pkg/openstack/loadbalancer.go

+		if _, err := neutrontags.ReplaceAll(lbaas.network, "security-groups", lbSecGroupID, neutrontags.ReplaceAllOpts{Tags: tags}).Extract(); err != nil {
+			return fmt.Errorf("failed to add custom tags %s to security group %s (%s)", tags, lbSecGroupID, lbSecGroupName)
+		}


Same here should only do tagging when len(tags) > 0.

dulek · 2023-11-20T17:57:27Z

Hm, I also don't like an idea that we need to tag all the resources at once. Neutron is modular and there is a possibility that it doesn't have tags module enabled, while octavia can have this module. If tags in neutron are not enabled, the OCCM will fail the reconciliation with an error. Though, cascade tagging of the octavia resources seems fine with me.

True, we can do that and make sure that tagging will not happen when Neutron doesn't have the extension. Octavia is not modular, so it's just a matter of API version, but we should check for tags extension in Neutron case too.

@KingDaemonX - we need to add a function checking this and only attempt to tag FIPs and SGs when it exists. This is an example of how to do it:

cloud-provider-openstack/pkg/openstack/openstack.go

Lines 475 to 484 in fdba36b

    
           netExts, err := openstackutil.GetNetworkExtensions(network) 
        
           if err != nil { 
        
           	klog.Warningf("Failed to list neutron extensions: %v", err) 
        
           	return nil, false 
        
           } 
        
           if !netExts["extraroute"] && !netExts["extraroute-atomic"] { 
        
           	klog.V(3).Info("Neutron extraroute extension not found, required for Routes support") 
        
           	return nil, false 
        
           }

We need to check for standard-attr-tag extension as explained in Neutron API reference.

Also security groups and especially FIPs can have their own set of tags, e.g. FIP can be preallocated in advance and a special tag can be assigned on it. Once OCCM starts to manage this FIP, it will remove the tags assigned manually. This should not happen.

Current code (after my remarks will be addressed) is only tagging FIPs on creation. Same happens with SGs, we assume it's always us creating it, if manage-security-groups=false that code won't be executed. Would that work for you? I also suggested to drop the tags replace call if user haven't defined any tags in the config.

k8s-ci-robot · 2023-12-15T04:53:28Z

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-triage-robot · 2024-05-15T21:17:17Z

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

Mark this PR as fresh with /remove-lifecycle stale
Close this PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

dulek · 2024-05-23T09:30:12Z

/remove-lifecycle stale

dulek · 2024-05-23T09:30:39Z

Ah wait, this is indeed stale. I'll just close this, we have other PRs looking at this.

k8s-ci-robot requested review from anguslees and FengyunPan2 October 18, 2023 14:41

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Oct 18, 2023

k8s-ci-robot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label Oct 18, 2023

yorubad-dev marked this pull request as ready for review October 18, 2023 14:41

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 18, 2023

kayrus reviewed Oct 18, 2023

View reviewed changes

pkg/openstack/loadbalancer.go Outdated Show resolved Hide resolved

pkg/openstack/loadbalancer.go Outdated Show resolved Hide resolved

pkg/openstack/loadbalancer.go Outdated Show resolved Hide resolved

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Oct 19, 2023

pierreprinetti reviewed Oct 19, 2023

View reviewed changes

pkg/openstack/loadbalancer.go Outdated Show resolved Hide resolved

sakshi-1505 reviewed Oct 22, 2023

View reviewed changes

pkg/openstack/loadbalancer.go Outdated Show resolved Hide resolved

pkg/openstack/loadbalancer.go Outdated Show resolved Hide resolved

pkg/openstack/loadbalancer.go Outdated Show resolved Hide resolved

yorubad-dev requested a review from sakshi-1505 October 22, 2023 19:43

yorubad-dev marked this pull request as draft October 22, 2023 19:45

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 22, 2023

yorubad-dev changed the title ~~[occm] feat : add load balancer listener tag using service annotation~~ WIP : [occm] feat : add load balancer listener tag using service annotation Oct 22, 2023

yorubad-dev force-pushed the occm-listener-tags branch from 97a664a to 8cf9de8 Compare October 23, 2023 19:21

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Oct 23, 2023

yorubad-dev changed the title ~~WIP : [occm] feat : add load balancer listener tag using service annotation~~ [occm] feat : add load balancer listener tag using service annotation Oct 23, 2023

dulek requested changes Nov 8, 2023

View reviewed changes

dulek mentioned this pull request Nov 8, 2023

[occm] - add annotation for custom octavia listener tags #2327

Open

yorubad-dev force-pushed the occm-listener-tags branch from 3d114cc to 03c525d Compare November 9, 2023 03:42

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 9, 2023

dulek reviewed Nov 9, 2023

View reviewed changes

pkg/openstack/loadbalancer.go Outdated Show resolved Hide resolved

pkg/openstack/loadbalancer.go Outdated Show resolved Hide resolved

yorubad-dev force-pushed the occm-listener-tags branch from 03c525d to 954ee65 Compare November 13, 2023 14:45

dulek requested changes Nov 13, 2023

View reviewed changes

pkg/openstack/loadbalancer.go Outdated Show resolved Hide resolved

yorubad-dev force-pushed the occm-listener-tags branch from 954ee65 to 0236639 Compare November 15, 2023 02:56

yorubad-dev force-pushed the occm-listener-tags branch from 0236639 to 10649fe Compare November 15, 2023 03:27

[occm] feat : add custom tags to load balancer and other resources us…

fbd351b

…ing service annotation

yorubad-dev force-pushed the occm-listener-tags branch from 10649fe to fbd351b Compare November 17, 2023 12:16

dulek requested changes Nov 20, 2023

View reviewed changes

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 15, 2023

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label May 15, 2024

k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label May 23, 2024

dulek closed this May 23, 2024

	func (lbaas LbaasV2) createFloatingIP(msg string, floatIPOpts floatingips.CreateOpts) (floatingips.FloatingIP, error) {
	klog.V(4).Infof("%s floating ip with opts %+v", msg, floatIPOpts)
	mc := metrics.NewMetricContext("floating_ip", "create")
	floatIP, err := floatingips.Create(lbaas.network, floatIPOpts).Extract()
	err = PreserveGopherError(err)
	if mc.ObserveRequest(err) != nil {
	return floatIP, fmt.Errorf("error creating LB floatingip: %s", err)
	}
	return floatIP, err
	}

[occm] feat : add load balancer listener tag using service annotation #2439

[occm] feat : add load balancer listener tag using service annotation #2439

Conversation

yorubad-dev commented Oct 18, 2023 • edited

k8s-ci-robot commented Oct 18, 2023

k8s-ci-robot commented Oct 18, 2023

yorubad-dev commented Oct 18, 2023

kayrus left a comment • edited

Choose a reason for hiding this comment

jichenjc commented Oct 19, 2023

pierreprinetti commented Oct 19, 2023

yorubad-dev commented Oct 19, 2023

k8s-ci-robot commented Oct 23, 2023

yorubad-dev commented Oct 23, 2023

dulek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jeffyjf commented Nov 9, 2023

dulek left a comment

Choose a reason for hiding this comment

dulek commented Nov 9, 2023

dulek commented Nov 9, 2023

yorubad-dev commented Nov 9, 2023

yorubad-dev commented Nov 13, 2023

dulek left a comment

Choose a reason for hiding this comment

yorubad-dev commented Nov 13, 2023

kayrus commented Nov 14, 2023

k8s-ci-robot commented Nov 15, 2023 • edited

dulek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dulek commented Nov 20, 2023 • edited

k8s-ci-robot commented Dec 15, 2023

k8s-triage-robot commented May 15, 2024

dulek commented May 23, 2024

dulek commented May 23, 2024

yorubad-dev commented Oct 18, 2023 •

edited

kayrus left a comment •

edited

k8s-ci-robot commented Nov 15, 2023 •

edited

dulek commented Nov 20, 2023 •

edited