feat: Support extra listeners on NLBs #529

jhuntwork · 2022-07-26T16:01:44Z

The zalando.org/aws-nlb-extra-listeners annotation accepts a JSON
string that describes a list of extra listen/target ports to add to an
NLB. These will be routed to pods matching a specific label in the same
namespace as the ingress. As such, this depends on the AWS CNI mode
feature.

szuecs · 2022-07-27T18:07:19Z

Thanks @jhuntwork , I think because of vacations and some pressing issues we won't be able to review your PR until September.
I hope it's fine and not too much blocking!

jhuntwork · 2022-07-27T22:21:50Z

@szuecs Thanks for the note! I think I was hoping for a quicker review than that, but I understand... vacations are important! Looking forward to the feedback when it comes.

szuecs · 2022-08-25T10:00:41Z

@jhuntwork can you please rebase on current master, I had to drop the golint linter in order to make tests work again.

README.md

szuecs · 2022-08-25T10:24:14Z

aws/cf.go

@@ -230,6 +254,7 @@ func createStack(svc cloudformationiface.CloudFormationAPI, spec *stackSpec) (st
 			cfParam(parameterIpAddressTypeParameter, spec.ipAddressType),
 			cfParam(parameterLoadBalancerTypeParameter, spec.loadbalancerType),
 			cfParam(parameterHTTP2Parameter, fmt.Sprintf("%t", spec.http2)),
+			cfParam(parameterCascadeParameter, fmt.Sprintf("%t", spec.cascade)),


this will change the CF stack as far as I understand, so it will retrigger an update.
@mikkeloscar @AlexanderYastrebov I guess it's fine to do in this case, right?

szuecs · 2022-08-25T10:31:11Z

aws/cf_template.go

+	if spec.cascade {
+		listener = ExtraListener{ListenProtocol: "TCP", ListenPort: 443, TargetPort: 443, cascade: true}
+	}
+	template.AddResource(httpsTargetGroupName, newTargetGroup(spec, parameterTargetGroupTargetPortParameter, listener))


can we do the code path with less changes on current CF stacks?
I would imagine it would be fine like:

extraListener := ExtraListener{} if spec.cascade { listener = ExtraListener{ListenProtocol: "TCP", ListenPort: 443, TargetPort: 443, cascade: true} template.AddResource(httpsTargetGroupName, newTargetGroup(spec, parameterTargetGroupTargetPortParameter, extraListener)) } else { template.AddResource(httpsTargetGroupName, newTargetGroup(spec, parameterTargetGroupTargetPortParameter)) }

and later only use extraListener in case we have if spec.cascade.

I think you still have to pass in the parameter to the function? Which is why it's just initialized as empty first before the if statement.

aws/cf_template.go

szuecs · 2022-08-25T10:38:09Z

@jhuntwork can you please split the 2 features into 2 separate PRs? It will be easier to review.
Another request is to add tests that cover the new feature in some way.

jhuntwork · 2022-08-25T13:49:42Z

@jhuntwork can you please split the 2 features into 2 separate PRs? It will be easier to review. Another request is to add tests that cover the new feature in some way.

Sure, I can do that. I needed both to be able to handle my specific use case, but happy to split them apart. And agreed, test coverage would be good. Thanks for the feedback so far.

jhuntwork · 2022-09-15T14:12:00Z

@szuecs This is reduced to just the first feature now and also rebased on latest master. I'm still working on a couple of tests, so those will be coming soon, hopefully.

jhuntwork · 2022-10-06T02:37:24Z

@szuecs Updated with some tests. Should be ready for review again, please.

jhuntwork · 2022-11-17T21:07:02Z

Hi @szuecs. Is there something more I need to do to get this reviewed?

szuecs · 2022-11-21T19:32:17Z

@jhuntwork the feature enables to have git like interfaces right?
The extra listener would be for ssh for example. Do I get this right or do you see any other relevant use case?
This week is cyberweek, so likely next week. And I will try to review with @AlexanderYastrebov

webframp · 2023-03-08T14:22:12Z

Thanks for sharing the update @szuecs, looking forward to this getting merged eventually.

szuecs · 2023-03-08T14:33:38Z

You can see the progress in our test strategy #587

szuecs · 2023-04-22T15:07:45Z

@jhuntwork can you rebase and create a golden file test?

jhuntwork · 2023-04-25T18:10:46Z

@jhuntwork can you rebase and create a golden file test?

Thanks, will do!

jhuntwork · 2023-11-16T17:38:15Z

Just coming back to this after a long while. Rebased and will now work on the goldenfile and tests.

The `zalando.org/aws-nlb-extra-listeners` annotation accepts a JSON string that describes a list of extra listen/target ports to add to an NLB. These will be routed to pods matching a specific label in the same namespace as the ingress. As such, this depends on the AWS CNI mode feature. Signed-off-by: Jeremy Huntwork <jeremy@merelinux.org>

jhuntwork · 2023-11-17T01:56:04Z

The tests are working, but there's just a couple things more I want to refactor. I'll submit a few more changes and remove the Draft status once it's in better shape.

jhuntwork · 2023-12-19T13:40:28Z

@szuecs I was thinking maybe some parts could use a refactor, but I might just need to get some of your input on this again now before changing anything else. Do you have time to review?

szuecs · 2023-12-22T20:13:40Z

aws/adapter.go

@@ -655,8 +665,33 @@ func (a *Adapter) UpdateTargetGroupsAndAutoScalingGroups(stacks []*Stack, proble
 		a.TargetCNI.TargetGroupCh <- targetTypesARNs[elbv2.TargetTypeEnumIp]
 	}

+	// run through any target groups with ALB targets and register all ALBs
+	for _, tg := range targetTypesARNs[elbv2.TargetTypeEnumAlb] {


What's about NLBs?

szuecs · 2023-12-22T20:16:32Z

aws/adapter.go

+func (a *Adapter) getRegisteredTargets(tgARN string) ([]string, error) {
+	tgh, err := a.elbv2.DescribeTargetHealth(&elbv2.DescribeTargetHealthInput{TargetGroupArn: &tgARN})
+	if err != nil {
+		log.Errorf("unable to describe target health %v", err)


log.Errorf("unable to describe target health %v", err) -> log.Errorf("Failed to describe target health %v", err)

szuecs · 2023-12-22T20:20:13Z

aws/adapter.go

@@ -1102,36 +1139,56 @@ func nonTargetedASGs(ownedASGs, targetedASGs map[string]*autoScalingGroupDetails
 	return nonTargetedASGs
 }

+func (a *Adapter) getRegisteredTargets(tgARN string) ([]string, error) {
+	tgh, err := a.elbv2.DescribeTargetHealth(&elbv2.DescribeTargetHealthInput{TargetGroupArn: &tgARN})


&tgARN is correct but we should use aws.String(tgARN)

szuecs · 2023-12-22T20:20:54Z

aws/adapter.go

+	}
+	registeredTargets := make([]string, len(tgh.TargetHealthDescriptions))
+	for i, target := range tgh.TargetHealthDescriptions {
+		registeredTargets[i] = *target.Target.Id


*target.Target.Id is correct but we should use aws.StringValue(target.Target.Id) instead

szuecs · 2023-12-22T20:21:35Z

aws/adapter.go

+func (a *Adapter) registerAndDeregister(new []string, old []string, tgARN string) error {
+	toRegister := difference(new, old)
+	if len(toRegister) > 0 {
+		log.Info("Registering CNI targets: ", toRegister)


better use formatstring

szuecs · 2023-12-22T20:22:02Z

aws/adapter.go

+	}
+	toDeregister := difference(old, new)
+	if len(toDeregister) > 0 {
+		log.Info("Deregistering CNI targets: ", toDeregister)


better use formatstring

szuecs · 2023-12-22T20:27:46Z

aws/asg.go

+					log.Debugf("Looking for tags on %s", aws.StringValue(tg.TargetGroupArn))
+					out, err := elbv2svc.DescribeTags(&elbv2.DescribeTagsInput{ResourceArns: []*string{tg.TargetGroupArn}})
+					if err != nil {
+						log.Errorf("cannot describe tags on target group: %v", err)


We always start uppercase for logs "Failed to describe...".

Does it make sense to continue here, because otherwise you work with empty string below the else.
Or omit setting the label with empty string.
Wdyt?

szuecs · 2023-12-22T20:34:36Z

aws/cf.go

-	}
-	if arn, ok := o[outputHTTPTargetGroupARN]; ok {
-		arns = append(arns, arn)
+	for k, v := range o {


hmm, I think it's better to have the 2 map lookups instead of a range loop over the whole map.
Do we have more than these 2 map lookups, now?

szuecs · 2023-12-22T20:38:55Z

aws/cf.go

@@ -473,6 +504,13 @@ func mapToManagedStack(stack *cloudformation.Stack) *Stack {
 		if key == ingressOwnerTag {
 			ownerIngress = value
 		}
+
+		if key == extraListenersTag {
+			decodedListeners, _ := base64.StdEncoding.DecodeString(value)


you should use err and return early if there is one

szuecs · 2023-12-22T20:42:31Z

aws/cf.go

+		if key == extraListenersTag {
+			decodedListeners, _ := base64.StdEncoding.DecodeString(value)
+			if err := json.Unmarshal(decodedListeners, &extraListeners); err != nil {
+				return &Stack{}, err


should not this return nil?

szuecs · 2023-12-22T20:43:48Z

aws/cf.go

 				}
 			}
 			return true
 		})
 	if err != nil {
 		return nil, fmt.Errorf("findManagedStacks failed to list stacks: %w", err)
 	}
+	if len(errors) > 0 {
+		return nil, fmt.Errorf("mapToManagedStacks returned errors: %v", errors)


szuecs · 2023-12-22T20:46:22Z

aws/cf.go

+					if err != nil {
+						errors = append(errors, err)
+					}
+					stacks = append(stacks, stack)


we would here either add empty stack (&Stack{}) or nil. We could think of a continue in the error path instead.
I am not sure if we need empty stacks or nil stacks for later use, maybe for deletion or new creation.

szuecs · 2023-12-22T20:52:37Z

aws/cf_test.go

+		s := stackOutput{outputLoadBalancerARN: want}
+		got := s.lbARN()
+		if want != got {
+			t.Errorf("unexpected result. wanted %+v, got %+v", want, got)


want and got should be string so %s

szuecs · 2023-12-22T20:53:27Z

kubernetes/adapter.go

+			return nil, errors.New("extra listeners are only supported on NLBs")
+		}
+		if err := json.Unmarshal([]byte(rawlisteners), &extraListeners); err != nil {
+			return nil, fmt.Errorf("unable to parse aws-nlb-extra-listeners annotation: %v", err)


in fmt.Errorf always use %w for passing err.

szuecs · 2023-12-22T20:54:56Z

kubernetes/adapter.go

+			return nil, fmt.Errorf("unable to parse aws-nlb-extra-listeners annotation: %v", err)
+		}
+		for idx, listener := range extraListeners {
+			if listener.ListenProtocol != "TCP" && listener.ListenProtocol != "UDP" && listener.ListenProtocol != "TCP_UDP" {


maybe nicer to switch:

switch listener.ListenProtocol { case "TCP","UDP","TCP_UDP": // ok default: return nil, error.. }

szuecs · 2023-12-22T20:59:35Z

kubernetes/pods.go

 	"sync"
 	"time"

 	log "github.com/sirupsen/logrus"
+	"github.com/zalando-incubator/kube-ingress-aws-controller/aws"


this should not be a dependency in the kubernetes package
We should convert it into aws.CNIEnpoint in aws/adapter.go or in worker.go

szuecs · 2023-12-22T21:11:18Z

@jhuntwork I did my review. In general try to have no "aws" in "kubernetes" package and also the other way around.
We build a business object "Ingress" for the business logic for aws.

szuecs · 2023-12-22T23:25:39Z

Maybe split the pr into more than one. One refactoring (type introduction can be one) and the actual extraListener/extraTg.

jhuntwork · 2024-01-01T13:44:27Z

Thanks for the feedback! I'll see about making it smaller, refactoring in prep for the feature.

jhuntwork force-pushed the master branch from 703516e to 379d788 Compare July 26, 2022 16:02

jhuntwork force-pushed the master branch 2 times, most recently from 7394ef1 to dc5436f Compare August 19, 2022 14:10

szuecs reviewed Aug 25, 2022

View reviewed changes

README.md Show resolved Hide resolved

szuecs reviewed Aug 25, 2022

View reviewed changes

aws/cf_template.go Outdated Show resolved Hide resolved

jhuntwork force-pushed the master branch from dc5436f to d5d1a01 Compare September 15, 2022 14:09

jhuntwork changed the title ~~feat: Advanced options for NLBs~~ feat: Support extra listeners on NLBs Sep 15, 2022

jhuntwork force-pushed the master branch 2 times, most recently from 3531c9b to 1411785 Compare September 15, 2022 14:17

jhuntwork force-pushed the master branch 7 times, most recently from 9ef95a9 to d89b01f Compare October 6, 2022 02:28

jhuntwork force-pushed the master branch from d89b01f to 2724f44 Compare November 8, 2022 20:37

jhuntwork requested a review from szuecs November 8, 2022 20:55

szuecs requested a review from AlexanderYastrebov November 21, 2022 19:32

jhuntwork force-pushed the master branch from 2724f44 to ac3667d Compare November 16, 2023 17:06

jhuntwork force-pushed the master branch from ac3667d to af40cbe Compare November 16, 2023 18:40

jhuntwork marked this pull request as draft November 17, 2023 01:55

szuecs reviewed Dec 22, 2023

View reviewed changes

feat: Support extra listeners on NLBs #529

Are you sure you want to change the base?

feat: Support extra listeners on NLBs #529

Conversation

jhuntwork commented Jul 26, 2022 • edited

szuecs commented Jul 27, 2022

jhuntwork commented Jul 27, 2022

szuecs commented Aug 25, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

szuecs commented Aug 25, 2022

jhuntwork commented Aug 25, 2022

jhuntwork commented Sep 15, 2022

jhuntwork commented Oct 6, 2022

jhuntwork commented Nov 17, 2022

szuecs commented Nov 21, 2022

webframp commented Mar 8, 2023

szuecs commented Mar 8, 2023

szuecs commented Apr 22, 2023

jhuntwork commented Apr 25, 2023

jhuntwork commented Nov 16, 2023

jhuntwork commented Nov 17, 2023

jhuntwork commented Dec 19, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

szuecs Dec 22, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

szuecs Dec 22, 2023 • edited

Choose a reason for hiding this comment

szuecs commented Dec 22, 2023

szuecs commented Dec 22, 2023

jhuntwork commented Jan 1, 2024

jhuntwork commented Jul 26, 2022 •

edited

szuecs Dec 22, 2023 •

edited

szuecs Dec 22, 2023 •

edited