## Description #1080

amuraru · 2024-04-08T13:02:13Z

Please provide a meaningful description of what this change will do, or is for. Bonus points for including links to
related issues, other PRs, or technical references.

Note that by not including a description, you are asking reviewers to do extra work to understand the context of this
change, which may lead to your PR taking much longer to review, or result in it not being reviewed at all.

Type of Change

Checklist

I have read the contributing guidelines
Existing issues have been referenced (where applicable)
I have verified this change is not present in other open pull requests
Functionality is documented
All code style checks pass
New code contribution is covered by automated tests
All new and existing tests pass

As part of banzaicloud#929, the local replacement for sub-modules was removed in favor of using valid tags. Adobe Koperator fork has some internal changes in the api sub-module, so we need to use the local version of the sub-module instead of the upstream version, so we are forced to revert the changes from banzaicloud#929.

…adobe/kafka docker hub repos - build the koperator docker image - build Apache kafka docker image When `kafka-*` tags are created a github action is triggered to build and push a new adobe/kafka docker image version Allow broker libs injector to copy jars into KAFKA_HOME

We made some chnages for spinnaker annotations and `preserveUnknownFields` that would be overriden by `make manifests`

…17) (#22)

#26)

…ised.listeners configuration This is needed for old clients connecting to kafka through Zookeeper that does not have a way to infer the right listener. In this case, the first listener in the advertised.listener config is used to connect to brokers. This patch ensures the external listeners (those reachable from outside) are listed before internal ones

…and clients 1/ Kafka broker defines connections.max.idle.ms=600s To ensure envoy as a client for kafka broker is terminating the connection first to avoid network disconnects this patch is setting the idleTimeout to value slightly less than that 2/ Enable tcp-keep alive for all TCP connections established by envoy to kafka and to client (or fronting Load Balancer)

Added explicit typeconfig for envoy.filters.http.router ``` [2022-06-16 13:27:58.425][1][info][main] [source/server/server.cc:939] exiting Didn't find a registered implementation for 'envoy.filters.http.router' with type URL: '' ```

Co-authored-by: Adrian Muraru <adi.muraru@gmail.com>

* Implement disk removal feature --------- Co-authored-by: Alex Necula <anecula@adobe.com>

* Handle the case when the request to CC STATE endpoint returns the async task instead of the result When the request on /state endpoint of CC takes longer than webserver.request.maxBlockTimeMs, then CC will convert the request into an async task and will respond with the taskID instead of the result. The Koperator should handle this case by waiting for CC to complete the task.

Co-authored-by: Petruț™ <cpetrache@adobe.com>

* Added arm64 to docker build platforms * Regenerated headers for 2024

CLAassistant · 2024-04-08T13:02:21Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
6 out of 7 committers have signed the CLA.

✅ aguzovatii
✅ alungu
✅ amuraru
✅ dobrerazvan
✅ azun
✅ ctrlaltluc
❌ cristianpetrache
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

pkg/resources/envoy/deployment.go

 			exposedPorts = append(exposedPorts, corev1.ContainerPort{
 				Name:          fmt.Sprintf("broker-%d", brokerId),
-				ContainerPort: extListener.ExternalStartingPort + int32(brokerId),
+				ContainerPort: extListener.GetBrokerPort(int32(brokerId)),


pkg/resources/envoy/configmap.go

-				return ""
+
+			if elistener.TLSEnabled() {
+				filterChain, err = GenerateEnvoyTLSFilterChain(tcpProxy, ingressConfig.EnvoyConfig.GetBrokerHostname(int32(brokerId)), log)


pkg/resources/envoy/configmap.go

-				},
-			})
+
+			brokerPort := elistener.GetBrokerPort(int32(brokerId))


pkg/resources/envoy/service.go

+			if util.ShouldIncludeBroker(brokerConfig, kafkaCluster.Status, brokerId, defaultIngressConfigName, ingressConfigName) {
+				exposedPorts = append(exposedPorts, corev1.ServicePort{
+					Name:       fmt.Sprintf("broker-%d", brokerId),
+					Port:       extListener.GetBrokerPort(int32(brokerId)),


pkg/resources/envoy/service.go

+				exposedPorts = append(exposedPorts, corev1.ServicePort{
+					Name:       fmt.Sprintf("broker-%d", brokerId),
+					Port:       extListener.GetBrokerPort(int32(brokerId)),
+					TargetPort: intstr.FromInt(int(extListener.GetBrokerPort(int32(brokerId)))),


pkg/resources/istioingress/gateway.go

@@ -70,7 +70,7 @@
 		if util.ShouldIncludeBroker(brokerConfig, kc.Status, brokerId, defaultIngressConfigName, ingressConfigName) {
 			servers = append(servers, istioclientv1beta1.Server{
 				Port: &istioclientv1beta1.Port{
-					Number:   int(externalListenerConfig.ExternalStartingPort) + brokerId,
+					Number:   int(externalListenerConfig.GetBrokerPort(int32(brokerId))),


pkg/resources/istioingress/meshgateway.go

@@ -103,8 +103,8 @@
 			generatedPorts = append(generatedPorts, &istioOperatorApi.ServicePort{
 				Name:       fmt.Sprintf("tcp-broker-%d", brokerId),
 				Protocol:   string(corev1.ProtocolTCP),
-				Port:       externalListenerConfig.ExternalStartingPort + int32(brokerId),
-				TargetPort: &istioOperatorApi.IntOrString{IntOrString: intstr.FromInt(int(externalListenerConfig.ExternalStartingPort) + brokerId)},
+				Port:       externalListenerConfig.GetBrokerPort(int32(brokerId)),


pkg/resources/istioingress/meshgateway.go

-				Port:       externalListenerConfig.ExternalStartingPort + int32(brokerId),
-				TargetPort: &istioOperatorApi.IntOrString{IntOrString: intstr.FromInt(int(externalListenerConfig.ExternalStartingPort) + brokerId)},
+				Port:       externalListenerConfig.GetBrokerPort(int32(brokerId)),
+				TargetPort: &istioOperatorApi.IntOrString{IntOrString: intstr.FromInt(int(externalListenerConfig.GetBrokerPort(int32(brokerId))))},


pkg/resources/istioingress/virtualservice.go

@@ -79,7 +79,7 @@
 			tlsRoutes = append(tlsRoutes, istioclientv1beta1.TLSRoute{
 				Match: []istioclientv1beta1.TLSMatchAttributes{
 					{
-						Port:     util.IntPointer(int(externalListenerConfig.ExternalStartingPort) + brokerId),
+						Port:     util.IntPointer(int(externalListenerConfig.GetBrokerPort(int32(brokerId)))),


pkg/resources/istioingress/virtualservice.go

@@ -132,7 +132,7 @@
 			tcpRoutes = append(tcpRoutes, istioclientv1beta1.TCPRoute{
 				Match: []istioclientv1beta1.L4MatchAttributes{
 					{
-						Port: util.IntPointer(int(externalListenerConfig.ExternalStartingPort) + brokerId),
+						Port: util.IntPointer(int(externalListenerConfig.GetBrokerPort(int32(brokerId)))),


alungu and others added 25 commits December 12, 2023 05:15

[INTERNAL] Update helm for adobe builds (#52)

d2b8021

[INTERNAL] make manifests should be called manually, if needed (#25)

e5152f7

We made some chnages for spinnaker annotations and `preserveUnknownFields` that would be overriden by `make manifests`

[INTERNAL] Allow Kafka to use External DNS for inter-broker protocol (#…

fe9adaf

…17) (#22)

[INTERNAL] Allow external listeners to be used for inner communication (

8ca9d51

#26)

[INTERNAL] Generate CRDs resources

685da4e

[INTERNAL] Build kafka 3.4.1 using Oracle OpenJDK

84345fc

Enable envoy tls termination (#41)

450ebe7

Added TaintedBrokersSelector to kafkaClusterSpec (#48)

ecf3223

Co-authored-by: Adrian Muraru <adi.muraru@gmail.com>

Remove disk feature (#53)

a40d5fe

* Implement disk removal feature --------- Co-authored-by: Alex Necula <anecula@adobe.com>

Allow concurrent broker restarts from same AZ (broker rack) (#62)

514fa07

Fix flaky test by deleting nodeports explicitly (#67)

16a9fc2

Allow dashes when parsing broker rack (#68)

cdfb6b9

Upgrade Kafka to 3.6.0 (#69)

18e7253

Upgrade dependencies

edb7ebf

Fix wrong port on expectEnvoyWithConfigAz2Tls test (#70)

5f78c06

Upgrade Kafka to 3.6.1 (#71)

7383921

Co-authored-by: Petruț™ <cpetrache@adobe.com>

Upgrade Kafka image to use Java v21 (#72)

8172c6e

Added arm64 to docker build platforms (#73)

5bcb31f

* Added arm64 to docker build platforms * Regenerated headers for 2024

Upgrading Kafka to 3.7.0 (#77)

2b16b29

amuraru requested a review from a team as a code owner April 8, 2024 13:02

Update codeql-analysis.yml (#78)

aa25e15

github-advanced-security bot found potential problems Apr 8, 2024

View reviewed changes

amuraru force-pushed the updeps branch from f1db44a to 15b0018 Compare April 8, 2024 13:11

amuraru force-pushed the updeps branch from 15b0018 to 87b08a1 Compare April 8, 2024 13:43

amuraru and others added 3 commits April 8, 2024 15:44

Upgrade dependencies

87b08a1

[INTERNAL] Create uniq leader ID per operator deployment (#76)

bb519ee

Merge branch 'master' into updeps

9fa0856

amuraru closed this May 29, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

## Description #1080

## Description #1080

amuraru commented Apr 8, 2024

CLAassistant commented Apr 8, 2024 •

edited

## Description #1080

## Description #1080

Conversation

amuraru commented Apr 8, 2024

Type of Change

Checklist

CLAassistant commented Apr 8, 2024 • edited

CLAassistant commented Apr 8, 2024 •

edited