[KKP 2.24 k8s 1.28 ONLY] - Cannot get more than one Azure Internal Load Balancer Service

[dharapvj]

What happened?

In KKP 2.24, with a user cluster of k8s version 1.28, with Azure load balancer of type internal, we can only get Service to get load balancer assigned. 2nd service onwards all services with type LoadBalancer remains in Pending status forever.

The issue does not happen in KKP 2.25 (beta3) with k8s version 1.28. It also does not happen in KKP 2.24 with k8s version 1.27.x.

Expected behavior

There should be no issue in getting LoadBalancer ip assigned to as many Services we define of type LoadBalancer.

How to reproduce the issue?

Create 2 clusters with Azure provider in KKP 2.24 (e.g. demo environment currently). One cluster should be k8s version 1.27 and other should use k8s version 1.28.

Add Application type nginx with below values.yaml snipped so that the load-balancer uses internal ip

controller:
  service:
    annotations:
      service.beta.kubernetes.io/azure-load-balancer-internal: "true"

After the clusters become ready, validate that you have one service of type LoadBalancer present and has IP address assigned.

Create one more service of type LoadBalancer (again must use the same annotation for internal load balancer)
e.g.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  labels:
    app: nginx-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-deployment
  template:
    metadata:
      labels:
        app: nginx-deployment
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: null
  annotations:
    service.beta.kubernetes.io/azure-load-balancer-internal: "true"
  labels:
    app: nginx-deployment
    cluster: user-cluster
  name: nginx
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx-deployment
    cluster: user-cluster
  type: LoadBalancer
status:
  loadBalancer: {}

Observe that 2nd load balancer gives below error

❯ kds nginx
Name:                     nginx
Namespace:                default
Labels:                   app=nginx-deployment
                          cluster=user-cluster
Annotations:              service.beta.kubernetes.io/azure-load-balancer-internal: true
Selector:                 app=nginx-deployment,cluster=user-cluster
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.240.23.58
IPs:                      10.240.23.58
Port:                     <unset>  80/TCP
TargetPort:               80/TCP
NodePort:                 <unset>  31862/TCP
Endpoints:                <none>
Session Affinity:         None
External Traffic Policy:  Cluster
Events:
  Type     Reason                  Age               From                Message
  ----     ------                  ----              ----                -------
  Normal   EnsuringLoadBalancer    5s (x2 over 11s)  service-controller  Ensuring load balancer
  Warning  SyncLoadBalancerFailed  4s (x2 over 10s)  service-controller  Error syncing load balancer: failed to ensure load balancer: failed to map VM Name to NodeName: VM Name vj128-test-7f77d9d79f-rf72b

How is your environment configured?

• KKP version: 2.24
• Shared or separate master/seed clusters?: separate

Provide your KKP manifest here (if applicable)

Defaults

What cloud provider are you running on?

Azure

What operating system are you running in your user cluster?

Ubuntu

Additional information

We are most likely up against this issue for Azure cloud controller manager which is since then fixed and is available in KKP 2.25 which is why KKP 2.25 with k8s 1.28 does not have manifest this issue.

If we can just upgrade kkp 2.24 to use upgraded Azure CCM, I think that will solve the issue.

Azure CCM v1.28.3 has the fix available in it.

KKP 2.25 with k8s 1.28 uses Azure CCM v1.28.5
KKP 2.24 with k8s 1.28 uses Azure CCM v1.28.0