validating webhook is sent to non-leader pod will cause the webhook fail in gateway discovery mode

[randmonkey]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

Origin: FTI-5955
The customer installs 2 KIC pods and 2 Kong gateway pods with gateway discovery mode. When one of gateway pod is deleted, the creating of new resources may fail on webhook sent the validating requests to the outdated gateway pod IPs. The error says

Post "https://<old KG POD IP>.kong-gateway-admin.kong.svc:8444/schemas/routes/validate": 
dial tcp: lookup <old KG POD IP>.kong-gateway-admin.kong.svc on 10.96.0.10:53: no such host

I searched in the KIC logs, if the KIC is not elected as the leader, its gateway clients are not updated. While it is still in the "ready" endpoints of webhook service.

Expected Behavior

When gateway pod is deleted, the validation requests should not be sent to outdated pod IPs in time.

Steps To Reproduce

1. Install KIC with 2 KIC pods and 2 Kong gateway pods
2. delete one of the gateway pods
3. apply resources requiring to validate against Kong gateway right after KG pods deleted, like ingress. Example:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  namespace: httpbin
  name: 10ig-httpbin
  annotations:
    kubernetes.io/ingress.class: kong
    konghq.com/strip-path: "true"
    konghq.com/preserve-host: "true"
    konghq.com/methods: GET
spec:
  rules:
  - host: li.test.com
    http:
      paths:
      - path: /
        pathType: ImplementationSpecific
        backend:
          service: 
            name: httpbin
            port:
              number: 80

Kong Ingress Controller version

3.1.2, but related parts are not changed until the latest 3.1.4. I assume that this will also happen with the latest version.

Kubernetes version

1.27 (should not be related)

Anything else?

Possible Solutions:

• Let the non-leader not to accept validating webhooks
• Make the non-leader also able to reconcile admin service endpoints