Calico for Windows does not work with nginx load balancer or nginx-ingress

[SteveCurran]

What happened:
We are testing calico network policy for windows with 1.20.2. Our current configuration is a nginx load balancer directing traffic to nginx-ingress controllers in different AKS clusters. Unfortunately it appears since calico for windows requires and enables WinDSR it prevents network flow from nginx and nginx-ingress. Exposing the service with a public load balancer and making requests directly to the service's load balancer works.

What you expected to happen:
I would like clarification that this is indeed the case and what possible work arounds are available. Having to expose each deployment's endpoint with a public load balancer will be costly. Are there plans to implement windows calico network policies without the use of WinDSR? Does anyone know if WinDSR(DSR) will eventually support the use of ingress controllers?

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment: Mix node Linux/Windows pools

• Kubernetes version (use kubectl version): 1.20.2
• Size of cluster (how many worker nodes are in the cluster?) 4
• General description of workloads in the cluster (e.g. HTTP microservices, Java app, Ruby on Rails, machine learning, etc.)
• Others: nginx-ingress

[ghost]

Hi SteveCurran, AKS bot here 👋
Thank you for posting on the AKS Repo, I'll do my best to get a kind human from the AKS team to assist you.

I might be just a bot, but I'm told my suggestions are normally quite good, as such:

1. If this case is urgent, please open a Support Request so that our 24/7 support team may help you faster.
2. Please abide by the AKS repo Guidelines and Code of Conduct.
3. If you're having an issue, could it be described on the AKS Troubleshooting guides or AKS Diagnostics?
4. Make sure your subscribed to the AKS Release Notes to keep up to date with all that's new on AKS.
5. Make sure there isn't a duplicate of this issue already reported. If there is, feel free to close this one and '+1' the existing issue.
6. If you have a question, do take a look at our AKS FAQ. We place the most common ones there!

[ghost]

Triage required from @Azure/aks-pm

[miwithro]

@keikhara

[ghost]

Triage required from @Azure/aks-pm

[ghost]

Action required from @Azure/aks-pm

[keikhara]

@SteveCurran as responded on yammer, please open a case for this so we can investigate.

[ghost]

Triage required from @Azure/aks-pm

[AbelHu]

@SteveCurran Calico requires WinDSR so AKS enables WinDSR by default when enabling Calico on Windows nodes. For your request implement windows calico network policies without the use of WinDSR, I think that you can create on one issue in https://github.com/projectcalico/calico

[AbelHu]

@SteveCurran Do you have two or more AKS clusters and the nginx load balancer is in one cluster? Could you share the topology in your environment with us so we can investigate why the nginx load balancer does not work with nginx-ingress controllers in different AKS clusters when WinDSR is enabled?

[SteveCurran]

@AbelHu Very simple. We have one nginx (1.19.8) outside of cluster, forwards to the service with an azure provisioned public ip. With DSR enabled we get gateway timeout. No problems with clusters without DSR enabled.

[ghost]

Triage required from @Azure/aks-pm

[ghost]

Action required from @Azure/aks-pm

[SteveCurran]

@AbelHu Microsoft support is now telling us that DSR/AKS does not support the use of in cluster ingress controller similar to nginx. "DSR AKS does not seem to support using Load balancer service type due to current floating IP limitation
When Floating IP / DSR is being implemented, the Front-end IP is configured within the VM, and not in the Load Balancer.
With the Floating IP rule, your application must use the primary IP configuration for outbound SNAT flows. If your application binds to the frontend IP address configured on the loopback interface in the guest OS, Azure's outbound SNAT is not available to rewrite the outbound flow and the flow fails. Our recommendation is to switch to an AGIC instead of the in-cluster Nginx ingress controller."

I am hoping there is a way to turn off DSR in 1.20.+ Otherwise we will have to incur the cost of configuring and using an application gateway for each/all clusters. Our costs will significantly increase due to this limitation. We will have to modify all of our pipelines to accommodate this change.

[AbelHu]

@SteveCurran, we can disable WinDSR in your subscription if you need to turn off DSR in 1.20.x. This feature is included in AKS RP release v20210429 so we can disable WinDSR in your subscription after v20210429 is available.

[SteveCurran]

@AbelHu will disabling WinDSR also disable it for linux nodes. DSR on linux has the same issues.

[AbelHu]

@SteveCurran No. WinDSR is a Windows feature. cc @xuto2 for DSR on Linux.

[ghost]

Triage required from @Azure/aks-pm

[AbelHu]

@SteveCurran if you have not upgraded to v1.20.x, please file a support ticket to ask AKS PG to disable WinDSR in your subscriptions before upgrading.
If you have upgraded to v1.20.x, you also need to file a support ticket but you need to upgrade your clusters or do some update operations (for example, updating the Windows password) to update your Windows nodes without WinDSR after AKS PG disable WinDSR in your subscriptions.

[ghost]

Triage required from @Azure/aks-pm

[ghost]

Action required from @Azure/aks-pm

[ghost]

Issue needing attention of @Azure/aks-leads

[ghost]

Issue needing attention of @Azure/aks-leads

[AbelHu]

@SteveCurran Could you try to set --set controller.service.externalTrafficPolicy=Local in your ingress and test it again?
NOTE: You only can enable WinDSR in your cluster with enabling Calico Windows.

If you would like to enable client source IP preservation for requests to containers in your cluster, add --set controller.service.externalTrafficPolicy=Local to the Helm install command.

Reference: https://docs.microsoft.com/en-us/azure/aks/ingress-basic

[ghost]

Triage required from @Azure/aks-pm

[ghost]

Action required from @Azure/aks-pm

[ghost]

Issue needing attention of @Azure/aks-leads

[AbelHu]

The fix in kubeproxy is done in below PRs. I think that it will be fixed with Windows calico after the new k8s versions are supported in AKS.
1.22: kubernetes/kubernetes#103138
1.21: kubernetes/kubernetes#103140
1.20: kubernetes/kubernetes#103139

[ghost]

Triage required from @Azure/aks-pm

[ghost]

Action required from @Azure/aks-pm

[ghost]

Issue needing attention of @Azure/aks-leads

[SteveCurran]

@AbelHu Thanks for fixing this. We can now use nginx-ingress in aks 1.21.2 running windows calico.