cmd/{k8s-operator,k8s-nameserver},k8s-operator: update nameserver config with records for ingress/egress proxies

[irbekrm]

This PR adds functionality to the operator to optionally populate DNS records config for the in-cluster ts.net nameserver (added in #11017) with records for cluster proxies' MagicDNS names:

• For Tailscale Ingress the record is a mapping between Ingresss MagicDNS name and the proxy's Pod IP

• For cluster egress proxies configured via tailscale.com/tailnet-fqdn annotation the record is a mapping between the annotation's value and the proxy's Pod IP

No records are currently created for any other proxy types.

The records are created only if there is a ready tailscale.com/v1alpha1.DNSConfig instance in the cluster (see #11017 ) i.e if users have explicitly configured the operator to deploy the in-cluster ts.net nameserver.

Below see the user flow for a setup where:

• cluster A runs a service exposed over Tailscale Ingress that should be accesible by callers on tailnet as well as cluster workloads via the same DNS name (the MagicDNS of the Ingress) (see use case description).
  cluster A uses kube-dns.
• cluster B's workloads also need to access the service in cluster A and would like to do that using the same Ingress. They can do it by exposing the Ingress to their cluster (see cross-cluster connectivity). Because A's Service is exposed over HTTPS, they need to use the MagicDNS name of the Ingress (so it needs to resolve within the cluster). See a use case.
  cluster B uses CoreDNS.

In cluster A:

1. Deploy the operator
2. Create a DNSConfig to tell the operator to deploy an in-cluster ts.net nameserver, i.e

$ kubectl apply -f ./cmd/k8s-operator/examples/dnsconfig.yaml

4. Update kube-dns ConfigMap to tell kube-dns to use the ts.net nameserver to resolve ts.net DNS names

$ k get dnsconfig NAME NAMESERVERIP ts-dns 10.27.1.90 $ kubectl get cm kube-dns -n kube-system -oyaml apiVersion: v1 data: stubDomains: | { "ts.net": ["10.27.1.90"] } kind: ConfigMap metadata: name: kube-dns namespace: kube-system ...

6. Create a workload exposed via tailscale Ingress.
   Note the use of tailscale.com/experimental-forward-cluster-traffic-via-ingress annotation- this makes the ingress proxy to listen on the Pod's IP address which it does not do by default.

apiVersion: apps/v1 kind: Deployment metadata: name: kuardfoobar spec: selector: matchLabels: app: kuardfoobar template: metadata: labels: app: kuardfoobar spec: containers: - image: gcr.io/kuar-demo/kuard-amd64:1 name: kuardfoobar ports: - containerPort: 8080 --- apiVersion: v1 kind: Service metadata: name: kuardfoobar spec: ports: - port: 80 targetPort: 8080 type: ClusterIP selector: app: kuardfoobar --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: kuardfoobar annotations: tailscale.com/experimental-forward-cluster-traffic-via-ingress: "true" spec: tls: - hosts: - "dnstest" rules: - http: paths: - backend: service: name: kuardfoobar port: number: 80 pathType: Prefix path: / ingressClassName: tailscale

7. Observe the MagicDNS name of the Ingress and that the workload can be accessed over that from a non-tailscale Pod in cluster:

$ kubectl get ingress NAME CLASS HOSTS ADDRESS PORTS AGE kuardfoobar tailscale * dnstest-3.tailbd97a.ts.net 80, 443 64m $ kubectl exec -it somepod -- sh / # curl -vvv https://dnstest-3.tailbd97a.ts.net * Host dnstest-3.tailbd97a.ts.net:443 was resolved. * IPv6: (none) * IPv4: 10.0.0.6 * Trying 10.0.0.6:443... * Connected to dnstest-3.tailbd97a.ts.net (10.0.0.6) port 443 ....

In cluster B:

1. Deploy the operator
2. Create a DNSConfig to tell the operator to deploy an in-cluster ts.net nameserver, i.e

$ kubectl apply -f ./cmd/k8s-operator/examples/dnsconfig.yaml

4. Update CoreDNS config to tell it to use the ts.net nameserver to resolve ts.net DNS names:

$ kubectl get dnsconfig NAME NAMESERVERIP ts-dns 10.100.198.212 $ kubectl get cm coredns -n kube-system -oyaml apiVersion: v1 data: Corefile: | ts.net { cache 5 errors forward . 10.100.198.212 } .:53 { } kind: ConfigMap metadata: name: coredns namespace: kube-system

6. Create a tailscale egress proxy with a tailscale.com/tailnet-fqdn annotation pointing at the Ingress in cluster A:

kind: Service metadata: annotations: tailscale.com/tailnet-fqdn: dnstest-3.tailbd97a.ts.net name: kuard-egress spec: type: ExternalName

8. Verify that you can access the workload in cluster A over the MagicDNS name of Ingress in cluster A from a non-tailscale Pod in cluster B:

$ kubectl exec -it sompod -- sh # curl -vvv https://dnstest-3.tailbd97a.ts.net * Host dnstest-3.tailbd97a.ts.net:443 was resolved. * IPv6: (none) * IPv4: 172.31.24.228 * Trying 172.31.24.228:443... * Connected to dnstest-3.tailbd97a.ts.net (172.31.24.228) port 443 ...

Notes:

• users have to manually update kube-dns/CoreDNS config to add the stub nameserver's Service IP
• IPv6 Pod addresses currently not supported
• To access Tailscale Ingress from within a cluster via its MagicDNS name, users also have to annotate the Ingress resource with tailscale.com/experimental-forward-cluster-traffic-via-ingress annotation, see
  cmd/{containerboot,k8s-operator/deploy/manifests}: optionally allow proxying cluster traffic to a cluster target via ingress proxy #11036
• I have also tested that non-operator managed tailscale instances that accept Tailscale DNS, running in a Pod in a cluster configured with our nameserver, still use 100.100.100.100 to resolve tailnet DNS names. Operator configured proxies do not accept Tailscale DNS, so they would use the ts.net nameserver. I cannot think of cases where this would be an issue (because they don't need to talk to anything on tailnet via MagicDNS)

cc @jaxxstorm

Next steps:

• update build infra to publish nameserver images

[irbekrm]

These changes make these functions re-usable for dns-records reconciler.

[irbekrm]

I will add functionality to actually push the nameserver image in a follow-up PR

[irbekrm]

Initially I was creating records for all proxy types. However, the L3 ingress proxies don't route cluster traffic to the backend. Also no-one has asked for this functionality for cluster ingress proxies and with regards to the egress proxies, it seems good enough to only implement it for the ones configured via tailscale.com/tailnet-fqdn annotation and not tailscale.com/tailnet-ip. We can always add more functionality later.

[irbekrm]

Currently always a single IP address

[irbekrm]

I'll add IPv6 support in a follow-up. At the moment the nameserver only responds to queries for A records.

[Scalahansolo]

What's the status of this? did this PR get abandoned in favor of a different approach?

[irbekrm]

Hi @Scalahansolo , it's not abandoned, just that we're a bit low bandwidth right now. The current plan is still to get it in.

[oxtoacart]

Generally LGTM, though I'm not familiar with the K8s APIs being used.

I left a few comments/questions.

[irbekrm]

Updated this to not have dnsconfig.status.nameserverStatus