cmd/k8s-operator,k8s-operator,go.{mod,sum}: make individual proxy images/image pull policies configurable #11928

irbekrm · 2024-04-30T03:06:00Z

See #11675 for context - this PR attempts to make it easier to configure what images are used for the operator proxies and make it possible to configure image pull policies.

Adds a couple new fields to ProxyClass CRD to make it possilble to configure images/image pull policies for tailscale container and tailscale init container:

apiVersion: tailscale.com/v1alpha1
kind: ProxyClass
...
spec:
...
  statefulSet:
    pod:
      tailscaleContainer:
        image: "ghcr.io/tailscale/tailscale:v1.64.0"
        imagePullPolicy: IfNotPresent
      tailscaleInitContainer:
        image: "ghcr.io/tailscale/tailscale:v1.64.0"
        imagePullPolicy: IfNotPresent

To try this out:

build a new operator image
apply CRDs from this PR kubectl apply -f ./cmd/k8s-operator/deploy/crds
deploy the operator skipping the CRD and with the new image helm upgrade --install operator tailscale-dev/tailscale-operator -n tailscale --set operatorConfig.image.repo=<image> --set installCRDs=false --set operatorConfig.image.tag=<tag>...
apply a ProxyClass with images/image pull policies configured, i.e kubectl apply -f ./cmd/k8s-operator/examples/proxyclass.yaml
apply the ProxyClass to any operator's managed proxy https://tailscale.com/kb/1236/kubernetes-operator#cluster-resource-customization-using-proxyclass-custom-resource
observe that Image/ImagePullPolicy fields of the proxy Pod have the expected values

…ges/image pull policies configurable Allow to configure images and image pull policies for individual proxies via ProxyClass.Spec.StatefulSet.Pod.{TailscaleContainer,TailscaleInitContainer}.Image, and ProxyClass.Spec.StatefulSet.Pod.{TailscaleContainer,TailscaleInitContainer}.ImagePullPolicy fields. Document that we have images in ghcr.io on the relevant Helm chart fields. Updates #11675 Signed-off-by: Irbe Krumina <irbe@tailscale.com>

irbekrm · 2024-04-30T11:09:46Z

go.mod

@@ -22,6 +22,7 @@ require (
 	github.com/dave/patsy v0.0.0-20210517141501-957256f50cba
 	github.com/dblohm7/wingoes v0.0.0-20240119213807-a09d6be7affa
 	github.com/digitalocean/go-smbios v0.0.0-20180907143718-390a4f403a8e
+	github.com/distribution/reference v0.6.0


We also have https://github.com/tailscale/tailscale/blob/v1.64.2/go.mod#L184 which is a deprectated version of the same library pulled in by some dependency of mkctr it seems https://github.com/tailscale/mkctr/blob/main/go.mod#L13. We should probably update it there if possible

irbekrm · 2024-04-30T11:12:01Z

k8s-operator/apis/v1alpha1/types_proxyclass.go

+	Image string `json:"image,omitempty"`
+	// Image pull policy. One of Always, Never, IfNotPresent. Defaults to Always.
+	// https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#image
+	// +kubebuilder:validation:Enum=Always;Never;IfNotPresent


kubebuilder:validation:Enum=Always;Never;IfNotPresent

This ensures that any ProxyClass with an ImagePullPolicy value other than Always/Never/IfNotPresent gets rejected at apply time

irbekrm · 2024-04-30T11:13:08Z

k8s-operator/apis/v1alpha1/types_proxyclass.go

-	// Container security context.
-	// Security context specified here will override the security context by the operator.
-	// By default the operator:
-	// - sets 'privileged: true' for the init container
-	// - set NET_ADMIN capability for tailscale container for proxies that
-	// are created for Services or Connector.
-	// https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#security-context
+	// List of environment variables to set in the container.
+	// https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#environment-variables
+	// Note that environment variables provided here will take precedence
+	// over Tailscale-specific environment variables set by the operator,
+	// however running proxies with custom values for Tailscale environment
+	// variables (i.e TS_USERSPACE) is not recommended and might break in
+	// the future.
 	// +optional
-	SecurityContext *corev1.SecurityContext `json:"securityContext,omitempty"`
+	Env []Env `json:"env,omitempty"`
+	// Container image name. By default images are pulled from
+	// docker.io/tailscale/tailscale, but the official images are also
+	// available at ghcr.io/tailscale/tailscale. Image name provided here
+	// will override any proxy image values specified via the Kubernetes
+	// operator's Helm chart values or PROXY_IMAGE env var to the operator
+	// deployment.
+	// https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#image
+	// +optional


This is just me shifting these fields around to order them alphabetically (just to make it easier to read the code)

irbekrm · 2024-04-30T11:13:20Z

k8s-operator/apis/v1alpha1/types_proxyclass.go

-	// List of environment variables to set in the container.
-	// https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#environment-variables
-	// Note that environment variables provided here will take precedence
-	// over Tailscale-specific environment variables set by the operator,
-	// however running proxies with custom values for Tailscale environment
-	// variables (i.e TS_USERSPACE) is not recommended and might break in
-	// the future.
+	// Container security context.
+	// Security context specified here will override the security context by the operator.
+	// By default the operator:
+	// - sets 'privileged: true' for the init container
+	// - set NET_ADMIN capability for tailscale container for proxies that
+	// are created for Services or Connector.
+	// https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#security-context
 	// +optional
-	Env []Env `json:"env,omitempty"`
+	SecurityContext *corev1.SecurityContext `json:"securityContext,omitempty"`


This is just me shifting those fields around to order them alphabetically (just to make it easier to read the code)

oxtoacart · 2024-05-14T12:46:10Z

Given the below proxy class

apiVersion: tailscale.com/v1alpha1
kind: ProxyClass
metadata:
  name: prod
spec:
  statefulSet:
    annotations:
      platform-component: infra 
    pod:
      tailscaleContainer:
        env:
          - name: FOO
            value: BAZ
      labels:
        team: eng
      nodeSelector:
        beta.kubernetes.io/os: "linux"
      imagePullSecrets:
      - name: "foo"
      tailscaleContainer:
        image: "ghcr.io/tailscale/tailscale:v1.64.0"
        imagePullPolicy: IfNotPresent
      tailscaleInitContainer:
        image: "ghcr.io/tailscale/tailscale:v1.64.0"
        imagePullPolicy: IfNotPresent

In my testing, this worked for an ingress defined as follows:

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: kuardfoobar 
  labels:
    tailscale.com/proxy-class: "prod"
spec:
  tls:
  - hosts:
    - "dnstest"
  rules:
  - http:
      paths:
      - backend:
          service:
            name: kuardfoobar
            port:
              number: 80
        pathType: Prefix 
        path: /
  ingressClassName: tailscale

But, it did not work for a service defined as follows:

apiVersion: v1
kind: Service
metadata:
  name: kuardfoobar
  labels:
    tailscale.com/proxy-class: "prod"
spec:
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP
  selector:
    app: kuardfoobar

➜  oss git:(irbekrm/docim) ✗ kubectl get pod -n tailscale ts-kuardfoobar-k2pz9-0 -o yaml | grep image
    image: ghcr.io/tailscale/tailscale:v1.64.0
    imagePullPolicy: IfNotPresent
  imagePullSecrets:
    image: ghcr.io/tailscale/tailscale:v1.64.0
    imageID: ghcr.io/tailscale/tailscale@sha256:0e7dd9e2772c32958e9d92c06ab9ec696c74075a8d558b93eca5bc25dcc9dd19

➜  oss git:(irbekrm/docim) ✗ kubectl get pod -n tailscale ts-theservice-7vmgw-0 -o yaml | grep image
    image: us-central1-docker.pkg.dev/tailscale-sandbox/percy-images/proxy-dev:v0.0.11
    imagePullPolicy: Always
    image: us-central1-docker.pkg.dev/tailscale-sandbox/percy-images/proxy-dev:v0.0.11
    imagePullPolicy: IfNotPresent
    image: us-central1-docker.pkg.dev/tailscale-sandbox/percy-images/proxy-dev:v0.0.11
    imageID: us-central1-docker.pkg.dev/tailscale-sandbox/percy-images/proxy-dev@sha256:b9c479af8e0e1ec8edadcc75ea004afed12fd3354d9c7ca62e463c997bc94845
    image: us-central1-docker.pkg.dev/tailscale-sandbox/percy-images/proxy-dev:v0.0.11
    imageID: us-central1-docker.pkg.dev/tailscale-sandbox/percy-images/proxy-dev@sha256:b9c479af8e0e1ec8edadcc75ea004afed12fd3354d9c7ca62e463c997bc94845

irbekrm · 2024-05-17T20:07:50Z

But, it did not work for a service defined as follows:

The Service resource definition in the example does not seem to have a tailscale load balancer class or an annotation for the Tailscale Kubernetes operator so it does not seem that the operator would have been creating any proxies for it.

kubectl get pod -n tailscale ts-theservice-7vmgw-0

Judging from the Pod name, this is likely for a different Service?

irbekrm force-pushed the irbekrm/docim branch from 56fd9b1 to fcfffb2 Compare April 30, 2024 10:54

irbekrm changed the title ~~cmd/k8s-operator/deploy/chart: document that images are also available from ghcr.io~~ cmd/k8s-operator,k8s-operator,go.{mod,sum}: make individual proxy images/image pull policies configurable Apr 30, 2024

irbekrm commented Apr 30, 2024

View reviewed changes

irbekrm requested a review from oxtoacart April 30, 2024 11:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cmd/k8s-operator,k8s-operator,go.{mod,sum}: make individual proxy images/image pull policies configurable #11928

cmd/k8s-operator,k8s-operator,go.{mod,sum}: make individual proxy images/image pull policies configurable #11928

irbekrm commented Apr 30, 2024 •

edited

irbekrm Apr 30, 2024

irbekrm Apr 30, 2024

irbekrm Apr 30, 2024

irbekrm Apr 30, 2024

oxtoacart commented May 14, 2024

irbekrm commented May 17, 2024

cmd/k8s-operator,k8s-operator,go.{mod,sum}: make individual proxy images/image pull policies configurable #11928

Are you sure you want to change the base?

cmd/k8s-operator,k8s-operator,go.{mod,sum}: make individual proxy images/image pull policies configurable #11928

Conversation

irbekrm commented Apr 30, 2024 • edited

irbekrm Apr 30, 2024

Choose a reason for hiding this comment

irbekrm Apr 30, 2024

Choose a reason for hiding this comment

irbekrm Apr 30, 2024

Choose a reason for hiding this comment

irbekrm Apr 30, 2024

Choose a reason for hiding this comment

oxtoacart commented May 14, 2024

irbekrm commented May 17, 2024

irbekrm commented Apr 30, 2024 •

edited