Add kube-proxy non-privileged blog post #43680

uablrek · 2023-10-25T06:38:31Z

The post describes how the --init-only flag can be used to do privileged initialization in an initContainer, and run the main container with NET_ADMIN capabilities only

Moved to kubernetes/contributor-site#452

The post describes how the --init-only flag can be used to do privileged initialization in an initContainer, and run the main container with NET_ADMIN capabilities only

k8s-ci-robot · 2023-10-25T06:38:56Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign onlydole for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

content/en/blog/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

netlify · 2023-10-25T06:41:00Z

✅ Pull request preview available for checking

Built without sensitive environment variables

Name	Link
🔨 Latest commit	`efc6700`
🔍 Latest deploy log	https://app.netlify.com/sites/kubernetes-io-main-staging/deploys/653a7fe382839b0008900dc3
😎 Deploy Preview	https://deploy-preview-43680--kubernetes-io-main-staging.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

sftim · 2023-10-25T10:37:43Z

/sig network

sftim

Some early feedback.

/hold
pending assignment of a publication date

One thing I'd mention is that the narrowed kube-proxy Pod still only meets the privileged Pod security standard. There's still an improvement because the running container doesn't need to run privileged.

sftim · 2023-10-25T16:10:18Z

content/en/blog/_posts/0000-00-00-kube-proxy-non-privileged.md

+This post describes how the `--init-only` flag added in K8s v1.29 can
+be used to perform configuration that requires privileged mode in an
+initContainer, while the main `kube-proxy` container may run with a
+stricter securityContext.


stricter securityContext.

sftim · 2023-10-25T16:10:49Z

content/en/blog/_posts/0000-00-00-kube-proxy-non-privileged.md

+privileged mode. Security aware users wants to [use capabilities instead](
+https://github.com/kubernetes/kubernetes/issues/112171).


Readers will expect this to take them to a guide, not an issue. I'd omit the link.

content/en/blog/_posts/0000-00-00-kube-proxy-non-privileged.md

Co-authored-by: Tim Bannister <tim@scalefactory.com>

danwinship · 2023-10-25T17:13:36Z

content/en/blog/_posts/0000-00-00-kube-proxy-non-privileged.md

+This post describes how the `--init-only` flag to `kube-proxy`, added in K8s v1.29, can
+be used to perform configuration that requires privileged mode in an
+initContainer, while the main `kube-proxy` container may run with a
+stricter securityContext.


maybe flip that around to put the cool part first: "can be used to run the main kube-proxy container in a stricter securityContext by performing the configuration that requires privileged mode in a separate initContainer"

I also removed the "added in K8s v1.29", since it's now in the title.

content/en/blog/_posts/0000-00-00-kube-proxy-non-privileged.md

danwinship · 2023-10-25T17:25:32Z

content/en/blog/_posts/0000-00-00-kube-proxy-non-privileged.md

+        command:
+        - /usr/local/bin/kube-proxy
+        - --config=/var/lib/kube-proxy/config.conf
+        - --init-only


I would include the --hostname-override here as well. We know that it doesn't matter, but end users won't have a good sense of which options might matter and which ones don't, so they should just specify all the same options they specify for the real container.

danwinship · 2023-10-25T17:25:55Z

content/en/blog/_posts/0000-00-00-kube-proxy-non-privileged.md

+## Improve security even more
+
+
+


Ah, this was where I intended to describe "User Namespaces", but they can't be used. I will remove the header

danwinship · 2023-10-25T17:27:41Z

content/en/blog/_posts/0000-00-00-kube-proxy-non-privileged.md

+`NET_ADMIN` capabilities only. Installers like `kubeadm` may be
+altered to use this feature.


That's very passive... you should either submit a patch to make kubeadm do it, or if not (or if there's not time to get it in for 1.29), at least file a bug against it suggesting they should, and then you can say "kubeadm will like be altered to use this feature in the future".

Will do that tomorrow

Co-authored-by: Dan Winship <danwinship@redhat.com>

uablrek · 2023-10-25T17:48:14Z

What is the practice about the commits accepted from a PR review? My intention is to squash them, but keep the "Co-authored-by:" part.

uablrek · 2023-10-25T18:00:03Z

One thing I'd mention is that the narrowed kube-proxy Pod still only meets the privileged Pod security standard. There's still an improvement because the running container doesn't need to run privileged.

@sftim
I must admit I wasn't aware of the Pod security standard until I read the User Namespaces blog yesterday. I will try to mention it in some way.

neolit123 · 2023-10-26T12:00:20Z

content/en/blog/_posts/0000-00-00-kube-proxy-non-privileged.md

+    k8s-app: kube-proxy
+spec:
+  template:
+    spec:


best to include:

image: registry.k8s.io/kube-proxy:v1.28.2 imagePullPolicy: IfNotPresent

for both init and regular containers in the example.
maybe, replace v1.28.2 with a VERSION and add note.

danwinship · 2023-10-26T12:13:26Z

One thing I'd mention is that the narrowed kube-proxy Pod still only meets the privileged Pod security standard. There's still an improvement because the running container doesn't need to run privileged.

In theory you could run two separate pods, though that would be annoying in various ways (especially that there's no such thing as a run-once DaemonSet)...

We should figure out what can be done to make the situation better...

danwinship · 2023-10-26T12:13:45Z

@uablrek you should add something somewhere about this being Linux-only / not for Windows. (Maybe explaining that it's because k8s on Windows doesn't have the equivalent of capabilities, so there's no way to give the pod just a little bit of privilege).

sftim · 2023-10-26T13:40:07Z

content/en/blog/_posts/0000-00-00-kube-proxy-non-privileged.md

+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  labels:
+    k8s-app: kube-proxy
+spec:
+  template:
+    spec:
+      initContainers:
+      - name: kube-proxy-init
+        command:
+        - /usr/local/bin/kube-proxy
+        - --config=/var/lib/kube-proxy/config.conf
+        - --hostname-override=$(NODE_NAME)
+        - --init-only
+        securityContext:
+          privileged: true
+      containers:
+      - name: kube-proxy
+        command:
+        - /usr/local/bin/kube-proxy
+        - --config=/var/lib/kube-proxy/config.conf
+        - --hostname-override=$(NODE_NAME)
+        securityContext:
+          capabilities:
+            add: ["NET_ADMIN"]


Try this, which I think is a complete manifest.

Suggested change

apiVersion: apps/v1

kind: DaemonSet

metadata:

labels:

k8s-app: kube-proxy

spec:

template:

spec:

initContainers:

- name: kube-proxy-init

command:

- /usr/local/bin/kube-proxy

- --config=/var/lib/kube-proxy/config.conf

- --hostname-override=$(NODE_NAME)

- --init-only

securityContext:

privileged: true

containers:

- name: kube-proxy

command:

- /usr/local/bin/kube-proxy

- --config=/var/lib/kube-proxy/config.conf

- --hostname-override=$(NODE_NAME)

securityContext:

capabilities:

add: ["NET_ADMIN"]

apiVersion: apps/v1

kind: DaemonSet

metadata:

namespace: kube-system

name: kube-proxy-linux

labels:

k8s-app: kube-proxy

spec:

selector:

matchLabels:

k8s-app: kube-proxy

kubernetes.io/os: linux

template:

metadata:

labels:

k8s-app: kube-proxy

kubernetes.io/os: linux

annotations:

kubernetes.io/description: >-

Node-level network proxy; part of Kubernetes

spec:

tolerations:

- key: node-role.kubernetes.io/control-plane

effect: NoSchedule

operator: Exists

os: { name: linux } # On Windows, kube-proxy needs to run privileged

# You will need a second DaemonSet if your cluster

# includes Windows nodes

nodeSelector:

kubernetes.io/os: linux

initContainers:

- name: kube-proxy-init

image: registry.k8s.io/kube-proxy:v1.28.2 # adjust this to the Kubernetes version you use

imagePullPolicy: IfNotPresent

# you should set resource requests here

# see https://k8s.io/docs/concepts/configuration/manage-resources-containers/

command:

- /usr/local/bin/kube-proxy

- --config=/var/lib/kube-proxy/config.conf

- --hostname-override=$(NODE_NAME)

- --init-only

securityContext:

privileged: true

containers:

- name: kube-proxy

image: registry.k8s.io/kube-proxy:v1.28.2 # adjust this to the Kubernetes version you use

imagePullPolicy: IfNotPresent

# you should set resource requests here

# see https://k8s.io/docs/concepts/configuration/manage-resources-containers/

command:

- /usr/local/bin/kube-proxy

- --config=/var/lib/kube-proxy/config.conf

- --hostname-override=$(NODE_NAME)

securityContext:

capabilities:

add: ["NET_ADMIN"]

terminationGracePeriodSeconds: 30

I guess this doesn't cover a ConfigMap though, and maybe we should talk about that.

I disagree on this. The kubeadm maintainers, and others who really will use this, are totally capable of figuring out the exact procedure. IMHO the blog should give normal users an idea on what is done, not be a complete instruction. It should not be TL;DR; Besides, the kubeadm manifest is likely not the only one used

The confirMap is not affected. --init-only is a CLI option only, it can't be configured in the config file

OK, but: every bit we miss out, let's clearly tell readers what's not there. We don't need to give all the details but let's make it very very obvious that this manifest is incomplete.

Even if we do all that, the website repo is likely to get issues filed by people who don't get it. Let's do what we reasonably can to minimize the rate of those issues.

Updated.

It might be even more issues if we give the illusion that the manifests are complete, and are compatible with the ones the users may have (which may not be kubeadm generated).

So, on that: let's add a very clear note that this still isn't complete: nothing populates /var/lib/kube-proxy/config.conf.

content/en/blog/_posts/0000-00-00-kube-proxy-non-privileged.md

Co-authored-by: Tim Bannister <tim@scalefactory.com>

sftim · 2023-10-26T15:13:57Z

content/en/blog/_posts/0000-00-00-kube-proxy-non-privileged.md

+*The example manifests below are not complete, but narrowed down to what is
+essential to illustrate the function.*


Thanks, but I think we can safely assume that a good proportion of readers' eyes slip right past this and start at the manifest.

(we should keep the note here, but we should also make it clear in the example manifest about which details aren't included)

sftim · 2023-10-26T15:17:36Z

Following on from https://github.com/kubernetes/website/pull/43680/files#r1373331022

Another option: write a blog for https://k8s.dev/ that is aimed at a more informed audience.

If you write an article that is aimed at cluster lifecycle tool authors, you don't have to include a sample manifest marked as kind: DaemonSet, just some snippets.

The flip side though is that if we are writing on the Kubernetes blog, we need to write for an audience that includes people who'll copy and paste what we put here and then file an issue if it doesn't work. I want to control for that risk before the stream of issues turns up.

uablrek · 2023-10-26T16:34:24Z

A complete updated manifest would be something like in kubernetes/kubernetes#120864 (comment). In your example, the volume mounts seem to be missing. If anybody do a copy/paste on those, nothing would work.

@danwinship proposed a blog in kubernetes/kubernetes#120864 (comment),
but this may not be a post with a suitable level for Kubernetes blog.

Perhaps I should close this PR, and a better option would be that a kubeadm maintainer write a blog post when this feature is supported by kubeadm?

sftim · 2023-10-26T16:46:01Z

Perhaps I should close this PR, and a better option would be that a kubeadm maintainer write a blog post when this feature is supported by kubeadm?

It's good to announce the feature at around the time we add it. If you're willing to shift this writeup to https://k8s.dev/blog - https://github.com/kubernetes/contributor-site - then we don't have to worry much about people expecting working YAML. The contributor blog has fewer readers and different expectations.

So that's one good option; another good option is to explain how to set up kube-proxy the new way. That kind of detail is an unmet need: we tell people that kubeadm can set things up for them, but if they want to learn how to do it by hand they are largely on their own (I can explain why - maybe on Slack? - if you'd like more context).
The trouble is that if we make a blog article that looks like advice on deploying kube-proxy, I really think it needs to explain what readers are hoping to find. Even if that isn't the same as what we / you had set out to write.

You can still pick, but from a smaller set of choices than you might like. I'd prefer an article that explains kube-proxy setup for Linux the new recommended way. If that's not on the cards, then an article on https://k8s.dev/blog feels like a good second place option.

uablrek · 2023-10-26T17:23:23Z

I will move to https://k8s.dev/blog. I think it requires more understanding than copy/paste to update the kube-proxy manifest and get it right, even if a complete manifest is given as an example. It isn't really hard to do one time in KinD for example, but to maintain it in a live cluster, without help from kubeadm, requires skill.

I think it's better to keep this for experienced users and developers of installation tools for now. It can be brought to a broader audience when support in kubeadm exists.

uablrek · 2023-10-27T05:27:41Z

This PR is moved to kubernetes/contributor-site#452

uablrek · 2023-10-27T06:22:13Z

@sftim Thanks a lot for your input. I have already received a query on that image to use for this, so the copy/paste syndrome is very real. I wasn't aware, and I guess you would be the one taking the hit 😄

sftim · 2024-01-02T10:02:49Z

kubernetes/contributor-site#452 is merged.

Add kube-proxy non-privileged blog post

c657a12

The post describes how the --init-only flag can be used to do privileged initialization in an initContainer, and run the main container with NET_ADMIN capabilities only

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Oct 25, 2023

k8s-ci-robot requested review from nate-double-u and onlydole October 25, 2023 06:38

k8s-ci-robot added area/blog Issues or PRs related to the Kubernetes Blog subproject language/en Issues or PRs related to English language sig/docs Categorizes an issue or PR as relevant to SIG Docs. labels Oct 25, 2023

uablrek mentioned this pull request Oct 25, 2023

kube-proxy: Optionally do privileged configs only kubernetes/kubernetes#120864

Merged

k8s-ci-robot added the sig/network Categorizes an issue or PR as relevant to SIG Network. label Oct 25, 2023

sftim reviewed Oct 25, 2023

View reviewed changes

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 25, 2023

uablrek and others added 7 commits October 25, 2023 18:48

Update content/en/blog/_posts/0000-00-00-kube-proxy-non-privileged.md

ff5dc77

Co-authored-by: Tim Bannister <tim@scalefactory.com>

Update content/en/blog/_posts/0000-00-00-kube-proxy-non-privileged.md

9377f34

Co-authored-by: Tim Bannister <tim@scalefactory.com>

Update content/en/blog/_posts/0000-00-00-kube-proxy-non-privileged.md

a9164d8

Co-authored-by: Tim Bannister <tim@scalefactory.com>

Update content/en/blog/_posts/0000-00-00-kube-proxy-non-privileged.md

3a603c5

Co-authored-by: Tim Bannister <tim@scalefactory.com>

Update content/en/blog/_posts/0000-00-00-kube-proxy-non-privileged.md

1bd7a61

Co-authored-by: Tim Bannister <tim@scalefactory.com>

Update content/en/blog/_posts/0000-00-00-kube-proxy-non-privileged.md

5bd0044

Co-authored-by: Tim Bannister <tim@scalefactory.com>

Update content/en/blog/_posts/0000-00-00-kube-proxy-non-privileged.md

670dac7

Co-authored-by: Tim Bannister <tim@scalefactory.com>

danwinship reviewed Oct 25, 2023

View reviewed changes

uablrek and others added 3 commits October 25, 2023 19:43

Update content/en/blog/_posts/0000-00-00-kube-proxy-non-privileged.md

efdde79

Co-authored-by: Dan Winship <danwinship@redhat.com>

Update content/en/blog/_posts/0000-00-00-kube-proxy-non-privileged.md

56b0b5a

Co-authored-by: Dan Winship <danwinship@redhat.com>

Update content/en/blog/_posts/0000-00-00-kube-proxy-non-privileged.md

5df63c2

Co-authored-by: Dan Winship <danwinship@redhat.com>

Updated after review

b41d321

uablrek mentioned this pull request Oct 26, 2023

Install the kube-proxy main container without privileged mode kubernetes/kubeadm#2948

Open

uablrek marked this pull request as ready for review October 26, 2023 05:24

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 26, 2023

Update after review

8ffcb58

neolit123 reviewed Oct 26, 2023

View reviewed changes

Comment that this only works on Linux

4070424

sftim reviewed Oct 26, 2023

View reviewed changes

content/en/blog/_posts/0000-00-00-kube-proxy-non-privileged.md Outdated Show resolved Hide resolved

uablrek and others added 2 commits October 26, 2023 15:51

Update content/en/blog/_posts/0000-00-00-kube-proxy-non-privileged.md

c094093

Co-authored-by: Tim Bannister <tim@scalefactory.com>

Emphasise that the example manifests are narrowed

efc6700

sftim reviewed Oct 26, 2023

View reviewed changes

uablrek mentioned this pull request Oct 27, 2023

Add kube-proxy non-privileged blog post kubernetes/contributor-site#452

Merged

uablrek closed this Oct 27, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add kube-proxy non-privileged blog post #43680

Add kube-proxy non-privileged blog post #43680

uablrek commented Oct 25, 2023 •

edited

k8s-ci-robot commented Oct 25, 2023

netlify bot commented Oct 25, 2023 •

edited

sftim commented Oct 25, 2023

sftim left a comment

sftim Oct 25, 2023

sftim Oct 25, 2023

danwinship Oct 25, 2023

uablrek Oct 25, 2023

danwinship Oct 25, 2023

danwinship Oct 25, 2023

uablrek Oct 25, 2023

danwinship Oct 25, 2023 •

edited

uablrek Oct 25, 2023

uablrek commented Oct 25, 2023

uablrek commented Oct 25, 2023

neolit123 Oct 26, 2023 •

edited

danwinship commented Oct 26, 2023

danwinship commented Oct 26, 2023

sftim Oct 26, 2023 •

edited

sftim Oct 26, 2023

uablrek Oct 26, 2023 •

edited

uablrek Oct 26, 2023

sftim Oct 26, 2023

uablrek Oct 26, 2023

sftim Oct 26, 2023

sftim Oct 26, 2023

sftim Oct 26, 2023

sftim commented Oct 26, 2023

uablrek commented Oct 26, 2023

sftim commented Oct 26, 2023

uablrek commented Oct 26, 2023 •

edited

uablrek commented Oct 27, 2023

uablrek commented Oct 27, 2023

sftim commented Jan 2, 2024

		privileged mode. Security aware users wants to [use capabilities instead](
		https://github.com/kubernetes/kubernetes/issues/112171).

		`NET_ADMIN` capabilities only. Installers like `kubeadm` may be
		altered to use this feature.

		*The example manifests below are not complete, but narrowed down to what is
		essential to illustrate the function.*

Add kube-proxy non-privileged blog post #43680

Add kube-proxy non-privileged blog post #43680

Conversation

uablrek commented Oct 25, 2023 • edited

k8s-ci-robot commented Oct 25, 2023

netlify bot commented Oct 25, 2023 • edited

✅ Pull request preview available for checking

sftim commented Oct 25, 2023

sftim left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danwinship Oct 25, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

uablrek commented Oct 25, 2023

uablrek commented Oct 25, 2023

neolit123 Oct 26, 2023 • edited

Choose a reason for hiding this comment

danwinship commented Oct 26, 2023

danwinship commented Oct 26, 2023

sftim Oct 26, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

uablrek Oct 26, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sftim commented Oct 26, 2023

uablrek commented Oct 26, 2023

sftim commented Oct 26, 2023

uablrek commented Oct 26, 2023 • edited

uablrek commented Oct 27, 2023

uablrek commented Oct 27, 2023

sftim commented Jan 2, 2024

uablrek commented Oct 25, 2023 •

edited

netlify bot commented Oct 25, 2023 •

edited

danwinship Oct 25, 2023 •

edited

neolit123 Oct 26, 2023 •

edited

sftim Oct 26, 2023 •

edited

uablrek Oct 26, 2023 •

edited

uablrek commented Oct 26, 2023 •

edited