PSP Ephemeral Containers

[phillebaba]

Newer versions of Kubernetes have introduced the concept of ephemeral containers as a beta feature. This introduces yet another container spec ephemeralContainers on top of the existing initContainers and containers. We need to update all constraints and assigns to make sure that the new spec is covered.

[phillebaba]

I wrote a quick blog post about the issues that can occur if this is not fixed.

https://xenitab.github.io/blog/2022/04/12/ephemeral-container-security

Will start working on fixing this soon hopefully.

[ritazh]

Thanks for raising this issue @phillebaba! Would you interested in opening a PR for the suggested changes?

[phillebaba]

Yep should not be an issue for me 😄

[mac-chaffee]

Should we issue a CVE or other announcement for this?

It seems that any v1.23 cluster which depends on these policies to replace PodSecurityPolicies now has no protection against privileged pods. And since there's no versioning/releases for these policies, users may not know they need to upgrade ASAP once #191 is merged.

[maxsmythe]

Putting out an announcement is a good idea. Is there a best channel for this?

[mac-chaffee]

@maxsmythe Looks like there's a procedure outlined here: https://github.com/open-policy-agent/opa/blob/main/SECURITY.md

[srenatus]

I'm afraid that document might be a little outdated. Recently, we've been using GitHub security advisories, and those seemed to work quite well. It'll end up issuing a cve that'll be picked up by the usual trackers. I haven't checked all the history here, but I suppose the advisory would best fit in the GK repo, since it's the main user of this framework and GK 's users are there ones we'd like to inform....?

[ritazh]

I suppose the advisory would best fit in the GK repo, since it's the main user of this framework and GK 's users are there ones we'd like to inform....?

This issue is technically a gap in some of the policies in this gatekeeper-library when ephemeral containers are introduced. it's not a gap in gatekeeper. We currently do not have any security procedure defined for this repo.

Should we issue a CVE or other announcement for this?

I don't know if this issue requires a CVE to be assigned but at a minimum we should definitely pin this issue to the top of the repo and pin it to slack and opa announcements once the PR is merged so users can follow the issue/PR to make their own updates.

I have added this to tomorrow community call for further discussion of security procedure for this repo.

[ritazh]

As discussed in today's community call, for issues like these in the future, we will pin the issue to the top of the repo and make announcements on slack to notify users they need to make updates.

[mac-chaffee]

FYI sent a message to open-policy-agent-security@googlegroups.com with some more details on this subject. Mentioning it here since you mentioned the security.md doc may need an update

[mac-chaffee]

Note to anyone seeing this pinned issue: The new constraints provided by #191 are necessary but NOT sufficient to protect your v1.23+ cluster from people running privileged ephemeralcontainers. You also need to update Gatekeeper to validate pods/ephemeralcontainers (it DOES NOT do this by default at the time of writing).

Until this PR is merged, these are the steps required:

1. Apply the updated constraints from Add ephemeral containers to policies #191 to your cluster
2. If you installed Gatekeeper via Helm, upgrade gatekeeper with the following extra helm values:

validatingWebhookCustomRules:
  - apiGroups: ['*']
    apiVersions: ['*']
    resources:
    - '*'
    - 'pods/ephemeralcontainers'  # If you already have custom rules, just ensure this line is present
    operations:
    - CREATE
    - UPDATE

mutatingWebhookCustomRules:
  - apiGroups: ['']
    apiVersions: ['*']
    resources:
    - '*'
    - 'pods/ephemeralcontainers'  # If you already have custom rules, just ensure this line is present
    operations:
    - CREATE
    - UPDATE

3. If you installed Gatekeeper via manifest, run kubectl edit validatingwebhookconfiguration gatekeeper-validating-webhook-configuration and add pods/ephemeralcontainers to the list of resources (in addition to '*').

After this PR is merged, update to the latest version of Gatekeeper and ensure your validating/mutatingwebhookconfiguration is still configured to check pods/ephemeralcontainers.

[phillebaba]

@mac-chaffee good catch!

I had not verified my PR outside of gator tests which explains why it was not caught there. Not really sure how we could solve this in the future however without building larger real world e2e tests.