Proposal: decompose --privileged into individual --security-opt options

[neersighted]

Description

These two pull requests have (re-)surfaced recently, and are motivated by the the want to make --privileged less blunt/all-encompassing, in order to enable new use-cases that only require some of its features.

• seccomp: allow specifying a custom profile with --privileged #47500
• new security-opt: privileged-without-host-devices (for safe DinD with Kata) #39702
• Unable to run systemd in docker with ro /sys/fs/cgroup after systemd 248 host upgrade #42275 (comment)

Today there is a lot that only --privileged can do, and the semantics of combining it with existing --security-opt are unclear, even to maintainers. A partially correct/semi-out-of-date list is here:

• docs: properly document "--privileged" #24387

To try and solve this in a way that lets us move forward and doesn't incur more technical debt, blow up the combinatorial matrix in a way that is hard to understand, or introduce even more confusion for users, I'd like to propose eliminating --privileged entirely.

Instead, --privileged could be clientside sugar that decomposes into a (new) set of --security-opt flags that poke all the holes in the container we create today, but in a composable fashion. This would also enable us to solve the 'how to combine' problem above, by empowering the user to express exactly the semantics they desire; we can make --privileged and --security-opt an error, and print a suggested command line that specifies each option granularity instead (or alternatively, lists all the knobs but doesn't enumerate those left at default values).

While this isn't quite a full entitlements system (#32801), I think it does help with other use-cases, like moby/swarmkit#3072 as well (allowing for something more granular would solve most of the objections the SwarmKit maintainers have historically had), and would align nicely with recent work in SwarmKit:

• Add support for swarm seccomp and apparmor #46386.
• docker service create "--security-opt" option #41371

[thaJeztah]

I need to refresh my memory on this one; I seem to recall I proposed or discussed this as an option at some point, but there were some reasons why setting these options client-side (or all individually) to be more declarative had some issues.

But maybe I'm conflating with my suggestion to --cap-add ALL do something similar (lookup all capabilities and set them all, instead of ALL to be in the container's config).

[thaJeztah]

For reference; some prior discussion on those parts (problem at least was that if we wanted "defaults" to be set by the client, we would need some way to learn about those defaults); some prior discussions (but I think there may have been other locations)

• add cli integration for masked and readonly paths docker/cli#1347 (comment)
• [carry 2663] Add capabilities support to stack/service commands docker/cli#2687
• Add support for exact list of capabilities + capAdd / capDrop refactor #38380
• Missing from Swarmmode --cap-add  #25885 (comment)

[neersighted]

The list of things have been relatively stable enough that I think I'm okay with older clients defaulting to an older list of security-opts; that being said if we don't like that, we can add some sort of _ping endpoint equivalent for the client to discover the default set...

[tianon]

(added #42275 (comment) to the list in the OP)