WIP: Add a `/etc/containers/auth.json` #1746

cgwalters · 2022-12-08T16:28:39Z

A long-running tension in the docker/podman land is around running as a system service versus being executed by a user. (Specifically a "login user", i.e. a Unix user that can be logged
into via ssh etc.)

For login users, it makes total sense to configure the container
runtime in $HOME.

But for system services (e.g. code executed by systemd) it
is generally a bad idea to access or read the /root home
directory. On image based systems, /root may be dynamically
mutable state in contrast to /etc which may be managed
by OS upgrades, or even be read-only.

For these reasons, let's introduce /etc/contaners/auth.json.
If it is present, and the current process is executing in
systemd, it will be preferred. (There's some further logic
around this that is explained in the manpage; please see that
for details)

cc coreos/rpm-ostree#4180

Signed-off-by: Colin Walters walters@verbum.org

cgwalters · 2022-12-08T16:30:23Z

Specifically for example I plan to ensure that bootc to reads this file and not /root/.docker etc by default. For podman to ensure maximum compatibility I think we need to add this file into the set.

cgwalters · 2022-12-08T16:31:14Z

(Lightly tested with an override to point skopeo at this)

cgwalters · 2022-12-08T16:34:43Z

For OpenShift I think it would make sense to have /var/lib/kubelet/config.json be a symlink to /etc/containers/auth.json. This would fix a pile of long-running hacks that hardcode podman run --authfile=/var/lib/kubelet/config.json.

mtrmac

Thanks! Using a PR to discuss the details is definitely helpful. Just a quick first impression for now.

For OpenShift I think it would make sense to have /var/lib/kubelet/config.json be a symlink to /etc/containers/auth.json.

Yes, if there is any kind of computer where having a system-wide credentials file makes sense, an OpenShift node should be that. OTOH does this mean we are giving up on some SELinux restrictions on root? The /etc/containers path is necessarily unrestricted. I didn’t look into how the Kubelet is, or isn’t restricted.

Maybe we could do it the other way, and symlink /etc/containers/auth.json to the Kubelet path.

pkg/docker/config/config.go

mtrmac · 2022-12-08T18:08:31Z

pkg/docker/config/config.go

@@ -204,7 +208,17 @@ func GetAllCredentials(sys *types.SystemContext) (map[string]types.DockerAuthCon
 // The homeDir parameter should always be homedir.Get(), and is only intended to be overridden
 // by tests.
 func getAuthFilePaths(sys *types.SystemContext, homeDir string) []authPath {


Adding the new paths to this function means that writers would never write to systemPath without a specific override; is that intended?

I think that’s actually a good idea, so that root’s interactive login doesn’t write to a noninteractively-used file, but it should be documented.

mtrmac · 2022-12-08T18:10:52Z

pkg/docker/config/config.go

@@ -204,7 +208,17 @@ func GetAllCredentials(sys *types.SystemContext) (map[string]types.DockerAuthCon
 // The homeDir parameter should always be homedir.Get(), and is only intended to be overridden
 // by tests.
 func getAuthFilePaths(sys *types.SystemContext, homeDir string) []authPath {
+	runningInSystemd := os.Getenv("INVOCATION_ID") != ""
+	runningAsRoot := os.Getuid() == 0


At a first glance it seems that this should use rootless.GetRootlessEUID. (Admittedly the rest of this file doesn‘t, and it probably should as well…)

pkg/docker/config/config.go

mtrmac · 2022-12-08T18:19:08Z

pkg/docker/config/config.go

+		}
+		perms := st.Mode().Perm()
+		if (perms & 04) > 0 {
+			return dockerConfigFile{}, fmt.Errorf("refusing to process %s with world read permissions", path)


The documentation of the field says “readable by group or other”, the man page says “mode 0600”.

I’m not sure what we want[1] but it should be consistent.

[1] Starting with 0600 would allow us to possibly relax the restriction later, so that seems to be the conservative choice. Also, we are only reading the file when the user is root, so no other permissions seem necessary.

For now I chose to error on group/world read permissions. Forcing 0600 also forces write permissions, but I can imagine cases where one might not want that. (Not that it really matters because root usually has CAP_DAC_OVERRIDE)

mtrmac · 2022-12-08T18:20:17Z

pkg/docker/config/config.go

@@ -204,7 +208,17 @@ func GetAllCredentials(sys *types.SystemContext) (map[string]types.DockerAuthCon
 // The homeDir parameter should always be homedir.Get(), and is only intended to be overridden
 // by tests.
 func getAuthFilePaths(sys *types.SystemContext, homeDir string) []authPath {


With growing complexity, it seems more important to have unit tests for this. But that’s the very last step.

cgwalters · 2022-12-08T18:53:32Z

OTOH does this mean we are giving up on some SELinux restrictions on root? The /etc/containers path is necessarily unrestricted. I didn’t look into how the Kubelet is, or isn’t restricted.

Good question. Today,

$ ll -Z /etc/containers/
total 12
drwxr-xr-x. 1 root root system_u:object_r:etc_t:s0        0 Oct  6 15:46 certs.d
drwx------. 1 root root unconfined_u:object_r:etc_t:s0   26 Oct 30 17:55 networks
drwxr-xr-x. 1 root root system_u:object_r:etc_t:s0       14 Oct  6 15:46 oci
-rw-r--r--. 1 root root system_u:object_r:etc_t:s0      569 Oct  6 15:46 policy.json
-rw-r--r--. 1 root root system_u:object_r:etc_t:s0     3897 Oct  6 15:46 registries.conf
drwxr-xr-x. 1 root root system_u:object_r:etc_t:s0       38 Oct  6 15:46 registries.conf.d
drwxr-xr-x. 1 root root system_u:object_r:etc_t:s0      132 Nov 17 19:53 registries.d
-rw-r--r--. 1 root root system_u:object_r:etc_t:s0      699 Dec  9  2021 toolbox.conf

It could make sense to introduce a dedicated container_auth_t or so. But we can do that even inside /etc/containers. Or, we could have it be in /etc/containers/secrets and label the directory for future expansion - and if we do that it'd make sense to have the dir also mode 0700 for example for stock DAC protections.

cgwalters · 2022-12-08T19:07:03Z

A variant of this is that we could:

Just add a change to the man page here which documents the possible existence of /etc/containers/auth.json
I change bootc to use it by default

This I think would get about 45% of the value with 1% of the work.

mtrmac · 2022-12-08T19:07:44Z

I’m not thinking so much whether we could isolate this in root with SELinux in principle, it’s more that we can’t do that effectively: once /etc/containers/…auth.json becomes the Podman default for root, all plausible SELinux contexts that could run Podman must be allowed to access that file. The only way to truly isolate is to have more isolated files, i.e. not the system-wide /etc/containers/*.

But that’s just me rehashing the previous discussion, I guess. It seems to me that single-purpose appliances can use /etc/containers, traditional multi-user multi-purpose UNIX servers shouldn’t set it up at all, and should continue to use whatever they are doing now.

mtrmac · 2022-12-08T19:10:15Z

A variant of this is that we could: …

Overall, so far I’m not convinced that adding this default path is clearly worthwhile. But it’s useful enough in some contexts that I’m fine with it being added.

cgwalters · 2022-12-08T19:20:11Z

I’m not thinking so much whether we could isolate this in root with SELinux in principle, it’s more that we can’t do that effectively: once /etc/containers/…auth.json becomes the Podman default for root, all plausible SELinux contexts that could run Podman must be allowed to access that file.

I think it is possible because today most invocations of podman are going to start from unconfined_t or init_t, and we'd be sure that those contexts are allowed to access the file. The value of the distinct context is that lesser-privileged domains (say NetworkManager_t) would not be able to read it by default.

But that’s just me rehashing the previous discussion, I guess. It seems to me that single-purpose appliances can use /etc/containers, traditional multi-user multi-purpose UNIX servers shouldn’t set it up at all, and should continue to use whatever they are doing now.

Well, we have to account for mixed setups. For example, a valid user story here is a company/organization deploying Linux desktop systems using bootc and having that system upgrade use one pull secret, which would be distinct from the secret they use (today) in $HOME or /root for their application containers.

So I see this as just adding a dedicated file for "the system" without conflicting with any optional setup.

Overall, so far I’m not convinced that adding this default path is clearly worthwhile. But it’s useful enough in some contexts that I’m fine with it being added.

Thanks; this is a useful debate to have for sure because once added, it can't be removed and so there's long term maintenance implications for sure.

mtrmac · 2022-12-09T20:47:50Z

I think it is possible because today most invocations of podman are going to start from unconfined_t or init_t, and we'd be sure that those contexts are allowed to access the file. The value of the distinct context is that lesser-privileged domains (say NetworkManager_t) would not be able to read it by default.

(BTW OpenShift, at least in some versions, runs Podman from inside a NetworkManager hook… it’s containers all the way down.)

Instead of packing and unpacking it into two variables in various places. Prep for adding a third member of this struct as part of containers#1746

cgwalters · 2022-12-15T22:31:18Z

At this point it seems useful to make readJSONFile, findCredentialsInFile (and other functions?) methods on authPath, instead of passing the individual parameters; ideally in a separate commit.

Done in #1765
(unit tests don't quite pass for me locally, but they fail in the same way without this. They seem to assume some global state)

Instead of packing and unpacking it into two variables in various places. Prep for adding a third member of this struct as part of containers#1746 Signed-off-by: Colin Walters <walters@verbum.org>

mtrmac · 2022-12-16T19:38:55Z

(unit tests don't quite pass for me locally, but they fail in the same way without this. They seem to assume some global state)

I wouldn’t be too surprised if they were insufficiently isolated from the per-user state. If so, that’s not intentional.

Instead of packing and unpacking it into two variables in various places. Prep for adding a third member of this struct as part of containers#1746 Signed-off-by: Colin Walters <walters@verbum.org>

A long-running tension in the docker/podman land is around running as a system service versus being executed by a user. (Specifically a "login user", i.e. a Unix user that can be logged into via `ssh` etc.) For login users, it makes total sense to configure the container runtime in `$HOME`. But for system services (e.g. code executed by systemd) it is generally a bad idea to access or read the `/root` home directory. On image based systems, `/root` may be dynamically mutable state in contrast to `/etc` which may be managed by OS upgrades, or even be read-only. For these reasons, let's introduce `/etc/contaners/auth.json`. If it is present, and the current process is executing in systemd, it will be preferred. (There's some further logic around this that is explained in the manpage; please see that for details) cc coreos/rpm-ostree#4180 Signed-off-by: Colin Walters <walters@verbum.org>

cgwalters · 2023-01-03T15:15:45Z

(BTW OpenShift, at least in some versions, runs Podman from inside a NetworkManager hook… it’s containers all the way down.)

Yeah true 😢

But an attempt to default SELinux confine NetworkManager hooks IIRC failed, and I think the status quo there is the default is unconfined.

cgwalters · 2023-01-03T15:16:20Z

OK I rebased 🏄 this - it's definitely cleaner on top of #1765

Fixed some comments and issues, but still more to do.

mtrmac · 2023-01-04T00:27:10Z

pkg/docker/config/config.go

+			return dockerConfigFile{}, fmt.Errorf("stat %s: %w", path.path, err)
+		}
+		perms := st.Mode().Perm()
+		if (perms & 044) > 0 {


!= 0 seems a better fit with the “bit set” format.

pkg/docker/config/config.go

cgwalters · 2023-02-17T15:13:06Z

We might also want to investigate https://systemd.io/CREDENTIALS/ here.

cgwalters · 2023-10-17T18:12:29Z

We might also want to investigate https://systemd.io/CREDENTIALS/ here.

OK yes I dug into this a bit more recently and I think it'd be nice to support this more natively. The credentials path is a bit orthogonal in a way to a plain old /etc/containers/auth.json, but it has a lot of nicer properties.

My proposal here is basically we document and support a flow like this:

systemd-creds encrypt --name='' - - < /path/to/docker.cfg > /etc/containers/system-auth.json

This is invoked by provisioning/configuration management tooling (and the choice of filename here is just a recommendation). Same as all other uses of systemd credentials.

Then anything which wants to read this (e.g. skopeo inspect/podman pull) needs to be invoked from a systemd unit, and set LoadCredentialEncrypted=containers-auth.json:/etc/containers/system-auth.json.

Now what we would do inside containers/image here is check whether $CREDENTIALS_DIRECTORY/containers-auth.json exists - if it does it is preferred over anything else.

Here's an example command I used to test this flow:

$ systemd-run -p LoadCredentialEncrypted=containers-auth.json:/etc/containers/system-auth.json -qPG /bin/sh -c 'podman pull --authfile $CREDENTIALS_DIRECTORY/containers-auth.json docker://registry.ci.openshift.org/ocp/release:4.15.0-0.nightly-2023-10-17-065657'

Except the benefit of supporting $CREDENTIALS_DIRECTORY/containers-auth.json automatically is exactly that it becomes very ergonomic to use! We encourage people to use credentials for this. Also it becomes easier to juggle multiple pull secrets because one can do e.g.:

systemd-creds encrypt --name="" - - < /path/to/other-docker.cfg > /etc/containers/system-auth-internalregistry.json

(Cleanly separating out a distinct pull secret)

Then when invoking systemd units, one would set LoadCredentialEncrypted=containers-auth.json:/etc/containers/system-auth-internalregistry.json and if we did the automatic binding, that would mean the "internalregistry" pull secret is automatically used for all invocations of podman/skopeo etc.

cgwalters · 2024-03-27T11:03:39Z

I recently came across https://access.redhat.com/solutions/7002142 which has a really bad suggestion for how to make podman login persist. I do think this PR still makes sense; will try to rebase at some point soon and pick this back up.

cgwalters mentioned this pull request Dec 8, 2022

Document how to authenticate to private container repository. coreos/rpm-ostree#4180

Closed

cgwalters requested a review from mtrmac December 8, 2022 16:30

mtrmac reviewed Dec 8, 2022

View reviewed changes

mtrmac added the kind/feature A request for, or a PR adding, new functionality label Dec 8, 2022

cgwalters added a commit to cgwalters/image that referenced this pull request Dec 15, 2022

config: use authPath struct consistently

79ea7c2

Instead of packing and unpacking it into two variables in various places. Prep for adding a third member of this struct as part of containers#1746

cgwalters mentioned this pull request Dec 15, 2022

config: use authPath struct consistently #1765

Merged

cgwalters force-pushed the add-system-path branch 2 times, most recently from 76b414f to 838b0f9 Compare January 3, 2023 15:08

cgwalters force-pushed the add-system-path branch from 838b0f9 to e998b2f Compare January 3, 2023 15:10

mtrmac reviewed Jan 4, 2023

View reviewed changes

cgwalters mentioned this pull request Sep 19, 2023

man: Add ostree-container-auth ostreedev/ostree-rs-ext#543

Merged

cgwalters mentioned this pull request Oct 17, 2023

switch to being a systemd service containers/bootc#2

Open

cgwalters mentioned this pull request Dec 4, 2023

WIP: Use podman pull to fetch containers containers/bootc#215

Open

cgwalters mentioned this pull request Mar 25, 2024

install: Add --copy-container-credentials containers/bootc#428

Open

cgwalters mentioned this pull request May 3, 2024

bootc e2e workflow - build temp image shim with ssh keys around bootc containers/ai-lab-recipes#428

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: Add a `/etc/containers/auth.json` #1746

WIP: Add a `/etc/containers/auth.json` #1746

cgwalters commented Dec 8, 2022

cgwalters commented Dec 8, 2022

cgwalters commented Dec 8, 2022

cgwalters commented Dec 8, 2022

mtrmac left a comment •

edited

mtrmac Dec 8, 2022

mtrmac Dec 8, 2022

mtrmac Dec 8, 2022

cgwalters Jan 3, 2023

mtrmac Dec 8, 2022

cgwalters commented Dec 8, 2022

cgwalters commented Dec 8, 2022

mtrmac commented Dec 8, 2022 •

edited

mtrmac commented Dec 8, 2022

cgwalters commented Dec 8, 2022

mtrmac commented Dec 9, 2022

cgwalters commented Dec 15, 2022

mtrmac commented Dec 16, 2022 •

edited

cgwalters commented Jan 3, 2023

cgwalters commented Jan 3, 2023

mtrmac Jan 4, 2023

cgwalters commented Feb 17, 2023

cgwalters commented Oct 17, 2023

cgwalters commented Mar 27, 2024

WIP: Add a /etc/containers/auth.json #1746

Are you sure you want to change the base?

WIP: Add a /etc/containers/auth.json #1746

Conversation

cgwalters commented Dec 8, 2022

cgwalters commented Dec 8, 2022

cgwalters commented Dec 8, 2022

cgwalters commented Dec 8, 2022

mtrmac left a comment • edited

Choose a reason for hiding this comment

mtrmac Dec 8, 2022

Choose a reason for hiding this comment

mtrmac Dec 8, 2022

Choose a reason for hiding this comment

mtrmac Dec 8, 2022

Choose a reason for hiding this comment

cgwalters Jan 3, 2023

Choose a reason for hiding this comment

mtrmac Dec 8, 2022

Choose a reason for hiding this comment

cgwalters commented Dec 8, 2022

cgwalters commented Dec 8, 2022

mtrmac commented Dec 8, 2022 • edited

mtrmac commented Dec 8, 2022

cgwalters commented Dec 8, 2022

mtrmac commented Dec 9, 2022

cgwalters commented Dec 15, 2022

mtrmac commented Dec 16, 2022 • edited

cgwalters commented Jan 3, 2023

cgwalters commented Jan 3, 2023

mtrmac Jan 4, 2023

Choose a reason for hiding this comment

cgwalters commented Feb 17, 2023

cgwalters commented Oct 17, 2023

cgwalters commented Mar 27, 2024

WIP: Add a `/etc/containers/auth.json` #1746

WIP: Add a `/etc/containers/auth.json` #1746

mtrmac left a comment •

edited

mtrmac commented Dec 8, 2022 •

edited

mtrmac commented Dec 16, 2022 •

edited