Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Detect rootless mode #1484

Closed
wants to merge 11 commits into from
Closed

Detect rootless mode #1484

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
7c16c3e
fix #1243 - the -u option is not necessary on podman
themr0c Jun 2, 2020
5c1cd81
the -u option is not necessary on a rootless environment
themr0c Jun 3, 2020
a8887d2
the -u option is not necessary on a rootless environment (rewrite by …
themr0c Jun 3, 2020
4416c3d
add AttributeError again
themr0c Jun 3, 2020
818a909
output before try
themr0c Jun 4, 2020
79528ec
Not just output, move this whole thing out.
themr0c Jun 4, 2020
b0fc24f
docker.py reworked, thanks @hroncok
themr0c Jun 15, 2020
da85e73
docker.py tests, thanks @hroncok
themr0c Jun 15, 2020
12a40fd
docker.py tests thanks @hroncok
themr0c Jun 15, 2020
dff7d60
docker.py tests thanks @hroncok
themr0c Jun 15, 2020
9d4fe3a
add pragma: win32 no cover
themr0c Jun 15, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
23 changes: 23 additions & 0 deletions pre_commit/languages/docker.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import functools
import hashlib
import os
from typing import Sequence
Expand All @@ -9,6 +10,7 @@
from pre_commit.prefix import Prefix
from pre_commit.util import CalledProcessError
from pre_commit.util import clean_path_on_failure
from pre_commit.util import cmd_output
from pre_commit.util import cmd_output_b

ENVIRONMENT_DIR = 'docker'
Expand Down Expand Up @@ -76,7 +78,28 @@ def install_environment(
os.mkdir(directory)


def _docker_is_rootless() -> bool: # pragma: win32 no cover
returncode, stdout, stderr = cmd_output(
'docker', 'system', 'info',
)
Comment on lines +82 to +84
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be even faster and simplify both the logic below and the mock output used in testing to only query Docker/Podman for the field we care about. From the rootless-docker GitHub Action's README, docker info --format "{{ .ClientInfo.Context }}" outputs rootless in Docker when in rootless mode, and other strings (most commonly default, I believe) otherwise. I suspect, based on Podman's documentation, that podman info --format "{{ .host.security.rootless }}" outputs true when in rootless mode and false otherwise. Go templates are quite powerful, so we can say: docker info --format '{{ if .ClientInfo.Context }} {{ eq .ClientInfo.Context "rootless" }} {{ else }} {{ .host.security.rootless }} {{ end }}'. This command outputs true when running in rootless mode and false otherwise in both Docker and Podman, although I have only tested it on Docker. I don't have Podman installed, so hopefully someone who does can straighten me out if I'm mistaken. Disclosure: I am the author of the rootless-docker GitHub Action.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here on my machine (Linux Mint 19.3, Docker version 20.10.16, build aa7e414) your suggestion returns default for rootless docker, as rootless seems to be part of the Security Options under Server.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for letting me know. Bummer that the behavior isn't consistent across our machines even on the exact same build of Docker. I am on Ubuntu 22.04, but I wonder whether the difference has more to do with how we installed rootless Docker? It's possible to detect rootless mode under SecurityOptions too (e.g., docker info --format '{{ range .SecurityOptions }}{{ if eq . "name=rootless" }}true{{ end }}{{ end }}'), FWIW.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIRC I followed the official instructions for Ubuntu. Maybe you chose to set the client context explicitly where I went for the DOCKER_HOST environment variable? Both options are listed in the instructions: https://docs.docker.com/engine/security/rootless/#client

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, you are correct. I suppose one could check the DOCKER_HOST environment variable before running docker info.

for line in stdout.splitlines():
# rootless docker has "rootless"
# rootless podman has "rootless: true"
if line.strip().startswith('rootless'):
if 'false' not in line:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: Technically, the case where false is present is not tested. However in practice, we haven't found a case like this, this check is present as precaution.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would be useful as a code comment

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is that what is causing the coverage error?

Name                             Stmts   Miss Branch BrPart  Cover   Missing
----------------------------------------------------------------------------
pre_commit/languages/docker.py      72      2     10      2    95%   89->91, 91, 101->102, 102

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is weird. If tests include example outputs containing rootless, they should also hit this condition check.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@asottile any ideas about this?

Copy link
Member

@asottile asottile Jul 3, 2020 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there's no test for rootless: false so line 91 is never hit -- perhaps simpler would be if line.strip() == 'rootless: true': ...

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That would probably not catch docker's behavior. Better to just improve the test matrix

return True
break
themr0c marked this conversation as resolved.
Show resolved Hide resolved
return False


@functools.lru_cache(maxsize=1)
def docker_is_rootless() -> bool: # pragma: win32 no cover
return _docker_is_rootless()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: We had to do this to be able to test this with multiple mocked outputs. The cached version made that very hard.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could also be something like docker_is_rootless = functools.lru_cache(maxsize=1)(_docker_is_rootless)

Copy link
Contributor

@hroncok hroncok Jun 15, 2020 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could. Not sure if more or less readable.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there's an example of testing the inner part of an lru_cache function in pre_commit/languages/node.py (tests/languages/node_test.py) -- basically, access the __wrapped__ attribute of the cached object in the tests, then you don't need this indirection

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@themr0c @hroncok this will need to be addressed ^



def get_docker_user() -> Tuple[str, ...]: # pragma: win32 no cover
if docker_is_rootless():
return ()
try:
return ('-u', f'{os.getuid()}:{os.getgid()}')
except AttributeError:
Expand Down
206 changes: 206 additions & 0 deletions tests/languages/docker_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,188 @@
from pre_commit.languages import docker
from pre_commit.util import CalledProcessError

DOCKER_NOT_ROOTLESS_SYSTEM_INFO = '''
Client:
Debug Mode: false

Server:
Containers: 8
Running: 0
Paused: 0
Stopped: 8
Images: 2
Server Version: 19.03.8
Storage Driver: overlay2
Backing Filesystem: <unknown>
Supports d_type: true
Native Overlay Diff: true
Logging Driver: journald
Cgroup Driver: systemd
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: /usr/libexec/docker/docker-init
containerd version:
runc version: fbdbaf85ecbc0e077f336c03062710435607dbf1
init version:
Security Options:
seccomp
Profile: default
selinux
Kernel Version: 5.6.14-300.fc32.x86_64
Operating System: Fedora 32 (Thirty Two)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.29GiB
Name: carbon
ID: IUGE:4L5B:VVTV:JGIA:IIJN:G7MG:TVGF:XBVW:YHF7:MYRK:L524:4HBK
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: true
''' # noqa
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's usually better to specify specific violation codes instead of ignoring everything


DOCKER_ROOTLESS_SYSTEM_INFO = '''
Client:
Debug Mode: false

Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 19.03.11
Storage Driver: vfs
Logging Driver: json-file
Cgroup Driver: none
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
init version: fec3683
Security Options:
seccomp
Profile: default
rootless
Kernel Version: 5.6.16-300.fc32.x86_64
Operating System: Fedora 32 (Workstation Edition)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 23.36GiB
Name: laptop.centsix.taz
ID: 6EMS:RSJ4:NXM3:3D56:CV4N:VEUX:LBFM:KWHK:P7XA:HTCS:GVSC:UVUE
Docker Root Dir: /home/remote/user/.local/share/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: true
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine
''' # noqa

PODMAN_SYSTEM_INFO = '''
host:
arch: amd64
buildahVersion: 1.14.9
cgroupVersion: v2
conmon:
package: conmon-2.0.17-1.fc32.x86_64
path: /usr/libexec/crio/conmon
version: 'conmon version 2.0.17, commit: bb8e273f5925c1a51737644637ef65d094a67ab1'
cpus: 8
distribution:
distribution: fedora
version: "32"
eventLogger: file
hostname: laptop.centsix.taz
idMappings:
gidmap:
- container_id: 0
host_id: 1001
size: 1
- container_id: 1
host_id: 200000
size: 65536
uidmap:
- container_id: 0
host_id: 105971
size: 1
- container_id: 1
host_id: 200000
size: 65536
kernel: 5.6.16-300.fc32.x86_64
memFree: 7256297472
memTotal: 25081421824
ociRuntime:
name: crun
package: crun-0.13-2.fc32.x86_64
path: /usr/bin/crun
version: |-
crun version 0.13
commit: e79e4de4ac16da0ce48777afb72c6241de870525
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
os: linux
rootless: true
slirp4netns:
executable: /usr/bin/slirp4netns
package: slirp4netns-1.0.0-1.fc32.x86_64
version: |-
slirp4netns version 1.0.0
commit: a3be729152a33e692cd28b52f664defbf2e7810a
libslirp: 4.2.0
swapFree: 20372766720
swapTotal: 20372766720
uptime: 4h 46m 1.38s (Approximately 0.17 days)
registries: {}
store:
configFile: /home/remote/user/.config/containers/storage.conf
containerStore:
number: 66
paused: 0
running: 0
stopped: 66
graphDriverName: overlay
graphOptions:
overlay.mount_program:
Executable: /usr/bin/fuse-overlayfs
Package: fuse-overlayfs-1.0.0-1.fc32.x86_64
Version: |-
fusermount3 version: 3.9.1
fuse-overlayfs: version 1.0.0
FUSE library version 3.9.1
using FUSE kernel interface version 7.31
graphRoot: /home/remote/user/.local/share/containers/storage
graphStatus:
Backing Filesystem: extfs
Native Overlay Diff: "false"
Supports d_type: "true"
Using metacopy: "false"
imageStore:
number: 23
runRoot: /tmp/105971
volumePath: /home/remote/user/.local/share/containers/storage/volumes
''' # noqa


def test_docker_is_running_process_error():
with mock.patch(
Expand All @@ -21,3 +203,27 @@ def invalid_attribute():
getgid=invalid_attribute,
):
assert docker.get_docker_user() == ()


def test_docker_is_not_rootless():
with mock.patch.object(
docker, 'cmd_output',
return_value=(0, DOCKER_NOT_ROOTLESS_SYSTEM_INFO, ''),
):
assert docker._docker_is_rootless() is False


def test_docker_is_rootless():
with mock.patch.object(
docker, 'cmd_output',
return_value=(0, DOCKER_ROOTLESS_SYSTEM_INFO, ''),
):
assert docker._docker_is_rootless() is True


def test_podman_is_rootless():
with mock.patch.object(
docker, 'cmd_output',
return_value=(0, PODMAN_SYSTEM_INFO, ''),
):
assert docker._docker_is_rootless() is True