Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Mount the stdout callback plugin for containerized runs #957

Merged
merged 14 commits into from Feb 1, 2022
Merged

Mount the stdout callback plugin for containerized runs #957

Show file tree
Hide file tree
Changes from 12 commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
b01fca4
Support custom stdout callbacks
shanemcd Jul 30, 2021
6b83c6b
Refactor things down to a single stdout callback module
shanemcd Jul 31, 2021
0a705c2
Condense callback to 1 class, enable in BaseConfig
AlanCoding Jan 7, 2022
837e83e
Switch test to stdout callback in ansible.builtin
AlanCoding Jan 11, 2022
f098e00
Use correct metric for IS_ADHOC determination
AlanCoding Jan 11, 2022
49026da
Accomidate other config classes
AlanCoding Jan 11, 2022
d7eb76a
Remove test hack
AlanCoding Jan 12, 2022
70be623
Code move for single-file display callback
AlanCoding Jan 12, 2022
ee15e0b
Remove noqa comments with config change
AlanCoding Jan 12, 2022
dd8349f
Move display callback to required name
AlanCoding Jan 12, 2022
55d716c
Code and test changes for callback volume mount
AlanCoding Jan 13, 2022
851f4d3
Change mount location and introduce shared method
AlanCoding Jan 17, 2022
df44cea
Workaround when install is not owned by current user
AlanCoding Jan 27, 2022
3c3a461
Add test to simulate non-root container run
AlanCoding Jan 28, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
50 changes: 0 additions & 50 deletions ansible_runner/callbacks/awx_display.py
View file
Open in desktop

This file was deleted.

50 changes: 0 additions & 50 deletions ansible_runner/callbacks/minimal.py
View file
Open in desktop

This file was deleted.

43 changes: 23 additions & 20 deletions ansible_runner/config/_base.py
View file
Open in desktop
Expand Up @@ -39,6 +39,8 @@
from ansible_runner.defaults import registry_auth_prefix
from ansible_runner.loader import ArtifactLoader
from ansible_runner.utils import (
callback_mount,
get_callback_dir,
open_fifo_write,
args2cmdline,
sanitize_container_name,
Expand Down Expand Up @@ -199,8 +201,6 @@ def _prepare_env(self, runner_mode='pexpect'):
if self.containerized:
self.container_name = "ansible_runner_{}".format(sanitize_container_name(self.ident))
self.env = {}
# Special flags to convey info to entrypoint or process in container
self.env['LAUNCHED_BY_RUNNER'] = '1'

if self.process_isolation_executable == 'podman':
# A kernel bug in RHEL < 8.5 causes podman to use the fuse-overlayfs driver. This results in errors when
Expand Down Expand Up @@ -258,17 +258,23 @@ def _prepare_env(self, runner_mode='pexpect'):
if not self.containerized:
callback_dir = self.env.get('AWX_LIB_DIRECTORY', os.getenv('AWX_LIB_DIRECTORY'))
if callback_dir is None:
callback_dir = os.path.join(os.path.split(os.path.abspath(__file__))[0], "..", "callbacks")
python_path = self.env.get('PYTHONPATH', os.getenv('PYTHONPATH', ''))
self.env['PYTHONPATH'] = ':'.join([python_path, callback_dir])
if python_path and not python_path.endswith(':'):
python_path += ':'
callback_dir = get_callback_dir()
self.env['ANSIBLE_CALLBACK_PLUGINS'] = ':'.join(filter(None, (self.env.get('ANSIBLE_CALLBACK_PLUGINS'), callback_dir)))

if 'AD_HOC_COMMAND_ID' in self.env:
self.env['ANSIBLE_STDOUT_CALLBACK'] = 'minimal'
else:
self.env['ANSIBLE_STDOUT_CALLBACK'] = 'awx_display'
# this is an adhoc command if the module is specified, TODO: combine with logic in RunnerConfig class
is_adhoc = bool((getattr(self, 'binary', None) is None) and (getattr(self, 'module', None) is not None))

if self.env.get('ANSIBLE_STDOUT_CALLBACK'):
self.env['ORIGINAL_STDOUT_CALLBACK'] = self.env.get('ANSIBLE_STDOUT_CALLBACK')

if is_adhoc:
# force loading awx_display stdout callback for adhoc commands
self.env["ANSIBLE_LOAD_CALLBACK_PLUGINS"] = '1'
if 'AD_HOC_COMMAND_ID' not in self.env:
self.env['AD_HOC_COMMAND_ID'] = '1'

self.env['ANSIBLE_STDOUT_CALLBACK'] = 'awx_display'

self.env['ANSIBLE_RETRY_FILES_ENABLED'] = 'False'
if 'ANSIBLE_HOST_KEY_CHECKING' not in self.env:
self.env['ANSIBLE_HOST_KEY_CHECKING'] = 'False'
Expand Down Expand Up @@ -479,20 +485,17 @@ def wrap_args_for_containerization(self, args, execution_mode, cmdline_args):
dst_mount_path="/runner/artifacts",
labels=":Z")

# Mount the entire private_data_dir
# custom show paths inside private_data_dir do not make sense
self._update_volume_mount_paths(new_args,
"{}".format(self.private_data_dir),
dst_mount_path="/runner",
labels=":Z")
else:
subdir_path = os.path.join(self.private_data_dir, 'artifacts')
if not os.path.exists(subdir_path):
os.mkdir(subdir_path, 0o700)

# Mount the entire private_data_dir
# custom show paths inside private_data_dir do not make sense
self._update_volume_mount_paths(new_args, "{}".format(self.private_data_dir), dst_mount_path="/runner", labels=":Z")
# Mount the entire private_data_dir
# custom show paths inside private_data_dir do not make sense
self._update_volume_mount_paths(new_args, "{}".format(self.private_data_dir), dst_mount_path="/runner", labels=":Z")

# Mount the stdout callback plugin from the ansible-runner code base
self._update_volume_mount_paths(new_args, callback_mount()[0], dst_mount_path=callback_mount()[1], labels=":Z")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this might break if you install runner as root in the system site packages and then run as a non-root user, because you won't be able to relabel a file you dont own.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe try :O instead of :Z?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, this sounds more right to me, but I haven't been able to figure out how to artificially create the error to validate it fixes it. I got it to these permissions:

bash-4.4$ ls -la /var/lib/awx/venv/awx/lib/python3.8/site-packages/ansible_runner/display_callback/callback/
total 44
drwxr-xr-x. 3 root root  4096 Jan 26 03:18 .
drwxr-xr-x. 3 root root  4096 Jan 26 03:18 ..
-rw-r--r--. 1 root root 30788 Jan 26 03:18 awx_display.py
-rw-r--r--. 1 root root     0 Jan 26 03:18 __init__.py
drwxr-xr-x. 2 root root  4096 Jan 26 03:18 __pycache__

Then as the awx user, it still works. Is there any tweak I can do which should make this break?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wait, never mind, I was running the wrong thing. I am able to get an error with this type of setup. Right now it just gives the vague message

ERROR! Invalid callback for stdout specified: awx_display

But I'm still tinkering with the install by root user.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Testing the :0 in one environment, I get

Error: mounting overlay failed "/var/lib/awx/venv/awx/lib64/python3.8/site-packages/ansible_runner/display_callback/callback/": chown /var/lib/awx/.local/share/containers/storage/overlay-containers/78dce4e01f5d25bbfe804246e232815dd809cb5360237b1fddf35bc49bfe2fca/userdata/overlay/568435025/upper: invalid argument

I'm not sure how big of a problem this is going to be. In another environment it works. Reading related issues, people mention that kernel version needs to be relatively recent for a metacopy setting. Depending on what we support, I could see this getting complicated.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, trying on my own machine, I was able to reproduce something by doing only this:

sudo chown -R root:root ansible_runner/display_callback/

Resulting in

$ ansible-runner run demo/ -p test.yml --process-isolation
Error: lsetxattr /home/alancoding/repos/ansible-runner/ansible_runner/display_callback/callback: operation not permitted

So maybe I am figuring this out. Here, specifically, with root ownership and -rw-rw-r--. permissions, the display callback is not editable for the current user. Like you say, that shouldn't be a problem. I fear that :O will not work in all the contexts that we need it to... but it wouldn't be atypical for runner to create a new tmpdir to make something work. We could do this with a copy of the display callback. We know it should only be a few files.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Having a replicator, I developed a patch that fixes it.

I did not use the :O, because I'm worried this won't work in all cases. Instead, I copied the files (possible if you only have read access) to a tmpdir, and mounted the tmpdir.

Tomorrow I can work on a test case for this logic.


if self.container_auth_data:
# Pull in the necessary registry auth info, if there is a container cred
Expand Down
24 changes: 0 additions & 24 deletions ansible_runner/display_callback/__init__.py
View file
Open in desktop

This file was deleted.

0 ansible_runner/callbacks/__init__.py → ...ner/display_callback/callback/__init__.py
View file
Open in desktop
File renamed without changes.