Dev tooling - Updates to ansible-runner propagate through the dev env

[chrismeyersfsu]

I'm a developer. I need to add a feature to ansible-runner and consume it in awx. How do I do this? Well, first let's node all the places that the ansible-runner code is in the dev environment.

1. dev container quay.io/awx/awx_devel /var/lib/awx/venv/awx/lib/python3.8/site-packages/ansible_runner
2. The EE quay.io/ansible/awx-ee in the dev container /usr/local/lib/python3.8/site-packages/ansible_runner
3. The EE quay.io/ansible/awx-ee in the execution node dev containers /usr/local/lib/python3.8/site-packages/ansible_runner

Depending on the feature, the developer may want to update ansible-runner code on 1 or more of the 3 locations noted above.

I can't think of a scenario where the developer would REQUIRE ansible-runner changes they make to ONLY effect 1 or 2 of the 3 locations listed above. Well, unless the system is in a recreated state and the intent is to NOT bounce services, but we can separate that from a code update.

We have multiple locations defined that need code updated, 3 to be exact and it would seem there is no requirement to update only a subset of those locations. To let's shoot for that. We need to build something that let's us easily propagate code changes to ansible-runner to the 3 locations listed above. We also would want to bounce any services or other things that need to be done to trigger the new code being used, for the developer. Keep in mind that a developer might want to selectively subvert the trigger process but we need not have first-class support.

Idea 1

I really like how we map the awx git repo into the container so that changes in awx code based are able to be picked up inside the dev container. uwsgi watches for code changes and restarts when code changes are detected. This causes a cascade of restarts of dependent supervisor jobs (i.e. dispatcher and callback receiver). Can we do something similar for ansible-runner code?

• put ansible-runner repo inside of your awx repo. This way, runner will be available to the container.
• Use eggs like we do with awx but for runner
• volume mounts all the way down

mkvirtualenv giggity --clear
workon giggity
(giggity) ➜  ansible-runner git:(devel) pip install ansible-runner
(giggity) ➜  ansible-runner git:(devel) pip freeze
ansible-runner==2.0.3
docutils==0.18
lockfile==0.12.2
pexpect==4.8.0
ptyprocess==0.7.0
python-daemon==2.3.0
PyYAML==6.0
six==1.16.0
(giggity) ➜  ansible-runner git:(devel) git clone git@github.com:ansible/ansible-runner.git
(giggity) ➜  ansible cd ansible-runner
(giggity) ➜  ansible-runner git:(devel) pip install -e .
(giggity) ➜  ansible-runner git:(devel) pip freeze
-e git+git@github.com:ansible/ansible-runner@ef3754747b3c0ee3bfe8be7fbd3060f2e7cfcbfa#egg=ansible_runner
docutils==0.18
lockfile==0.12.2
pexpect==4.8.0
ptyprocess==0.7.0
python-daemon==2.3.0
PyYAML==6.0
six==1.16.0

The flow shown above isn't going to work for us. It will work on the dev container but not on the EE IMAGES. If we run the commands against an EE container then only that container, until it finishes, will have the changes. Future EE's container runs will create themselves from the underlying image, which we never modified in the above flow.

I see two choices:

1. modify the image (i.e. rebuild it or do some fancy snapshot of a container that makes an image) each time we want to make an ansible-runner change (blahhhh)
2. such that we can map in ansible-runner and it will "just work". (yeahhhh)

In Controller, we emulate what pip install -e . does via the Makefile target awx-link

/awx_devel/setup.py egg_info_dev
 cp -f /tmp/awx.egg-link /var/lib/awx/venv/awx/lib/python$(PYTHON_VERSION)/site-packages/awx.egg-link

I think we should adopt a similar patterns as ^^ for ansible-runner. One different will be that ansible-runner is already installed in many locations, awx is not. From my little bit of tinkering, this is problematic. If you already have ansible-runner installed and you try and just lay down .egg-link file(s) the installed ansible-runner takes precedence.

[AlanCoding]

There are 2 parts to ansible-runner:

• the python package which is a daemon to ansible jobs
• the callback plugin which dispatches event data

The python package is not used for any jobs except for container group jobs. If you're addressing container group jobs, nothing you mentioned here will help.

If you're trying to pick up changes to the callback plugin, I will note that the last change (ansible/ansible-runner@02dfbca) was Nov 3, 2020, which is over a year ago now.

I'm really not a fan of adding any kind of mounting or magic to help update a package in the development environment, if that package will categorically never be used in the development environment. If you're concerned about mismatch between the ansible-runner version and the callback plugin version with podman isolation, then we could consider some new mechanic in ansible-runner to copy the host machine's plugin into the container. I will note, I have argued annoyingly that we should make this a collection, and I could see that factoring into the mechanic.

Right now, I'm also much more concerned about the development environment's ability to update receptor, which is sorely out of date at the moment.

[AlanCoding]

I'm coming around to doing a volume mapping for the stdout callback. I've been thinking hard on this, and after ansible/ansible-runner#763 is merged, I may opt to combine the 3 modules we have into 1 to make the volume mount viable. This wouldn't be specific to development.

[AlanCoding]

I am increasingly convinced that ansible/ansible-runner#957 is the right thing to do, and addresses several points here. So I'm marking this as in-progress.

[AlanCoding]

https://github.com/ansible/automation-platform-collection/pull/333 for a problem I didn't know we had.

Once I get through this muck I intent to look into mounting /awx_devel on the execution nodes in the dev environment, which was the original thought of the issue.

[AlanCoding]

Last part of this: #11608

[AlanCoding]

I have merged several things to get everything on the same ansible-runner install. The one open item from discussion on this matter was #11656 so that user's have full flexibility to install in-place from the folder of their choice.

Closing as finished.