Load ansible.cfg in each playbook dir #10398

bdoublet91 · 2021-06-08T20:12:01Z

ISSUE TYPE

Bug Report

SUMMARY

Hello,

I'm looking for a solution to load local ansible.cfg at root playbook_dir.
This is my architeture folder of playbooks:

ansible
├── deploy_manager
│ ├── ansible.cfg
│ ├── deploy_manager.yml
│ ├── environments
│ │ ├── demo
│ │ │ ├── group_vars
│ │ │ │ └── demo.yml
│ │ │ ├── inventory.yml
│ │ │ └── vars
│ │ │ └── vault.yml
│ │ ├── int
│ │ │ ├── group_vars
│ │ │ │ └── int.yml
│ │ │ ├── inventory.yml
│ │ │ └── vars
│ │ │ └── vault.yml
│ │ └── prod
│ │ ├── group_vars
│ │ │ └── prod.yml
│ │ ├── inventory.yml
│ │ └── vars
│ │ └── vault.yml
│ ├── README.md
│ └── roles
│ ├── create_instance
│ │ └── tasks
│ │ └── main.yml

When I execute the playbook with ansible-playbook cli, I have an ansible.cfg in current dir so ansible.cfg is loaded.

When I execute the playbook from AWX, the project is in tmp/cev039fj/awx_1900_tw78u5vh/project.
There is no ansible.cfg in /tmp/ cev039fj/awx_1900_tw78u5vh so it's the /etc/ansible/ansible.cfg which is loaded.
I have an ansible.cfg in each playbook directory with different params so how could I setup the ANSIBLE_CONFIG path to playbook dir ansible.cfg when a playbook is launched by AWX ?

I did some tests unsuccessful with ANSIBLE_CONFIG setup.

Have you any ideas ?

ENVIRONMENT

AWX version: 17.1.0
AWX install methoddocker on linux
Ansible version: 2.9.0
Operating System: Centos 8
Web Browser: Firefox

AlanCoding · 2021-06-09T23:30:31Z

When I execute the playbook from AWX, the project is in tmp/cev039fj/awx_1900_tw78u5vh/project

That's right, so the top-level config of a project is used in job runs.

This is a known pain point. If you have any ideas for a natural way to configure a different config, I would like to hear them.

bdoublet91 · 2021-06-10T13:09:42Z

To sum up, I have this architecture and I need to load specific ansible.cfg when I execute deploy_manager.yml or deploy_manager2.yml
ansible
├── deploy_manager
│ ├── ansible.cfg
│ ├── deploy_manager.yml
│ ├── environments
│ └── roles
│ ├── create_instance
│ │ └── tasks
│ │ └── main.yml
├── deploy_manager_2
│ ├── ansible.cfg
│ ├── deploy_manager_2.yml
│ ├── environments
│ └── roles
│ ├── create_instance_2
│ │ └── tasks
│ │ └── main.yml

When I execute the playbook from the ansible-playbook cli, this is the ansible.cfg in current directory which is loaded according to the documentation:

ANSIBLE_CONFIG (environment variable if set)
ansible.cfg (in the current directory)
~/.ansible.cfg (in the home directory)
/etc/ansible/ansible.cfg

Now when I setup a template in AWX, the current directory is not the playbook directory but the top level project with no ansible.cfg or a global ansible.cfg.
I think a solution could be to check ansible.cfg at playbook path in template configuration.
Usually when I execute a playbook, I cd to the folder of the playbook so the ansible.cfg is in the current directory.

I don't know how difficult it could be to add this check ?

Another option could be to set the ANSIBLE_CONFIG to a variable set by AWX at job execution. I saw AWX_PRIVATE_DATA_DIR set to "tmp/cev039fj/awx_1900_tw78u5vh/project" but maybe there is another variable which contains playbook path ?
But I'm not sure if AWX checks ANSIBLE_CONFIG according to the documentation:
https://docs.ansible.com/ansible-tower/latest/html/administration/tipsandtricks.html#locate-and-configure-the-ansible-configuration-file ?

What do you think about it ?

AlanCoding · 2021-06-10T15:03:13Z

Another option could be to set the ANSIBLE_CONFIG to a variable set by AWX at job execution.

Yes, that's valid. Such a flag doesn't seem like a good fit for job templates to me. Why would you have a playbook in directory A and then use a config in directory B? Maybe this is common and I just don't get it.

I think a solution could be to check ansible.cfg at playbook path in template configuration.
Usually when I execute a playbook, I cd to the folder of the playbook so the ansible.cfg is in the current directory.

Yeah, I get it. Implementation isn't difficult. This could be done by either setting the working directory or ANSIBLE_CONFIG, as you mention. We just don't want the behavior to be a surprise.

bdoublet91 · 2021-06-10T18:58:41Z

Yes, that's valid. Such a flag doesn't seem like a good fit for job templates to me. Why would you have a playbook in directory A and then use a config in directory B? Maybe this is common and I just don't get it.

Yeah I understand your point. Not really usefull for other case just to load a specific ansible.cfg in playbook path.

Yeah, I get it. Implementation isn't difficult. This could be done by either setting the working directory or ANSIBLE_CONFIG, as you mention. We just don't want the behavior to be a surprise.

With working directory, you mean add a --playbok-dir arg ? or this is when AWX execute the playbook ? Do you have a reason to execute the playbook at the top level ?

Add a location for ANSIBLE_CONFIG is less destructive because if there is no, ansible will just check /etc/ansible/ansible.cfg
Maybe just add a way in the documentation to set up correctly ANSIBLE_CONFIG to have a dynamic path that match the template_path when a job is executed if possible ?

For example:
export ANSIBLE_CONFIG=$AWX_PLAYBOOK_PATH
I run deploy_manager.yml so AWX_PLAYBOOK_PATH is equal to absolute path deploy_manager directory

bdoublet91 · 2021-06-10T21:15:55Z

BTW, I have setup an ansibe.cfg at the root project, it gets loaded.
But now I have a problem about path for my plugins.
This is my archi folder:

ansible.cfg
callback_plugins
ansible
├── deploy_manager
│ ├── ansible.cfg
│ ├── deploy_manager.yml
│ ├── environments
│ └── roles
│ ├── create_instance
│ │ └── tasks
│ │ └── main.yml

and my ansible.cfg config:

[defaults]
callback_whitelist = custom
callback_plugins=./callback_plugins
stdout_callback = custom
gather_subset=!hardware,!ohai,!facte
interpreter_python=/usr/bin/python3

[persistent_connection]
retries = 3
connect_timeout = 60
command_timeout = 30

I have the callback_plugins folder next to ansible.cfg but when I execute the job, I have

ansible-playbook 2.9.18
  config file = /tmp/bwrap_2251_n3ryhpmf/awx_2251_o0ens3eg/project/ansible.cfg
  configured module search path = ['/var/lib/awx/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.6/site-packages/ansible
  executable location = /usr/bin/ansible-playbook
  python version = 3.6.8 (default, Aug 24 2020, 17:57:11) [GCC 8.3.1 20191121 (Red Hat 8.3.1-5)]
Using /tmp/bwrap_2251_n3ryhpmf/awx_2251_o0ens3eg/project/ansible.cfg as config file
Vault password: 
host_list declined parsing /tmp/bwrap_2251_n3ryhpmf/awx_2251_o0ens3eg/tmpsihhuqdm as it did not pass its verify_file() method
Parsed /tmp/bwrap_2251_n3ryhpmf/awx_2251_o0ens3eg/tmpsihhuqdm inventory source with script plugin
[WARNING]: Skipping callback plugin 'custom', unable to load

What I did wrong ? -> EDIT :(

Ansible Tower does not support the stdout callback plugin because Ansible only allows one, and it is already being used by Ansible Tower for streaming event data.

All I want was to load a plugin that print the stdout shell log in real time so I have to edit stdout callback ahah ^^
it worked with ansible-playbook cli but I think it won't with AWX
Just wait for ansible that lets us the possibility to print the log in real time (some issues exist)

However, the plugin should have loaded because I set it in the ansible.cfg

To have most callbacks shipped with Ansible applied globally, you must add them to the callback_whitelist section of your ansible.cfg. If you have a custom callbacks, refer to the Ansible documentation for Enabling callback plugins.

Another way is with ansible callback plugin settings but there is an issue
#4149

A good point could be to create a documentation about how to migrate ansible architecture folder to awx architecture folder because there are a lof of things that ansible is working but awx not ^^
Or a best practice of archi folder that meet AWX and ansible-playbook cli requirements.

wenottingham · 2021-06-11T17:22:10Z

plugin directories (and other configuration paths) are always relative to the playbook itself, not to the ansible.cfg location. Imagine the configuration being set in /etc/ansible/ansible.cfg - relative to that wouldn't make sense.

bdoublet91 · 2021-06-11T20:34:30Z

Yeah I understand your point of view for /etc/ansible/ansibe.cfg which is the default PATH. Now when I config a specific ansible.cfg file, I expect that all relative path configure in it which start from ansible.cfg path.
For example, when I have a bash script and I source an env file with relative path like source ./.env, I expect to have an .env file next to my script bash. That's why I though ansible worked like that.

But nevermind, Change stdout callback is not possible. I learned some point about how awx works so it's cool.
Just a last question, do you have an idea about how could I print log of a shell module every 5 seconds in awx stdout ?

Thanks

AlanCoding · 2021-06-22T13:52:43Z

[defaults]
callback_whitelist = custom
callback_plugins=./callback_plugins
stdout_callback = custom
gather_subset=!hardware,!ohai,!facte
interpreter_python=/usr/bin/python3

Okay, I know that ansible-runner won't respect your callback_plugins setting here. It tries to respect the user settings for callback plugins, and append its own to the end of the list.

https://github.com/ansible/ansible-runner/blob/4926a6afda13e768f55010f46509d2888e83e9a7/ansible_runner/config/_base.py#L242

But the issue is that you didn't set the env var ANSIBLE_CALLBACK_PLUGINS. You set the config file setting for the same thing. To combine with that, it would have to read the config file. It doesn't.

However, AWX does read the config file in cases, so that it doesn't clobber the user settings. This is the wrong place to do it. That needs to be moved into ansible-runner so that it can take responsibility for properly over-riding existing user settings. Or else, Ansible core needs to add some syntax to denote "existing values of a list-valued setting".

There is 1 other way we could fix this - we could put the ansible-runner callback plugin in a collection in an expected location. See ansible/ansible-runner#482. This avoids the need for ansible-runner to change the ANSIBLE_CALLBACK_PLUGINS setting at all and it just references the standard out callback plugin. However, it would still need to set the ANSIBLE_STDOUT_CALLBACK setting.

You are also setting the stdout callback setting. This doesn't make a lot of sense to me. There can only be 1 standard out callback plugin, because that's what dictates what gets written to standard out. I toyed with ideas for layering them, but they would step on each other's toes. Instead, I think you should consider doing your plugin as a "normal" callback plugin instead of a stdout callback plugin. If you do this, you can still enable it by changing the AWX_TASK_ENV setting without ansible-runner clobbering it. You can't do that with the config file - and that's our bug.

bdoublet91 · 2021-09-16T10:57:16Z

OK thanks for your explanation.
I learnt a lot !!

I close

keliansb · 2021-10-19T08:36:11Z

Hi all,

I want to achieve the same thing, load ansible.cfg from the playbook directory, but not from the project directory, as it breaks my other playbooks. I want a specific config file for a specific playbook to be able to preserve the type of variables when using Jinja2 (see this Stack Overflow thread for the related issue). My ansible.cfg is as follows :

[defaults]
jinja2_native = true

From what I understand from your discussion, I can be able to do that by setting an environment variable at playbook execution, but it does not seem to be working. Here is my test playbook :

- hosts: localhost
  tasks:
    - shell: "printenv PWD"
      register: pwd

    # The task which needs the custom ansible.cfg
    # ANSIBLE_CONFIG contains for example /tmp/awx_76572_5ineair_/project/<playbook_name>/ansible.cfg
    - uri:
        [...]
      environment:
        ANSIBLE_CONFIG: "{{ pwd.stdout }}/ansible.cfg"

P.S. : I also tried to set :

environment:
  ANSIBLE_JINJA2_NATIVE: True

on the uri: task but it does not work either.

If someone has a solution it would be great !

achebib · 2022-02-18T14:18:25Z

Did you get a solution for this?

keliansb · 2022-02-18T20:01:05Z

Unfortunately no...

bdoublet91 · 2022-02-21T11:12:45Z

I didn't find a way for AWX, I have only one ansible.cfg in project root folder and I keep each ansible.cfg in playbook folder when I execute ansible in CLI.
AWX version 17.1.0

achebib · 2022-02-21T15:00:19Z

But let me explain the issue.
I have an autocontained folder where all the variables are there (included ansible.cfg) how can I parse that when it runs the playbook?

#playbooks
#└── roles

└── web

├── README.md

├── defaults

│ └── main.yml

├── files

├── handlers

│ └── main.yml

├── meta

│ └── main.yml

├── tasks

│ └── main.yml

├── templates

├── tests

│ ├── inventory

│ └── test.yml

└── vars

└── main.yml

However in the job task

ansible-playbook 2.9.18
config file = /tmp/bwrap_11_nxntksfy/awx_11_x93anpiw/project/ansible.cfg
configured module search path = ['/var/lib/awx/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /usr/lib/python3.6/site-packages/ansible
executable location = /usr/bin/ansible-playbook
python version = 3.6.8 (default, Aug 24 2020, 17:57:11) [GCC 8.3.1 20191121 (Red Hat 8.3.1-5)]
Using /tmp/bwrap_11_nxntksfy/awx_11_x93anpiw/project/ansible.cfg as config file

awxbot added the type:bug label Jun 8, 2021

wenottingham added type:enhancement and removed type:bug labels Jun 11, 2021

bdoublet91 closed this as completed Sep 16, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Load ansible.cfg in each playbook dir #10398

Load ansible.cfg in each playbook dir #10398

bdoublet91 commented Jun 8, 2021

AlanCoding commented Jun 9, 2021

bdoublet91 commented Jun 10, 2021

AlanCoding commented Jun 10, 2021

bdoublet91 commented Jun 10, 2021

bdoublet91 commented Jun 10, 2021 •

edited

wenottingham commented Jun 11, 2021

bdoublet91 commented Jun 11, 2021 •

edited

AlanCoding commented Jun 22, 2021

bdoublet91 commented Sep 16, 2021

keliansb commented Oct 19, 2021

achebib commented Feb 18, 2022

keliansb commented Feb 18, 2022

bdoublet91 commented Feb 21, 2022

achebib commented Feb 21, 2022

Load ansible.cfg in each playbook dir #10398

Load ansible.cfg in each playbook dir #10398

Comments

bdoublet91 commented Jun 8, 2021

ISSUE TYPE

SUMMARY

ENVIRONMENT

AlanCoding commented Jun 9, 2021

bdoublet91 commented Jun 10, 2021

AlanCoding commented Jun 10, 2021

bdoublet91 commented Jun 10, 2021

bdoublet91 commented Jun 10, 2021 • edited

wenottingham commented Jun 11, 2021

bdoublet91 commented Jun 11, 2021 • edited

AlanCoding commented Jun 22, 2021

bdoublet91 commented Sep 16, 2021

keliansb commented Oct 19, 2021

achebib commented Feb 18, 2022

keliansb commented Feb 18, 2022

bdoublet91 commented Feb 21, 2022

achebib commented Feb 21, 2022

└── web

├── README.md

├── defaults

│ └── main.yml

├── files

├── handlers

│ └── main.yml

├── meta

│ └── main.yml

├── tasks

│ └── main.yml

├── templates

├── tests

│ ├── inventory

│ └── test.yml

└── vars

└── main.yml

bdoublet91 commented Jun 10, 2021 •

edited

bdoublet91 commented Jun 11, 2021 •

edited