Unable to find pip in the virtualenv (python 3.10)

[saltydk]

Summary

When running the following task (which worked fine on Ubuntu 20.04 which had older versions of both virtualenv and python)

- name: Install pip requirements
  ansible.builtin.pip:
    requirements: "/some/full/path/to/file"
    virtualenv_python: python3
    virtualenv: "/path/to/venv"
  become: true
  become_user: "{{ user.name }}"

returns an error like:

fatal: [localhost]: FAILED! => {"changed": false, "msg": "Unable to find pip in the virtualenv, /opt/cloudplow/venv, under any of these names: pip3, pip. Make sure pip is present in the virtualenv."}

When comparing the venv structure you get (Ubuntu 20.04 - Python 3.8 - Virtualenv 20.0.17):

/opt/script/venv/
├── bin
└── lib
  └── python3.8

Whereas on Ubuntu 22.04 (Python 3.10 - Virtualenv 20.13.0+ds) you get:

/opt/script/venv/
├── lib
│   └── python3.10
└── local
   ├── bin
   └── lib

Which, guessing here, is likely why it cannot find pip since the paths have changed.

Issue Type

Bug Report

Component Name

pip

Ansible Version

$ ansible --version
ansible [core 2.12.4]
  config file = None
  configured module search path = ['/home/salty/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.10/dist-packages/ansible
  ansible collection location = /home/salty/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible
  python version = 3.10.4 (main, Apr  2 2022, 09:04:19) [GCC 11.2.0]
  jinja version = 3.1.1
  libyaml = True

Configuration

$ ansible-config dump --only-changed
ANSIBLE_FORCE_COLOR(/srv/git/saltbox/ansible.cfg) = True
CALLBACKS_ENABLED(/srv/git/saltbox/ansible.cfg) = ['profile_tasks']
DEFAULT_FACT_PATH(/srv/git/saltbox/ansible.cfg) = ./ansible_facts.d
DEFAULT_HOST_LIST(/srv/git/saltbox/ansible.cfg) = ['/srv/git/saltbox/inventories/local']
DEFAULT_LOG_PATH(/srv/git/saltbox/ansible.cfg) = /srv/git/saltbox/saltbox.log
DEFAULT_ROLES_PATH(/srv/git/saltbox/ansible.cfg) = ['/srv/git/saltbox/roles', '/srv/git/saltbox/resources/roles']
INTERPRETER_PYTHON(/srv/git/saltbox/ansible.cfg) = /usr/bin/python3

OS / Environment

Ubuntu 22.04

Steps to Reproduce

- name: Install pip requirements
  ansible.builtin.pip:
    requirements: "/some/full/path/to/file"
    virtualenv_python: python3
    virtualenv: "/path/to/venv"
  become: true
  become_user: "{{ user.name }}"

Expected Results

Expected outcome is that pip binary is found and it installs the given requirements.

Actual Results

fatal: [localhost]: FAILED! => {"changed": false, "msg": "Unable to find pip in the virtualenv, /opt/script/venv, under any of these names: pip3, pip. Make sure pip is present in the virtualenv."}

Code of Conduct

• I agree to follow the Ansible Code of Conduct

[ansibot]

Files identified in the description:

• lib/ansible/modules/pip.py

If these files are incorrect, please update the component name section of the description or use the !component bot command.

click here for bot help

[bcoca]

We cannot reproduce this locally with the information given, it just works as expected. Is there any other factor that we are missing to trigger the issue?

needs_info

[saltydk]

It happened on a regular Ubuntu 22.04 install with the versions of python 3.10 and Virtualenv 20.13.0+ds. Are those the same versions you've tested?

[saltydk]

Example playbook:

---
- name: Virtual env example
  hosts: localhost

  tasks:
    - name: Install pip requirements
      ansible.builtin.pip:
        requirements: "/opt/cloudplow/requirements.txt"
        virtualenv_python: python3
        virtualenv: "/opt/cloudplow/venv"
      become: true
      become_user: "salty"

Example output:

sudo ansible-playbook playbook.yml -v
No config file found; using defaults
[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'

PLAY [Virtual env example] ***************************************************************************************************************************************************************************************************************************

TASK [Gathering Facts] *******************************************************************************************************************************************************************************************************************************
ok: [localhost]

TASK [Install pip requirements] **********************************************************************************************************************************************************************************************************************
fatal: [localhost]: FAILED! => {"changed": false, "msg": "Unable to find pip in the virtualenv, /opt/cloudplow/venv, under any of these names: pip3, pip. Make sure pip is present in the virtualenv."}

PLAY RECAP *******************************************************************************************************************************************************************************************************************************************
localhost                  : ok=1    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

venv folder layout:

tree /opt/cloudplow/venv -L 2
/opt/cloudplow/venv
├── lib
│   └── python3.10
├── local
│   ├── bin
│   └── lib
└── pyvenv.cfg

5 directories, 1 file

[saltydk]

The pip module only looks after the binaries at /opt/cloudplow/venv/bin so it never finds the binaries at /opt/cloudplow/venv/local/bin based on

ansible/lib/ansible/modules/pip.py

Lines 432 to 441 in d321fa3

	else:
	# If we're using a virtualenv we must use the pip from the
	# virtualenv
	venv_dir = os.path.join(env, 'bin')
	candidate_pip_basenames = (candidate_pip_basenames[0], 'pip')
	for basename in candidate_pip_basenames:
	candidate = os.path.join(venv_dir, basename)
	if os.path.exists(candidate) and is_executable(candidate):
	pip = candidate
	break

[mkrizek]

Which, guessing here, is likely why it cannot find pip since the paths have changed.

But is the change in paths expected?

Doing a quick google search it appears it is not expected and is a bug somewhere else, see pypa/setuptools#3278 or https://bugs.launchpad.net/ubuntu/+source/python3.10/+bug/1962791.

As per above it is not a bug in Ansible and Ansible should not try to work around that, it needs to be fixed somewhere else.

[saltydk]

can confirm that doing

---
- name: Virtual env example
  hosts: localhost

  tasks:
    - name: Install pip requirements
      ansible.builtin.pip:
        requirements: "/opt/cloudplow/requirements.txt"
        virtualenv_python: python3
        virtualenv: "/opt/cloudplow/venv"
      become: true
      become_user: "salty"
      environment:
        SETUPTOOLS_USE_DISTUTILS: stdlib

fixes the problem. I'll close this then as it would need a fix outside of Ansible.