Charms fail install hook if python3-setuptools > 60

[mkalcok]

On machines that have (system-wide) setuptools version 60 or higher, installation fails with:

unit-ovn-chassis-0: 16:07:11 WARNING unit.ovn-chassis/0.install Traceback (most recent call last):
unit-ovn-chassis-0: 16:07:11 WARNING unit.ovn-chassis/0.install   File "/var/lib/juju/agents/unit-ovn-chassis-0/charm/hooks/install", line 8, in <module>
unit-ovn-chassis-0: 16:07:11 WARNING unit.ovn-chassis/0.install     basic.bootstrap_charm_deps()
unit-ovn-chassis-0: 16:07:11 WARNING unit.ovn-chassis/0.install   File "/var/lib/juju/agents/unit-ovn-chassis-0/charm/lib/charms/layer/basic.py", line 202, in bootstrap_charm_deps
unit-ovn-chassis-0: 16:07:11 WARNING unit.ovn-chassis/0.install     _update_if_newer(pip, pre_install_pkgs)
unit-ovn-chassis-0: 16:07:11 WARNING unit.ovn-chassis/0.install   File "/var/lib/juju/agents/unit-ovn-chassis-0/charm/lib/charms/layer/basic.py", line 362, in _update_if_newer
unit-ovn-chassis-0: 16:07:11 WARNING unit.ovn-chassis/0.install     installed = _load_installed_versions(pip)
unit-ovn-chassis-0: 16:07:11 WARNING unit.ovn-chassis/0.install   File "/var/lib/juju/agents/unit-ovn-chassis-0/charm/lib/charms/layer/basic.py", line 278, in _load_installed_versions
unit-ovn-chassis-0: 16:07:11 WARNING unit.ovn-chassis/0.install     pip_freeze = check_output([pip, 'freeze']).decode('utf8')
unit-ovn-chassis-0: 16:07:11 WARNING unit.ovn-chassis/0.install   File "/usr/lib/python3.10/subprocess.py", line 421, in check_output
unit-ovn-chassis-0: 16:07:11 WARNING unit.ovn-chassis/0.install     return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
unit-ovn-chassis-0: 16:07:11 WARNING unit.ovn-chassis/0.install   File "/usr/lib/python3.10/subprocess.py", line 503, in run
unit-ovn-chassis-0: 16:07:11 WARNING unit.ovn-chassis/0.install     with Popen(*popenargs, **kwargs) as process:
unit-ovn-chassis-0: 16:07:11 WARNING unit.ovn-chassis/0.install   File "/usr/lib/python3.10/subprocess.py", line 971, in __init__
unit-ovn-chassis-0: 16:07:11 WARNING unit.ovn-chassis/0.install     self._execute_child(args, executable, preexec_fn, close_fds,
unit-ovn-chassis-0: 16:07:11 WARNING unit.ovn-chassis/0.install   File "/usr/lib/python3.10/subprocess.py", line 1863, in _execute_child
unit-ovn-chassis-0: 16:07:11 WARNING unit.ovn-chassis/0.install     raise child_exception_type(errno_num, err_msg, err_filename)
unit-ovn-chassis-0: 16:07:11 WARNING unit.ovn-chassis/0.install FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/juju/agents/unit-ovn-chassis-0/.venv/bin/pip'

I think it's connected to virtualenv using different layouts, influenced by different versions of setuptools as described here pypa/setuptools#3278. Charm fails to find pip because instead of usual <venv>/bin/pip it's located in <venv>/local/bin/pip

[mkalcok]

The problem seems to go deeper than just difference in layout. Virtual environments created on machines with virtualenv 20.13.0+ds and python3-setuptools 68.1.2-2ubuntu1 do not seem to isolate at all. Creating clean venv and running pip freeze shows system-wide dist packages and using pip from the virtual environment to install packages attempts to mess with the system packages as well:

pip install -U --force-reinstall --no-index --no-cache-dir -f wheelhouse PyYAML==6.0.1
Looking in links: wheelhouse
Processing ./wheelhouse/PyYAML-6.0.1-cp310-cp310-linux_x86_64.whl
Installing collected packages: PyYAML
  Attempting uninstall: PyYAML
    Found existing installation: PyYAML 6.0.1
ERROR: Cannot uninstall PyYAML 6.0.1, RECORD file not found. Hint: The package was installed by debian.