Exception in configuring venv when the project has a setup.py file

[mrc0mmand]

Hey!

When playing around with python CodeQL analysis I stumbled across an exception during the init phase:

  Installing collected packages: virtualenv-clone, pipenv
  Successfully installed pipenv-2022.9.8 virtualenv-clone-0.5.7
  + command -v python2
  /usr/bin/python3 -B /home/runner/work/_actions/github/codeql-action/v2/python-setup/auto_install_packages.py /opt/hostedtoolcache/CodeQL/0.0.0-20220825/x64/codeql
  Source root appears to be the real root.
  Will try to guess Python version, as it was not specified in `lgtm.yml`
  Trying to guess Python version based on Trove classifiers in setup.py
  Found Trove classifiers for both Python 2 and Python 3 in setup.py -- will use Python 3
  Found setup.py, will install package with pip in editable mode
  Creating venv in /home/runner/work/python-systemd/codeql-action-python-autoinstall
  + python3 -m virtualenv /home/runner/work/python-systemd/codeql-action-python-autoinstall
  /home/runner/.local/lib/python3.10/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
    warnings.warn(
  created virtual environment CPython3.10.4.final.0-64 in 491ms
    creator CPython3Posix(dest=/home/runner/work/python-systemd/codeql-action-python-autoinstall, clear=False, no_vcs_ignore=False, global=False)
    seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/runner/.local/share/virtualenv)
      added seed packages: pip==21.3.1, setuptools==58.3.0, wheel==0.37.0
    activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
  + /home/runner/work/python-systemd/codeql-action-python-autoinstall/bin/pip install -e .
  Traceback (most recent call last):
    File "/home/runner/work/_actions/github/codeql-action/v2/python-setup/auto_install_packages.py", line 175, in <module>
      python_executable_path = install_packages(codeql_base_dir)
    File "/home/runner/work/_actions/github/codeql-action/v2/python-setup/auto_install_packages.py", line 163, in install_packages
      return install_with_setup_py(version)
    File "/home/runner/work/_actions/github/codeql-action/v2/python-setup/auto_install_packages.py", line 134, in install_with_setup_py
      _check_call([venv_pip, 'install', '-e', '.'])
    File "/home/runner/work/_actions/github/codeql-action/v2/python-setup/auto_install_packages.py", line 14, in _check_call
      subprocess.check_call(command, stdin=subprocess.DEVNULL)
    File "/usr/lib/python3.10/subprocess.py", line 364, in check_call
      retcode = call(*popenargs, **kwargs)
    File "/usr/lib/python3.10/subprocess.py", line 345, in call
      with Popen(*popenargs, **kwargs) as p:
    File "/usr/lib/python3.10/subprocess.py", line 966, in __init__
      self._execute_child(args, executable, preexec_fn, close_fds,
    File "/usr/lib/python3.10/subprocess.py", line 1842, in _execute_child
      raise child_exception_type(errno_num, err_msg, err_filename)
  FileNotFoundError: [Errno 2] No such file or directory: '/home/runner/work/python-systemd/codeql-action-python-autoinstall/bin/pip'
Warning: An error occurred while trying to automatically install Python dependencies: Error: The process '/usr/bin/python3' failed with exit code 1
Please make sure any necessary dependencies are installed before calling the codeql-action/analyze step, and add a 'setup-python-dependencies: false' argument to this step to disable our automatic dependency installation and avoid this warning.

Taken from: https://github.com/systemd/python-systemd/actions/runs/3062293076/jobs/4943107761#step:3:439
Also appears in: https://github.com/systemd/mkosi/actions/runs/3051938808/jobs/4920754739#step:3:443

It doesn't seem to be fatal (at least in the projects I currently tested it with), but it generates a warning on the results overview page:

This seems to happen only when the project has a setup.py file; in projects without it the relevant step fails cleanly:

   Successfully installed pipenv-2022.9.8 virtualenv-clone-0.5.7
  + command -v python2
  /usr/bin/python3 -B /home/runner/work/_actions/github/codeql-action/0c670bbf0414f39666df6ce8e718ec5662c21e03/python-setup/auto_install_packages.py /opt/hostedtoolcache/CodeQL/0.0.0-20220825/x64/codeql
  No directories containing root identifiers were found. Returning working directory as root.
  Will try to guess Python version, as it was not specified in `lgtm.yml`
  Trying to guess Python version based on Trove classifiers in setup.py
  Did not find setup.py (expected it to be at /home/runner/work/systemd/systemd/setup.py)
  Trying to guess Python version based on travis file
  Did not find any travis files (expected them at either ['/home/runner/work/systemd/systemd/.travis.yml', '/home/runner/work/systemd/systemd/travis.yml'])
  Trying to guess Python version based on installed versions
  Only Python 3 installed -- will use that version
  was not able to install packages automatically

See: https://github.com/systemd/systemd/actions/runs/3061829175/jobs/4942084517#step:3:459
or: https://github.com/systemd/systemd-centos-ci/actions/runs/3060692901/jobs/4939517444#step:3:424

[RasmusWL]

Hi @mrc0mmand, that does look like a bug, thanks for reporting it 🙏 Let me see if I can come up with a fix later today.

[RasmusWL]

I wasn't able to figure out what was going on, but will look more at it Monday.

To unblock you, you can do the dependency installation manually as described here: https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning#analyzing-python-dependencies

[mrc0mmand]

I wasn't able to figure out what was going on, but will look more at it Monday.

To unblock you, you can do the dependency installation manually as described here: https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning#analyzing-python-dependencies

Hey, thanks for looking into it and no worries, it doesn't really affect any of the projects I tried it with so far.

[RasmusWL]

Great to hear 👍

I've found the underlying cause. As described here, when using Ubuntu 22.04 with new enough versions of setuptools (60.0.0+), the virtual environment created with virtualenv will put binaries in <venv-path>/local/bin instead of <venv-path>/bin.

Next release after 20.16.5 of virtualenv will include a fix for this (PR).

I've created a temporary fix for this in #1257. again, thanks for reporting 💪 (our automated testing didn't catch this, since it's using ubuntu-latest, and not 22.04)

[mrc0mmand]

Thank you!