RuntimeError: Virtual environment creation failed, executable /usr/local/bin/python missing

[nschloe]

When running

python3 -m build --sdist --wheel .

on Mac OS, I'm getting

Traceback (most recent call last):
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/runpy.py", line 193, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/Users/nschloe/Library/Python/3.8/lib/python/site-packages/build/__main__.py", line 210, in <module>
    main(sys.argv[1:], 'python -m build')
  File "/Users/nschloe/Library/Python/3.8/lib/python/site-packages/build/__main__.py", line 202, in main
    build_package(args.srcdir, outdir, distributions, config_settings, not args.no_isolation, args.skip_dependencies)
  File "/Users/nschloe/Library/Python/3.8/lib/python/site-packages/build/__main__.py", line 81, in build_package
    _build_in_isolated_env(builder, outdir, distributions)
  File "/Users/nschloe/Library/Python/3.8/lib/python/site-packages/build/__main__.py", line 45, in _build_in_isolated_env
    with IsolatedEnvBuilder() as env:
  File "/Users/nschloe/Library/Python/3.8/lib/python/site-packages/build/env.py", line 55, in __enter__
    executable, pip_executable = _create_isolated_env(self._path)
  File "/Users/nschloe/Library/Python/3.8/lib/python/site-packages/build/env.py", line 171, in _create_isolated_env
    executable = _find_executable(path)
  File "/Users/nschloe/Library/Python/3.8/lib/python/site-packages/build/env.py", line 208, in _find_executable
    raise RuntimeError('Virtual environment creation failed, executable {} missing'.format(executable))
RuntimeError: Virtual environment creation failed, executable /usr/local/bin/python missing

Not sure at all whe it would be looking for python in /usr/local. Any idea what's going wrong?

[gaborbernat]

What folder are you running this in?

[nschloe]

A subfolder of $HOME, where the software sits (complete with pyproject.toml etc.).

[layday]

How did you install Python? It looks like sysconfig might've been patched to ignore the value of base.

[nschloe]

Using default Python on MacOS 11.1 here. Hm...

[uranusjr]

default Python on MacOS 11.1

macOS 11.1 does not have a default Python. (Well, technically there is, but it’s not usable in user land, and certainly not located at /usr/local/bin/python.)

[nschloe]

Then I have no idea. It resides in

which python3

/usr/bin/python3

[layday]

I wonder if sysconfig from another Python installation is on your sys.path. Not sure if it'll get us very far, but could you attach the output of these two commands:

python -c "import sysconfig; print(sysconfig.get_paths())"
python -c "import sys; print(sys.path)"

[nschloe]

I played around with it a little more and I'd say this is too hard to figure out remotely. Thanks anyways for your help! I'll post a solution when I found one.

[bryanheinz]

@nschloe were you doing this on an M1 Mac? i ran into this issue last night on an M1 Mac, but ran fine when i tried on an Intel Mac.

[nschloe]

It's an old Intel MacBook Air, but fully upgraded (Big Sur 11.1).

[Patol75]

I am encountering a similar issue on Ubuntu 20.04.3 LTS using Python 3.10, which I installed from the deadsnakes PPA.
python3.10 -m build returns

* Creating venv isolated environment...

Traceback (most recent call last):
  File "/home/thomas/.local/lib/python3.10/site-packages/build/__main__.py", line 372, in main
    built = build_call(
  File "/home/thomas/.local/lib/python3.10/site-packages/build/__main__.py", line 229, in build_package_via_sdist
    sdist = _build(isolation, builder, outdir, 'sdist', config_settings, skip_dependency_check)
  File "/home/thomas/.local/lib/python3.10/site-packages/build/__main__.py", line 140, in _build
    return _build_in_isolated_env(builder, outdir, distribution, config_settings)
  File "/home/thomas/.local/lib/python3.10/site-packages/build/__main__.py", line 104, in _build_in_isolated_env
    with _IsolatedEnvBuilder() as env:
  File "/home/thomas/.local/lib/python3.10/site-packages/build/env.py", line 104, in __enter__
    executable, scripts_dir = _create_isolated_env_venv(self._path)
  File "/home/thomas/.local/lib/python3.10/site-packages/build/env.py", line 258, in _create_isolated_env_venv
    executable, script_dir, purelib = _find_executable_and_scripts(path)
  File "/home/thomas/.local/lib/python3.10/site-packages/build/env.py", line 303, in _find_executable_and_scripts
    raise RuntimeError(f'Virtual environment creation failed, executable {executable} missing')
RuntimeError: Virtual environment creation failed, executable /tmp/build-env-lcuno53i/local/bin/python missing

ERROR Virtual environment creation failed, executable /tmp/build-env-lcuno53i/local/bin/python missing

If, instead, I use Python 3.9, also installed from the deadsnakes PPA, python3.9 -m build completes without errors.

[layday]

It looks like the prefix scheme is changed to posix_local as a result of this patch and build expects that venvs are created using the default (prefix) scheme. However, virtualenv has added support for the venv scheme and the same patch includes:

+_INSTALL_SCHEMES['venv'] = _INSTALL_SCHEMES['posix_prefix']

so venvs are created using a different scheme.

[layday]

We could add another condition here to check for the existence of the venv scheme although I'm -1 on supporting venv before it lands in CPython.

[layday]

@gaborbernat Can we not grab the Python executable path from virtualenv directly?

Wait, this is the stdlib venv... so something else is going on.

Oh right, of course, the venv module doesn't use sysconfig, so deadsnakes would need to patch venv if they are using an incompatible default scheme.

Please report this to deadsnakes.

[Patol75]

@layday As is tracked above, I have reported the issue to deadsnakes, but the developer there believes it is actually related to build. I can confirm that I can replicate the behaviour he documents in his reply, as well as actually activating the virtual environment and deactivating it.

[layday]

Yes, venvs are created "correctly". The issue is that we cannot extract the venv paths if a different scheme is used for the venv because we operate on the assumption that the system default and the venv schemes are the same. That is how venvs have historically worked and it's how they are supposed to work. If Debian or deadnakes or Fedora or whoever wish to deviate from the spec, they need to put in the effort to make sure their changes are compatible with the wider ecosystem.

[layday]

Can you actually pip install anything in this venv with the latest version of pip? If you can, i.e. if pip falls back on distutils, then it should also emit a warning.

[Patol75]

Thanks for your help @layday.

Does the following change anything to your argument?

thomas@thomas-GL502VM:~$ python3.10 -m venv venv
thomas@thomas-GL502VM:~$ source venv/bin/activate
(venv) thomas@thomas-GL502VM:~$ python3 -m pip install --upgrade pip
Requirement already satisfied: pip in ./venv/lib/python3.10/site-packages (21.2.4)
Collecting pip
  Using cached pip-21.3.1-py3-none-any.whl (1.7 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 21.2.4
    Uninstalling pip-21.2.4:
      Successfully uninstalled pip-21.2.4
Successfully installed pip-21.3.1
(venv) thomas@thomas-GL502VM:~$ python3 -m pip install --upgrade numpy
Collecting numpy
  Using cached numpy-1.22.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (16.8 MB)
Installing collected packages: numpy
Successfully installed numpy-1.22.1
(venv) thomas@thomas-GL502VM:~$ ls venv/lib
lib/   lib64/ 
(venv) thomas@thomas-GL502VM:~$ ls venv/lib/python3.10/site-packages/
_distutils_hack  distutils-precedence.pth  numpy  numpy-1.22.1.dist-info  numpy.libs  pip  pip-21.3.1.dist-info  pkg_resources  setuptools  setuptools-58.1.0.dist-info
(venv) thomas@thomas-GL502VM:~$ deactivate
thomas@thomas-GL502VM:~$ 

[layday]

I assume pip calls get_preferred_scheme in 3.10; deadsnakes is varying the prefix depending on whether it's being called from inside a venv, meaning that it might or might not work for build depending on whether build itself was installed inside a venv. get_preferred_scheme wasn't introduced so people could fudge the paths at runtime.

[Patol75]

Do you mind checking the latest reply from the deadsnakes developer? The thread is tracked above.

[layday]

We create venvs programmatically. We do not know where the venv Python is located, or where any of the other paths are located inside the venv, which is why we are calling sysconfig.get_paths in the first place. So no, whatever works for virtualenv and venv (or even pip) won't work for us. We could hardcode the Python path as venv-dir/Scripts/python on Windows and venv-dir/bin/python everywhere else, attempt to call get_paths in a subprocess and hope for the best. Personally, I'd rather Debian roll back their patches and work with the core Python team to find a workable solution.

[gaborbernat]

A workaround would be go install build[virtualenv], I think that would not have this issue 🤔

[Patol75]

The developer at deadsnakes is wondering if an approach similar to this one could work for build.

[layday]

I mentioned that as a possibility earlier. I do think it's the wrong way to approach altering the venv scheme. If a different scheme is sought for venvs, a new entry should be added to the preferred scheme dict, for venvs. get_preferred_scheme is supposed to be the single source of truth for schemes by install type. And because it bears repeating: they should absolutely not be varying the schemes depending on the active prefix. I'm lukewarm to the idea of relying on an extension to the venv spec that has scarcely been discussed and has not been merged into CPython. But if it's gonna get us out of this bind, I suppose we could look for venv in the scheme names, like we do with the framework prefix for Apple's Python. Or we could recommend that people use virtualenv, like Bernat has suggested.

[henryiii]

Requiring build[virtualenv] is not an option. We can't force everyone to use the option to get a non-broken build. If we really can't get it properly fixed, we will be forced to mitigate it here.

@layday, can you try to engage the Debian developers (and/or any relevant parties) to see if this can be fixed? This is likely still fixable, since AFAIK it's only shipped to deadsnakes and probably Debian unstable.

[FFY00]

The behavior is technically correct on Python 3.9 and older, it only becomes a problem when distributors break assumptions and change the install layout, which is not supported by the Python upstream. However, this becomes an issue in 3.10 as the Python upstream provides an official mechanism to customize the install layout.

Using the venv scheme would be possible, but that has not been standardized in the Python upstream, only Fedora currently implements it.

We can use the venv scheme if it exists, and fallback on the expected hardcoded path.

The deadsnakes developer can adopt the same approach of specifying the venv scheme if he wants. I do not want to interact with him, so this will be the full extent of my advice.

See https://bugs.python.org/issue45413 for more details.

[hroncok]

Using the venv scheme would be possible, but that has not been standardized in the Python upstream, only Fedora currently implements it.

That was proposed in #434

python/cpython#31034 is the CPython PR that adds it -- if merged, others can start using it soon (it should even fix macOS Python once they update to 3.11)

[hroncok]

deadsnakes now defines the "venv" installation scheme as well: deadsnakes/python3.10@9b93d73

[layday]

Yes, deadsnakes implementation is what prompted the discussion. Sent with [ProtonMail](https://protonmail.com/) Secure Email. ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

…

On Monday, January 31st, 2022 at 18:21, Miro Hrončok ***@***.***> wrote: deadsnakes now defines the "venv" installation scheme as well: ***@***.***(deadsnakes/python3.10@9b93d73) — Reply to this email directly, [view it on GitHub](#215 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/AHNRFWEFC5SLDFDFMAMAE33UY2ZJHANCNFSM4WXKMP2A). Triage notifications on the go with GitHub Mobile for [iOS](https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675) or [Android](https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub). You are receiving this because you were mentioned.Message ID: ***@***.***>

[linsomniac]

A workaround would be go install build[virtualenv], I think that would not have this issue 🤔

Thank you for this, I was following the Python packaging documentation ( https://packaging.python.org/en/latest/tutorials/packaging-projects/ ) and got stuck with a similar error message, this was the first google hit on a search for it, and installing build[virtualenv] solved my problem.

[petersilva]

same thing here... we are told python3 setup.py bdist_wheel is "obsolete" and to use pythom -m build...
but that crashes it a totally stock (ubuntu 22.04, os provided python everything):

fractal% python3 -m build
* Creating venv isolated environment...

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/build/__main__.py", line 372, in main
    built = build_call(
  File "/usr/lib/python3/dist-packages/build/__main__.py", line 229, in build_package_via_sdist
    sdist = _build(isolation, builder, outdir, 'sdist', config_settings, skip_dependency_check)
  File "/usr/lib/python3/dist-packages/build/__main__.py", line 140, in _build
    return _build_in_isolated_env(builder, outdir, distribution, config_settings)
  File "/usr/lib/python3/dist-packages/build/__main__.py", line 104, in _build_in_isolated_env
    with _IsolatedEnvBuilder() as env:
  File "/usr/lib/python3/dist-packages/build/env.py", line 104, in __enter__
    executable, scripts_dir = _create_isolated_env_venv(self._path)
  File "/usr/lib/python3/dist-packages/build/env.py", line 258, in _create_isolated_env_venv
    executable, script_dir, purelib = _find_executable_and_scripts(path)
  File "/usr/lib/python3/dist-packages/build/env.py", line 303, in _find_executable_and_scripts
    raise RuntimeError(f'Virtual environment creation failed, executable {executable} missing')
RuntimeError: Virtual environment creation failed, executable /tmp/build-env-tcdplhrn/local/bin/python missing

ERROR Virtual environment creation failed, executable /tmp/build-env-tcdplhrn/local/bin/python missing
fractal% 

[henryiii]

Hmm, wonder if it's only making python3 instead of python? venv isn't supposed to do that, but maybe it's been hacked a bit?

FYI, build works fine on stock Ubuntu 22.04 in a docker container:

$ docker run --rm -it ubuntu:22.04
# apt update && apt install -y python3-venv python3-pip git
# git clone https://github.com/pypa/build
# cd build/
# python3 -m pip install build
# python3 -m build --wheel
* Creating venv isolated environment...
* Installing packages in isolated environment... (flit-core >= 3.4)
* Getting build dependencies for wheel...
* Building wheel...
Successfully built build-0.9.0-py3-none-any.whl

[henryiii]

Ahah, the python3-build package is busted. Not sure that's our fault. It's based on 0.7.0, which also shows this problem. So Debian / Ubuntu should update it. 0.8.0 and 0.9.0 both work.

[henryiii]

As a workaround you can install python3-virtualenv, which fixes this (and might be the best fix for Debian / Ubuntu).

[zoot-inge]

I found: https://bugs.launchpad.net/ubuntu/+source/python-build/+bug/1992108 with quick fix