env._find_executable_and_scripts(): Prefer the venv installation scheme if it exists #434

hroncok · 2022-01-31T13:43:50Z

Python distributors with custom default installation scheme can set a
scheme that can't be used to expand the paths in a venv.
This can happen if build itself is not installed in a venv.
The distributors are encouraged to set a "venv" scheme to be used for this.
See https://bugs.python.org/issue45413
and pypa/virtualenv#2208

Since Python that ships with the macOS developer tools does not have the "venv"
scheme yet, we keep the special case below. Once it gains the "venv" scheme,
it will be preferred.

Fixes #433

…me if it exists Python distributors with custom default installation scheme can set a scheme that can't be used to expand the paths in a venv. This can happen if build itself is not installed in a venv. The distributors are encouraged to set a "venv" scheme to be used for this. See https://bugs.python.org/issue45413 and pypa/virtualenv#2208 Since Python that ships with the macOS developer tools does not have the "venv" scheme yet, we keep the special case below. Once it gains the "venv" scheme, it will be preferred. Fixes pypa#433

src/build/env.py

layday · 2022-01-31T13:54:59Z

src/build/env.py

@@ -293,14 +293,21 @@ def _find_executable_and_scripts(path: str) -> Tuple[str, str, str]:
    """
    config_vars = sysconfig.get_config_vars().copy()  # globally cached, copy before altering it
    config_vars['base'] = path
+    # Python distributors with custom default installation scheme can set a
+    # scheme that can't be used to expand the paths in a venv.
+    # This can happen if build itself is not installed in a venv.


I don't think this is relevant or something that distributors should be doing, varying the scheme at runtime depending on the prefix.

That is the reality. We need to do that. See mostly https://bugs.python.org/issue43976

Unfortunately, this was made possible with the introduction of sysconfig._get_preferred_schemes in 3.10, so something like this is needed.

Co-authored-by: layday <layday@protonmail.com>

FFY00

This change looks alright, and is indeed necessary to deal with distributors overriding the default scheme.

@hroncok would you be able to move each part of the comment to above its relevant if/elif clause? I think that makes the code easier to read, instead of having this big comment.

hroncok · 2022-02-01T22:40:53Z

@hroncok would you be able to move each part of the comment to above its relevant if/elif clause? I think that makes the code easier to read, instead of having this big comment.

Done.

hroncok · 2022-03-02T17:21:19Z

In python/cpython#31034 the name of the installation scheme has been changed. We can either land this now and adapt later or wait a bit until the actual way forward is finalized there.

hroncok · 2022-03-18T10:03:26Z

python/cpython#31034 was merged, hence this is no longer a draft

FFY00 · 2022-03-18T17:34:33Z

Thanks!

FFY00 · 2022-03-18T17:35:30Z

Oops, forgot to squash merge 😕

hroncok · 2022-03-18T18:13:54Z

Oops, forgot to squash merge 😕

🤷‍♂️

…v" scheme After pypa#434 the test would error: ____________________ test_executable_missing_post_creation _____________________ mocker = <pytest_mock.plugin.MockerFixture object at 0x7f9a75401a80> @pytest.mark.skipif(IS_PYPY3, reason='PyPy3 uses get path to create and provision venv') def test_executable_missing_post_creation(mocker): mocker.patch.object(build.env, '_should_use_virtualenv', lambda: False) original_get_paths = sysconfig.get_paths def _get_paths(vars): # noqa shutil.rmtree(vars['base']) return original_get_paths(vars=vars) get_paths = mocker.patch('sysconfig.get_paths', side_effect=_get_paths) with pytest.raises(RuntimeError, match='Virtual environment creation failed, executable .* missing'): > with build.env.IsolatedEnvBuilder(): tests/test_env.py:73: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ .../site-packages/build/env.py:104: in __enter__ executable, scripts_dir = _create_isolated_env_venv(self._path) .../site-packages/build/env.py:258: in _create_isolated_env_venv executable, script_dir, purelib = _find_executable_and_scripts(path) .../site-packages/build/env.py:298: in _find_executable_and_scripts paths = sysconfig.get_paths(scheme='venv', vars=config_vars) /usr/lib64/python3.10/unittest/mock.py:1104: in __call__ return self._mock_call(*args, **kwargs) /usr/lib64/python3.10/unittest/mock.py:1108: in _mock_call return self._execute_mock_call(*args, **kwargs) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <MagicMock name='get_paths' id='140301368829136'>, args = () kwargs = {'scheme': 'venv', 'vars': {'ABIFLAGS': '', 'AC_APPLE_UNIVERSAL_BUILD': 0, 'AIX_BUILDDATE': 0, 'AIX_GENUINE_CPLUSPLUS': 0, ...}} effect = <function test_executable_missing_post_creation.<locals>._get_paths at 0x7f9a754ac5e0> def _execute_mock_call(self, /, *args, **kwargs): # separate from _increment_mock_call so that awaited functions are # executed separately from their call, also AsyncMock overrides this method effect = self.side_effect if effect is not None: if _is_exception(effect): raise effect elif not _callable(effect): result = next(effect) if _is_exception(result): raise result else: > result = effect(*args, **kwargs) E TypeError: test_executable_missing_post_creation.<locals>._get_paths() got an unexpected keyword argument 'scheme' /usr/lib64/python3.10/unittest/mock.py:1169: TypeError That only happens when not installed in a virtual environment, so it was not discovered on CI or via tox.

hroncok · 2022-03-18T18:35:58Z

Also, #448

…v" scheme After pypa#434 the test would error: ____________________ test_executable_missing_post_creation _____________________ mocker = <pytest_mock.plugin.MockerFixture object at 0x7f9a75401a80> @pytest.mark.skipif(IS_PYPY3, reason='PyPy3 uses get path to create and provision venv') def test_executable_missing_post_creation(mocker): mocker.patch.object(build.env, '_should_use_virtualenv', lambda: False) original_get_paths = sysconfig.get_paths def _get_paths(vars): # noqa shutil.rmtree(vars['base']) return original_get_paths(vars=vars) get_paths = mocker.patch('sysconfig.get_paths', side_effect=_get_paths) with pytest.raises(RuntimeError, match='Virtual environment creation failed, executable .* missing'): > with build.env.IsolatedEnvBuilder(): tests/test_env.py:73: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ .../site-packages/build/env.py:104: in __enter__ executable, scripts_dir = _create_isolated_env_venv(self._path) .../site-packages/build/env.py:258: in _create_isolated_env_venv executable, script_dir, purelib = _find_executable_and_scripts(path) .../site-packages/build/env.py:298: in _find_executable_and_scripts paths = sysconfig.get_paths(scheme='venv', vars=config_vars) /usr/lib64/python3.10/unittest/mock.py:1104: in __call__ return self._mock_call(*args, **kwargs) /usr/lib64/python3.10/unittest/mock.py:1108: in _mock_call return self._execute_mock_call(*args, **kwargs) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <MagicMock name='get_paths' id='140301368829136'>, args = () kwargs = {'scheme': 'venv', 'vars': {'ABIFLAGS': '', 'AC_APPLE_UNIVERSAL_BUILD': 0, 'AIX_BUILDDATE': 0, 'AIX_GENUINE_CPLUSPLUS': 0, ...}} effect = <function test_executable_missing_post_creation.<locals>._get_paths at 0x7f9a754ac5e0> def _execute_mock_call(self, /, *args, **kwargs): # separate from _increment_mock_call so that awaited functions are # executed separately from their call, also AsyncMock overrides this method effect = self.side_effect if effect is not None: if _is_exception(effect): raise effect elif not _callable(effect): result = next(effect) if _is_exception(result): raise result else: > result = effect(*args, **kwargs) E TypeError: test_executable_missing_post_creation.<locals>._get_paths() got an unexpected keyword argument 'scheme' /usr/lib64/python3.10/unittest/mock.py:1169: TypeError That only happens when not installed in a virtual environment, so it was not discovered on CI or via tox.

…v" scheme After #434 the test would error: ____________________ test_executable_missing_post_creation _____________________ mocker = <pytest_mock.plugin.MockerFixture object at 0x7f9a75401a80> @pytest.mark.skipif(IS_PYPY3, reason='PyPy3 uses get path to create and provision venv') def test_executable_missing_post_creation(mocker): mocker.patch.object(build.env, '_should_use_virtualenv', lambda: False) original_get_paths = sysconfig.get_paths def _get_paths(vars): # noqa shutil.rmtree(vars['base']) return original_get_paths(vars=vars) get_paths = mocker.patch('sysconfig.get_paths', side_effect=_get_paths) with pytest.raises(RuntimeError, match='Virtual environment creation failed, executable .* missing'): > with build.env.IsolatedEnvBuilder(): tests/test_env.py:73: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ .../site-packages/build/env.py:104: in __enter__ executable, scripts_dir = _create_isolated_env_venv(self._path) .../site-packages/build/env.py:258: in _create_isolated_env_venv executable, script_dir, purelib = _find_executable_and_scripts(path) .../site-packages/build/env.py:298: in _find_executable_and_scripts paths = sysconfig.get_paths(scheme='venv', vars=config_vars) /usr/lib64/python3.10/unittest/mock.py:1104: in __call__ return self._mock_call(*args, **kwargs) /usr/lib64/python3.10/unittest/mock.py:1108: in _mock_call return self._execute_mock_call(*args, **kwargs) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <MagicMock name='get_paths' id='140301368829136'>, args = () kwargs = {'scheme': 'venv', 'vars': {'ABIFLAGS': '', 'AC_APPLE_UNIVERSAL_BUILD': 0, 'AIX_BUILDDATE': 0, 'AIX_GENUINE_CPLUSPLUS': 0, ...}} effect = <function test_executable_missing_post_creation.<locals>._get_paths at 0x7f9a754ac5e0> def _execute_mock_call(self, /, *args, **kwargs): # separate from _increment_mock_call so that awaited functions are # executed separately from their call, also AsyncMock overrides this method effect = self.side_effect if effect is not None: if _is_exception(effect): raise effect elif not _callable(effect): result = next(effect) if _is_exception(result): raise result else: > result = effect(*args, **kwargs) E TypeError: test_executable_missing_post_creation.<locals>._get_paths() got an unexpected keyword argument 'scheme' /usr/lib64/python3.10/unittest/mock.py:1169: TypeError That only happens when not installed in a virtual environment, so it was not discovered on CI or via tox.

hroncok mentioned this pull request Jan 31, 2022

Custom default sysconfig INSTALL_SCHEME leads to superfulous "Virtual environment creation failed" #433

Closed

hroncok force-pushed the venv_scheme branch from 0b56f28 to 211bd2f Compare January 31, 2022 13:46

layday reviewed Jan 31, 2022

View reviewed changes

src/build/env.py Outdated Show resolved Hide resolved

layday reviewed Jan 31, 2022

View reviewed changes

Save and reuse sysconfig.get_scheme_names()

3c08a32

Co-authored-by: layday <layday@protonmail.com>

hroncok mentioned this pull request Jan 31, 2022

RuntimeError: Virtual environment creation failed, executable /usr/local/bin/python missing #215

Closed

FFY00 approved these changes Feb 1, 2022

View reviewed changes

FFY00 requested a review from gaborbernat February 1, 2022 22:21

Fixup: Move each part of the comment to its relevant if/elif clause

512f1fb

hroncok marked this pull request as ready for review March 18, 2022 10:02

hroncok requested a review from henryiii as a code owner March 18, 2022 10:02

FFY00 merged commit 22de4ab into pypa:main Mar 18, 2022

hroncok deleted the venv_scheme branch March 18, 2022 18:11

hroncok mentioned this pull request Mar 18, 2022

Fixup mock in test_executable_missing_post_creation to work with "ven… #448

Merged

torsava mentioned this pull request May 6, 2022

Make pipenv work with the venv install scheme if it is detected pypa/pipenv#5096

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

env._find_executable_and_scripts(): Prefer the venv installation scheme if it exists #434

env._find_executable_and_scripts(): Prefer the venv installation scheme if it exists #434

hroncok commented Jan 31, 2022

layday Jan 31, 2022 •

edited

hroncok Jan 31, 2022

FFY00 Feb 1, 2022

FFY00 left a comment

hroncok commented Feb 1, 2022

hroncok commented Mar 2, 2022

hroncok commented Mar 18, 2022

FFY00 commented Mar 18, 2022

FFY00 commented Mar 18, 2022

hroncok commented Mar 18, 2022 •

edited

hroncok commented Mar 18, 2022

env._find_executable_and_scripts(): Prefer the venv installation scheme if it exists #434

env._find_executable_and_scripts(): Prefer the venv installation scheme if it exists #434

Conversation

hroncok commented Jan 31, 2022

layday Jan 31, 2022 • edited

Choose a reason for hiding this comment

hroncok Jan 31, 2022

Choose a reason for hiding this comment

FFY00 Feb 1, 2022

Choose a reason for hiding this comment

FFY00 left a comment

Choose a reason for hiding this comment

hroncok commented Feb 1, 2022

hroncok commented Mar 2, 2022

hroncok commented Mar 18, 2022

FFY00 commented Mar 18, 2022

FFY00 commented Mar 18, 2022

hroncok commented Mar 18, 2022 • edited

hroncok commented Mar 18, 2022

layday Jan 31, 2022 •

edited

hroncok commented Mar 18, 2022 •

edited