Set 'home' to parent directory of system_executable

[vfazio]

PEP 405 says of the "home" key:

"If a home key is found, this signifies that the Python binary belongs to a virtual environment, and the value of the home key is the directory containing the Python executable used to create this virtual environment."

And:

"In this case, prefix-finding continues as normal using the value of the home key as the effective Python binary location, which finds the prefix of the base installation."

Previously, "home" was being set to interpreter.system_exec_prefix which does not abide by the PEP specification.

In Python 3.11, the "home" directory is used to determine the value of sys._base_executable, so if the path specified is incorrect, the path + interpreter returned will be invalid. This can cause headaches later when trying to probe info via the discovery module.

Now, set this to the parent directory of interpreter.system_executable.

close #2440

It may not resolve the problem fully as virtualenv cannot control the sys._base_executable "calculation" short of specifying the home directory. This means there's still a chance that a path to "python" gets returned even though there may not be one available in the parent-most non-venv python directory (the python interpreter path that created the upper-most venv).

Thanks for contributing, make sure you address all the checklists (for details on how see development documentation)!

• ran the linter to address style issues (tox -e fix_lint)
• wrote descriptive pull request text
• ensured there are test(s) validating the fix
• added news fragment in docs/changelog folder
• updated/extended the documentation

[gaborbernat]

Needs test too 👍

[vfazio]

Needs test too 👍

I don't mind stashing a test in some current unittest, but when i glanced around i didn't see any existing tests really validating the contents of pyvenv.cfg.

Do you have a suggestion on where to place a test for this?

[vfazio]

I'm a little confused by the failure on the macos-12 3.8 pipeline that doesn't occur in 3.9. I don't have a mac to test and debug this on.

One thing I could do is skip this for venv since it technically has the ability to overwrite this key from what is provided in the base class

    def set_pyenv_cfg(self):
        # prefer venv options over ours, but keep our extra
        venv_content = copy(self.pyenv_cfg.refresh())
        super().set_pyenv_cfg()
        self.pyenv_cfg.update(venv_content)

otherwise i'm not sure what the nuance is. The native venv should be doing the same we're doing https://github.com/python/cpython/blob/3.8/Lib/venv/__init__.py#L109

I see

using tox-3.27.0 from /usr/local/lib/python3.8/site-packages/tox/__init__.py (pid 5370)
/usr/local/opt/python@3.8/bin/python3.8 (/usr/local/opt/python@3.8/bin/python3.8) is {'executable': '/usr/local/opt/python@3.8/bin/python3.8', 'implementation': 'CPython', 'version_info': [3, 8, 15, 'final', 0], 'version': '3.8.15 (default, Oct 11 2022, 21:52:37) \n[Clang 14.0.0 (clang-1400.0.29.102)]', 'is_64': True, 'sysplatform': 'darwin', 'os_sep': '/', 'extra_version_info': None}

unless there's a symlink between /usr/local/Cellar/ and /usr/local/opt/ and it's getting resolved in one code path and not in another but the only reasonable place that could be happening is in python's getpath code when setting base_executable as part of:

https://github.com/python/cpython/blob/3.8/Modules/getpath.c#L1234

which happens before

https://github.com/python/cpython/blob/3.8/Modules/getpath.c#L1242

unless the test I wrote isn't actually testing what i thought it was

[gaborbernat]

Seems we're failing on ubuntu too, can you take a look?

[vfazio]

Seems we're failing on ubuntu too, can you take a look?

Looks related to installing nushell:

2022-11-10 02:37:39 (119 MB/s) - ‘nushell.tar.gz’ saved [25369685/25369685]

cp: cannot stat 'nu': No such file or directory
Error: Process completed with exit code 1.

[gaborbernat]

This still seems to be failling on macos 🤔

[vfazio]

This still seems to be failling on macos 🤔

I'll try to setup a VM with MacOS 12 and see if i can recreate it... it's just odd that it fails on 3.8 but not 3.9

[gaborbernat]

brew macos is weird like that 🤦 they change the layout every version, e.g. 3.9 and 3.10 does not even pass our test suite...

[vfazio]

well, i can still try to setup a VM to debug this. trying to debug it through a CI pipeline seems insane. If you have hardware and can help debug that'd be great. I probably won't be able to get around to this for the next few days however. My guess is it's a symlink issue and that maybe i need to realpath something, but i'd like to understand it before making changes since it's working on all of the other platforms.

or if there's a way for me to mark this as skip, i can do that too if that's your preference.

[vfazio]

As suspected:

vfazio@Vincents-Mac tmp.TKRRHjTO % readlink  /usr/local/opt/python@3.8 
../Cellar/python@3.8/3.8.15

So certain code paths will end up using "resolved" paths and others won't. So instead of doing an explicit path comparison, i'm just going to check that python exists in the path pointed to by the "home" key, regardless of if venv was the creator or if virtualenv was.

[vfazio]

There is no reason why that ubuntu 3.11 pipeline should be failing... it seems to imply that apt-get install wget curl -y is returning some non-zero error code or that hitting the nushell release API endpoint is failing, both of which seems weird