Ensure writable paths are used for wheel installer #9014

abn · 2024-02-25T02:09:59Z

This change fixes an issue where when system environment is used as a destination for wheel installation as a user without access to system site packages, installation fails with permission error.

Functionally, this now follows the same behaviour as when a package is installed to user site.

Resolves: #8408

podman run --rm -i --entrypoint bash docker.io/apache/airflow:2.7.1-python3.11 <<EOF
set -xe
python -m pip install --disable-pip-version-check -q pipx
pipx install https://api.github.com/repos/abn/poetry/tarball/fix-system-env-installer-paths
export POETRY_VIRTUALENVS_CREATE=false
poetry new demo
pushd demo
poetry add cowsay
pip show cowsay
EOF

console.log

+ python -m pip install --disable-pip-version-check -q pipx
+ pipx install https://api.github.com/repos/abn/poetry/tarball/fix-system-env-installer-paths
creating virtual environment...
creating shared libraries...
upgrading shared libraries...
determining package name from 'https://api.github.com/repos/abn/poetry/tarball/fix-system-env-installer-paths'...
creating virtual environment...
installing poetry from spec 'https://api.github.com/repos/abn/poetry/tarball/fix-system-env-installer-paths'...
  installed package poetry 1.8.0.dev0, installed using Python 3.11.5
  These apps are now globally available
    - poetry
done! ✨ 🌟 ✨
+ export POETRY_VIRTUALENVS_CREATE=false
+ POETRY_VIRTUALENVS_CREATE=false
+ poetry new demo
Created package demo in demo
+ pushd demo
/opt/airflow/demo /opt/airflow
+ poetry add cowsay
Skipping virtualenv creation, as specified in config file.
Using version ^6.1 for cowsay

Updating dependencies
Resolving dependencies...

Package operations: 1 install, 0 updates, 0 removals

  - Installing cowsay (6.1)

Writing lock file
+ pip show cowsay
Name: cowsay
Version: 6.1
Summary: The famous cowsay for GNU/Linux is now available for python
Home-page: https://github.com/VaasuDevanS/cowsay-python
Author: Vaasudevan Srinivasan
Author-email: vaasuceg.96@gmail.com
License: GNU-GPL
Location: /home/airflow/.local/lib/python3.11/site-packages
Requires: 
Required-by:

This change fixes an issue where when system environment is used as a destination for wheel installation as a user without access to system site packages, installation fails with permission error. Functionally, this now follows the same behaviour as when a package is installed to user site. Resolves: python-poetry#8408

dimbleby · 2024-02-25T09:14:54Z

src/poetry/utils/env/base_env.py

@@ -312,7 +349,9 @@ def run_pip(self, *args: str, **kwargs: Any) -> str:
    def run_python_script(self, content: str, **kwargs: Any) -> str:
        return self.run(
            self._executable,
-            "-I",
+            # if isolation is enabled for system env, user site would be


I suppose this reintroduces #6627, which was the reason for adding this flag

Good catch thanks. Let me see if there is a better way.

dimbleby · 2024-02-25T09:19:26Z

src/poetry/utils/env/base_env.py

+                    "data",
+                ]:
+                    if not is_dir_writable(path=Path(paths[key]), create=True):
+                        overrides[key] = paths[key].replace(


Is it ok to end up potentially with a mix of system and user directories here? This is not something I have looked into but I would have guessed you'd want all or nothing

pretty sure there is a feature request somewhere asking for --user type installs, perhaps it makes sense to do that instead, have this be explicit

would it get messy if eg you provide --user on install but forget it on update, or similar? I dont know.

I suspect that would confuse things more from the user perspective. Especially because we are predominantly servicing virtual environments.

Such a flag would in theory only be used when the user knows they are running in a system environment and does not have write access to the system environment. Hence, I'm not sure how valuable it might be as a cli option. Maybe a config option, I do not know.

If I understood the issues requesting --user, this would simply solve the root issue for them.

Is it ok to end up potentially with a mix of system and user directories here? This is not something I have looked into but I would have guessed you'd want all or nothing

In practice it shouldn't matter. And in reality it is highly unlikely that parts of it is writable and others not.

Python will look everywhere by default.

hmm. pip checks all directories when making the same decision I think - https://github.com/pypa/pip/blob/f4a543917eac822a3b06ab0fcc4df7c4e37e387a/src/pip/_internal/commands/install.py#L652. And then once it has made that decision it is fully committed to a user install rather than a site-wide install.

We can replicate that I guess.

dimbleby · 2024-02-25T09:20:18Z

src/poetry/utils/env/base_env.py

+                            base_path, paths["userbase"]
+                        )
+
+                for k, v in overrides.items():


paths.update(overrides)?

radoering · 2024-02-25T11:02:11Z

src/poetry/utils/env/base_env.py

@@ -242,6 +244,41 @@ def paths(self) -> dict[str, str]:

        return self._paths

+    @property


We may use @cached_property instead so we don't need _installer_scheme_dict.

dimbleby reviewed Feb 25, 2024

View reviewed changes

radoering reviewed Feb 25, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ensure writable paths are used for wheel installer #9014

Ensure writable paths are used for wheel installer #9014

abn commented Feb 25, 2024 •

edited

dimbleby Feb 25, 2024

abn Feb 25, 2024

dimbleby Feb 25, 2024

dimbleby Feb 25, 2024

abn Feb 25, 2024

abn Feb 25, 2024

dimbleby Feb 25, 2024 •

edited

abn Feb 25, 2024

dimbleby Feb 25, 2024

radoering Feb 25, 2024

		@@ -242,6 +244,41 @@ def paths(self) -> dict[str, str]:

		return self._paths

		@property

Ensure writable paths are used for wheel installer #9014

Are you sure you want to change the base?

Ensure writable paths are used for wheel installer #9014

Conversation

abn commented Feb 25, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dimbleby Feb 25, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abn commented Feb 25, 2024 •

edited

dimbleby Feb 25, 2024 •

edited