Exclude /usr/lib/python3.*/EXTERNALLY-MANAGED to fix system-wide pip install due to PEP-668

[em-]

The Python PEP-668 introduces a big breaking change by forbidding the system-wide installation of Python packages via pip.

All the many many Dockerfiles out there running pip install are now broken as soon as they try to move to Bookworm-based image:

WARNING: The directory '/github/home/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
    python3-xyz, where xyz is the package you are trying to
    install.
    
    If you wish to install a non-Debian-packaged Python package,
    create a virtual environment using python3 -m venv path/to/venv.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
    sure you have python3-full installed.
    
    If you wish to install a non-Debian packaged Python application,
    it may be easiest to use pipx install xyz, which will manage a
    virtual environment for you. Make sure you have pipx installed.
    
    See /usr/share/doc/python3.11/README.venv for more information.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.

The goal is to prevent issues when mixing and matching distro-managed packages and pip-installed ones.

However, this is not really an issue for containers. Citing the use-case #5 in PEP-668 itself:

... builders of base container images may want to ensure that the marker file is not present, even if the underlying OS ships one by default.

The suggestion is thus for Docker containers to remove the /usr/lib/python3.11/EXTERNALLY-MANAGED file.

Rather than having everyone rm it, could the base container ship a dpkg exclusion to prevent any /usr/lib/python3.*/EXTERNALLY-MANAGED files from being installed?

[yosifkit]

I don't agree that we should apply this generally in the Debian images. Users of a system installed python should be aware that they are possibly breaking system tools. Users of the python images are already using a manually installed, non-distro python so they are unaffected (if they are, then they are using the distro python and ought to be aware that it might be a different version than they expected).

It is the view of the authors of this PEP that it’s still a good idea to use virtual environments with distro-installed Python interpreters, even in single-application container images. Even though they run a single application, that application may run commands from the OS that are implemented in Python, and if you’ve installed or upgraded the distro-shipped Python packages using Python-specific tools, those commands may break.

[tianon]

I'm leaning towards agreeing that this is probably not something we should set for the general case; the workarounds for users to "opt into" the breakage are relatively unobtrusive while still pretty clearly communicating user intent (adding --break-system-packages to their pip install lines or adding something like rm -f /usr/lib/python*/EXTERNALLY-MANAGED to their RUN lines somewhere after Python is installed).

My concern is primarily around what the PEP describes as "single-application containers", which is the ideal mode for Docker container images, but I don't think it's fair to say that it's how everyone is using the debian:xxx images. 😅

[tianon]

Oh, this can be set via environment variable too: ENV PIP_BREAK_SYSTEM_PACKAGES=yes