Debian-specific behavior lost when Setuptools adopts distutils

[prometheanfire]

pip install -e $PACKAGE_PATH installs commands to /usr/bin and not /usr/local/bin on Ubuntu as it did in the past.

pip install $PACKAGE_PATH continues to install to /usr/local/bin as expected.

Openstack downstream has temporarily capped setuptools until we know if this is expected behaviour or not. Please see http://lists.openstack.org/pipermail/openstack-discuss/2020-July/015779.html for more context.

[jaraco]

Thanks for the report. I'm initially unable to replicate the reported baseline behavior:

$ docker run -it jaraco/multipy-tox bash -c 'git clone -q https://github.com/pypa/twine; python3 -m pip-run -q "setuptools<48" -- -m pip install -q --no-deps -e twine; ls /usr/bin/twine /usr/local/bin/twine'
ls: cannot access '/usr/local/bin/twine': No such file or directory
/usr/bin/twine

note: multipy-tox is Ubuntu Focal with Python 3.8.2 installed

In that command, I use Docker to install setuptools<48 to then run pip install -e twine (a local checkout), which runs yet no command exist as /usr/local/bin/twine, but the command does exist as /usr/bin/twine.

So it seems maybe the discriminating factor is not the Setuptools 48 release (though its change is plausibly implicated). In Setuptools 49.1, I've (temporarily) disabled the functionality introduced in Setuptools 48. Does that correct the issue (in your environment)?

What other factors could be causing this unexpected behavior, even with older Setuptools? How can I replicate the issue?

[cboylan]

This shows the behavior:

sudo docker run -it --rm ubuntu:20.04 bash -c 'apt-get update && apt-get install -y python3-pip git && pip3 install -U pip && pip --version && pip install -U "setuptools<49" && git clone https://github.com/pypa/twine && pip install ./twine && which twine && pip uninstall -y twine && pip install -e ./twine && which twine'

Using setuptools 49.1.0 seems to fix it (the difference between the commands is the version of setuptools installed):

sudo docker run -it --rm ubuntu:20.04 bash -c 'apt-get update && apt-get install -y python3-pip git && pip3 install -U pip && pip --version && pip install -U "setuptools==49.1.0" && git clone https://github.com/pypa/twine && pip install ./twine && which twine && pip uninstall -y twine && pip install -e ./twine && which twine'

I won't bother to paste the output of the commands here because it is quite verbose, but if that would be useful I can add it as attachments I guess.

[jaraco]

I see the mistake with my example above; I was using pip-run to temporarily override the version of setuptools, but twine uses PEP 517 to build, so the build isolation causes that value to be discarded. If I'm using pip-run to override build requirements, I should also pass --no-build-isolation to pip install.

I'm able to quickly replicate the issue with this command:

$ docker run -it jaraco/multipy-tox bash -c 'git clone -q https://github.com/pypa/twine; python3 -m pip install -q "setuptools==48"; python3 -m pip install -q --no-deps -e twine; which twine'
/usr/bin/twine

[jaraco]

With this command, I'm able to replicate the behavior natively in setuptools:

$ docker run -it jaraco/multipy-tox bash -c 'git clone -q https://github.com/pypa/twine; cd twine; python3 -m pip install -q "setuptools==48"; python3 setup.py develop --no-deps; which twine'
running develop
running egg_info
creating twine.egg-info
writing twine.egg-info/PKG-INFO
writing dependency_links to twine.egg-info/dependency_links.txt
writing entry points to twine.egg-info/entry_points.txt
writing requirements to twine.egg-info/requires.txt
writing top-level names to twine.egg-info/top_level.txt
writing manifest file 'twine.egg-info/SOURCES.txt'
writing manifest file 'twine.egg-info/SOURCES.txt'
running build_ext
Creating /usr/lib/python3.8/site-packages/twine.egg-link (link to .)
Adding twine 3.2.1.dev1+g141428b to easy-install.pth file
Installing twine script to /usr/bin

Installed /twine
/usr/bin/twine

[jaraco]

It seems that Debian and Ubuntu keep their own forks of distutils. I copied /usr/lib/python3.8/distutils over setuptools/_distutils and found diffs like this:

diff --git a/setuptools/_distutils/command/install.py b/setuptools/_distutils/command/install.py
index 13feeb89..2934862f 100644
--- a/setuptools/_distutils/command/install.py
+++ b/setuptools/_distutils/command/install.py
@@ -30,33 +30,33 @@ WINDOWS_SCHEME = {
 INSTALL_SCHEMES = {
     'unix_prefix': {
         'purelib': '$base/lib/python$py_version_short/site-packages',
-        'platlib': '$platbase/$platlibdir/python$py_version_short/site-packages',
+        'platlib': '$platbase/lib/python$py_version_short/site-packages',
         'headers': '$base/include/python$py_version_short$abiflags/$dist_name',
         'scripts': '$base/bin',
         'data'   : '$base',
         },
+    'unix_local': {
+        'purelib': '$base/local/lib/python$py_version_short/dist-packages',
+        'platlib': '$platbase/local/lib/python$py_version_short/dist-packages',
+        'headers': '$base/local/include/python$py_version_short/$dist_name',
+        'scripts': '$base/local/bin',
+        'data'   : '$base/local',
+        },
+    'deb_system': {
+        'purelib': '$base/lib/python3/dist-packages',
+        'platlib': '$platbase/lib/python3/dist-packages',
+        'headers': '$base/include/python$py_version_short/$dist_name',
+        'scripts': '$base/bin',
+        'data'   : '$base',
+        },
     'unix_home': {
         'purelib': '$base/lib/python',
-        'platlib': '$base/$platlibdir/python',
+        'platlib': '$base/lib/python',
         'headers': '$base/include/python/$dist_name',
         'scripts': '$base/bin',
         'data'   : '$base',
         },
     'nt': WINDOWS_SCHEME,
-    'pypy': {
-        'purelib': '$base/site-packages',
-        'platlib': '$base/site-packages',
-        'headers': '$base/include/$dist_name',
-        'scripts': '$base/bin',
-        'data'   : '$base',
-        },
-    'pypy_nt': {
-        'purelib': '$base/site-packages',
-        'platlib': '$base/site-packages',
-        'headers': '$base/include/$dist_name',
-        'scripts': '$base/Scripts',
-        'data'   : '$base',
-        },
     }
 
 # user site schemes
@@ -145,6 +145,9 @@ class install(Command):
 
         ('record=', None,
          "filename in which to record list of installed files"),
+
+        ('install-layout=', None,
+         "installation layout to choose (known values: deb, unix)"),
         ]
 
     boolean_options = ['compile', 'force', 'skip-build']
@@ -165,6 +168,7 @@ class install(Command):
         self.exec_prefix = None
         self.home = None
         self.user = 0
+        self.prefix_option = None
 
         # These select only the installation base; it's up to the user to
         # specify the installation scheme (currently, that means supplying
@@ -186,6 +190,10 @@ class install(Command):
         self.install_userbase = USER_BASE
         self.install_usersite = USER_SITE
 
+        # enable custom installation, known values: deb
+        self.install_layout = None
+        self.multiarch = None
+        
         self.compile = None
         self.optimize = None
 
@@ -312,7 +320,6 @@ class install(Command):
                             'sys_exec_prefix': exec_prefix,
                             'exec_prefix': exec_prefix,
                             'abiflags': abiflags,
-                            'platlibdir': getattr(sys, 'platlibdir', 'lib'),
                            }
 
         if HAS_USER_SITE:
@@ -428,6 +435,7 @@ class install(Command):
             self.install_base = self.install_platbase = self.home
             self.select_scheme("unix_home")
         else:
+            self.prefix_option = self.prefix
             if self.prefix is None:
                 if self.exec_prefix is not None:
                     raise DistutilsOptionError(
@@ -442,7 +450,28 @@ class install(Command):
 
             self.install_base = self.prefix
             self.install_platbase = self.exec_prefix
-            self.select_scheme("unix_prefix")
+            if self.install_layout:
+                if self.install_layout.lower() in ['deb']:
+                    import sysconfig
+                    self.multiarch = sysconfig.get_config_var('MULTIARCH')
+                    self.select_scheme("deb_system")
+                elif self.install_layout.lower() in ['unix']:
+                    self.select_scheme("unix_prefix")
+                else:
+                    raise DistutilsOptionError(
+                        "unknown value for --install-layout")
+            elif ((self.prefix_option and
+                   os.path.normpath(self.prefix) != '/usr/local')
+                  or sys.base_prefix != sys.prefix
+                  or 'PYTHONUSERBASE' in os.environ
+                  or 'VIRTUAL_ENV' in os.environ
+                  or 'real_prefix' in sys.__dict__):
+                self.select_scheme("unix_prefix")
+            else:
+                if os.path.normpath(self.prefix) == '/usr/local':
+                    self.prefix = self.exec_prefix = '/usr'
+                    self.install_base = self.install_platbase = '/usr'
+                self.select_scheme("unix_local")
 
     def finalize_other(self):
         """Finalizes options for non-posix platforms"""
@@ -469,12 +498,6 @@ class install(Command):
     def select_scheme(self, name):
         """Sets the install directories by applying the install schemes."""
         # it's the caller's problem if they supply a bad name!
-        if (hasattr(sys, 'pypy_version_info') and
-                not name.endswith(('_user', '_home'))):
-            if os.name == 'nt':
-                name = 'pypy_nt'
-            else:
-                name = 'pypy'
         scheme = INSTALL_SCHEMES[name]
         for key in SCHEME_KEYS:
             attrname = 'install_' + key

So when Setuptools adopts distutils from the standard library and uses it, any tweaks that downstream packagers have made to distutils will be lost.

Where are these tweaks managed? Can I get some help from Debian and/or Ubuntu to apply these tweaks to distutils?

[pradyunsg]

Can I get some help from Debian and/or Ubuntu to apply these tweaks to distutils?

/cc @kitterma (one of the Debian Python folks)

[pradyunsg]

Also relevant: https://sources.debian.org/patches/python3.8/

Specifically: https://sources.debian.org/patches/python3.8/3.8.4%7Erc1-1/distutils-install-layout.diff/

[jaraco]

I took a quick look at the patch and it looks pretty manageable. However, the options seem to change the default behavior. What we would need for distutils is something that behaves the same as currently by default, but on Debian honors the alternate behaviors.

[doko42]

Making the installation into /usr the default is definitely not appreciated. I think we discussed this at length at one of the Cleveland sprints.

The behavior that you don't want to have is to install, remove, or update a package into /usr which was installed using the Debian package manager (apt/dpkg). Afaics this is now exactly the thing which happens with the current setuptools behavior. It's unfortunate that a change like this is again introduced. I don't think that the solution is to change any default for setuptools, but to make pip aware of dpkg/apt managed installations.

[jaraco]

Here, Setuptools is only relying on the distutils behavior from CPython. All that's needed is to port the Debian-specific behavior into pypa/distutils and (especially if there's a test capturing the expectation), Setuptools will ever-after support that behavior without the need for Debian to patch the software when installing it. If the Debian behavior is preferable for more environments, the project can consider employing it there as well.

To be certain, this change was not expected and was disabled within 24 hours. Now all we're seeking is to settle on a long-term solution in the pypa/distutils repo that can become the reference implementation without any downstream patches.

[jaraco]

I think we discussed this at length at one of the Cleveland sprints.

By the way, I remember the discussions and enjoyed our exchange. Unfortunately, so much time has passed that I haven't retained any actionable points from our discussion.

Do you have a recommendation? I don't think the diverse community of Setuptools users can readily adopt the Debian patch wholesale. Perhaps Setuptools/distutils could adopt those changes over time, giving other operating systems time to adapt to the approach Debian uses... or maybe those platforms have sound reasons for not having those changes. In any case, I believe the near-term solution, to unblock Setuptools from adopting distutils, would be for pypa/distutils to adopt the Debian patch but enable the Debian default settings only on Debian. Can you advise how to reliably detect hosts that would opt into this behavior?

[pradyunsg]

Unfortunately, so much time has passed that I haven't retained any actionable points from our discussion.

If I remember correctly...

The conclusion was that Debian and other Linux distros should stop shipping RECORD in dist-info folders for packages that they want to be managing (which renders pip unable to uninstall them, and thus won't do anything with them). This was a quick-fix-that-works-now and would have a suboptimal error message at the start which could be changed a bit in future releases and the change would be backported by the distros.

Another, longer-term approach for presenting a better error message would be to have some agreement on how to indicate a package as "externally managed" in the metadata. I'd initially proposed just using INSTALLER, but folks did flag that "externally managed" is different piece of metadata than "end user installer" (pip/pipenv/dpkg are valid INSTALLER values, but only 1 is externally managed, and pip shouldn't be keeping an allowlist of any kind) and that we just need to augment the information - either yet-another file or changing the contents of installer.

I do remember that someone from the Debian team did start working on a patch to update all the Debian packages to remove RECORD files at the Sprints, but idk if this ever rolled out. Some folks (not me!) did volunteer to start a discussion / PEP for the "externally managed" meradata. I don't think I ever saw a PEP draft for this, but I could be wrong.

[pradyunsg]

Oh, and for pip, adding an INSTALLER file that didn't contain "pip" was a cleaner patch than manipulating the code to disable the version check.

We needed these details to be documented in some form, and IIRC, one of the Linux distro folks (or me?) did volunteer to file a PR but I don't think that ever happened.

[pradyunsg]

This was of course, in the direction of distros stopping patches like debian distutils patch, in favor of better metadata to indicate these details and to have behaviors originating from upstream (like pip's pypa/pip#7002) that remove the likelihood of users getting PermissionError, overwritten files and things like that.

[jaraco]

Please have a look at #2240, which attempts to address #2232 by integrating the Debian patch and enabling its default behaviors on Debian only.

[doko42]

I don't like the idea of having "Debian only" code upstream. Would it be possible to add options instead, like --unversioned-eggs to do the same in a more generic way?

[jaraco]

In #2240, there seems to be some consensus that incorporating the Debian patches as debian-only is undesirable.

Here are the options I see:

• Simply re-enable the functionality with the known degradation for Debian systems. Let Debian maintainers patch Setuptools where possible but otherwise Debian users get the general/unpatched behavior. Possibly enhance the behavior later to satisfy Debian needs in a generic way.
• Defer enabling distutils adoption until a satisfactory solution can be implemented that supports Debian's needs and Debian is prepared to enable that functionality.
• Accept the PR to incorporate Debian-specific behavior and feature parity with Debian-patched distutils with an understanding that Debian maintainers will help develop a more permanent, general solution ASAP.

Thoughts? Other options?

[pganssle]

I may not fully understand the scope of who is affected, but my vote is for the first option. Debian (and other Debian-like systems) should be able to pin their adoption of setuptools until they have patches in place, and it's best to keep the roll-out moving so people on other distros can find their own regressions 😅.

[jaraco]

Without more feedback, Setuptools is very likely to re-release the behavior that led to this report with the recommendation for Debian users to pin to older Setuptools. I'm open to arguments or suggestions on how to smooth the transition for Debian users. If there's another person or forum for engaging on this issue, please direct us there.

[pradyunsg]

I'm generally in favor of option 1, although I want our primary intent to be to work with Debian maintainers to improve the situation ASAP.

With my limited understanding of the history here (and nuances here due to it), I think a lot of the recent changes in Python Packaging tooling have made a lot of debian's patches to Python packaging tools in general to be no-longer-necessary (or at least very-defensive) and I am generally in favor of Linux distros dropping their patches on Python Packaging tooling, and collaborating with us to come up with more general upstream solutions instead.

[cboylan]

Without more feedback, Setuptools is very likely to re-release the behavior that led to this report with the recommendation for Debian users to pin to older Setuptools. I'm open to arguments or suggestions on how to smooth the transition for Debian users. If there's another person or forum for engaging on this issue, please direct us there.

Is it a problem that pip install and pip install -e will install to different path prefixes? Consistency there would be good even if the path prefix ultimately changes.

[jaraco]

Is it a problem that pip install and pip install -e will install to different path prefixes? Consistency there would be good even if the path prefix ultimately changes.

I agree it would be nice if there was consistency and maybe there's a deficiency in Setuptools that could be addressed here.

I don't have a grasp of the design here, or even what is intentionally designed versus a defacto approach that's evolved over time.

I do see that pip relies on stdlib distutils at least in some scenarios.

And in fact, if you enable the distutils patch by way of importing setuptools before invoking pip, pip also exhibits the same behavior:

$ docker run -it jaraco/multipy-tox sh -c 'python -m pip-run -q setuptools==49.0.1 -- -c "import setuptools; from pip._internal.cli.main import main; main()" install -q twine && which twine'
/usr/bin/twine

So it does seem that when distutils is deprecated in the standard library and pip begins to use distutils as supplied by either pypa/distutils or Setuptools, pip also will begin installing to /usr/bin on Debian. So the behavior is theoretically consistent; it's just out of sync.

I found pypa/virtualenv#1770 which has some good leads for expanding the visibility of this issue and with any luck to garner more support around addressing the issue properly. I plan to reach out on IRC to let others know this change is coming.

I'm going to draft the PR for restoring distutils adoption by default, which will re-apply this undesirable effect for editable installs in Debian, but I'm hopeful that a long term solution can be developed against pypa/distutils#2.

[jaraco]

I've reached out to the Debian community on IRC. @prometheanfire Can you advise the OpenStack community on this change and ensure that a Setuptools pin or the environment variable escape hatch will be suitable to limit any damage?

[prometheanfire]

sure, so it sounds like we are rolling forward with the original behavior for now?

[Kixunil]

Can someone explain to me why this is Debian-related? Putting stuff in /usr goes against Linux FHS, which explicitly states:

The /usr/local hierarchy is for use by the system administrator when installing software locally. It needs to be safe from being overwritten when the system software is updated. It may be used for programs and data that are shareable amongst a group of hosts, but not found in /usr.

Locally installed software must be placed within /usr/local rather than /usr unless it is being installed to replace or upgrade software in /usr.

So putting anything outside /usr/local can cause trouble on all Linux distributions, not just Debian.

Sorry if I'm missing something, I'm newbie in setuptools.

[jaraco]

It's possible that other distros have their own fork of distutils and that this issue affects those distros (in likely different ways). I welcome feedback about the behavior and investigation into other Linux distros as well. As a tool, Setuptools is committed to honoring the best practices, and that includes limiting these downstream patches to repair upstream deficiencies.

[mikepurvis]

I'm just a random member of the public, but have done some work on distributing Python packages in the context of Robot Operating System (ROS), which sits on top of Ubuntu LTS. I think Debian's historical policy of wanting apt-supplied Python to use dist-packages and source/upstream Python to use site-packages made a lot more sense in the days when ABI compatibility between minor versions was not as well defined. I don't have a firm grasp of what the fallout would be from them dropping that at this point, but I've been exposed to a bunch of packaging scenarios where this does have to be manually detected and hacked around, for example google/or-tools#2107 and logic like this:

https://github.com/ros/catkin/blob/dfc93c72d9799f57442d08c2f6e9fd494bba95c1/cmake/python.cmake#L16-L29

All that said, I don't think it's realistic in the short- or medium term for the PyPA to make this purely a Debian problem. Every time it's run, apt-installed pip visits the internet and detects the availability of newer pip versions, encouraging the user to install them on top of the system-supplied versions as an overlay, so that the whole Python ecosystem is "up to date". This is reasonable to want to do, but you're also wresting away control of the local pip/python environment from the Debian maintainers and their ability to patch in the behaviour that they feel works best with apt/dpkg.

[merwok]

Without more feedback, Setuptools is very likely to re-release the behavior that led to this report with the recommendation for Debian users to pin to older Setuptools. I'm open to arguments or suggestions on how to smooth the transition for Debian users. If there's another person or forum for engaging on this issue, please direct us there.

Can you define «Debian users» here? There are at least three contexts:

• deb package building: something I don’t see directly, and uses bespoke options from debian-specific patches to change install locations
• people installing Python packages to /usr/local: the dist/site-packages split avoids local installs disturbing the system Python, so fine
• people installing Python packages to virtualenvs: no reason to deviate from regular options here

[encukou]

I only found out about this issue now :( Fedora is also affected.
Is there a supported way to make distutils install to /usr/local/ by default, but still allow building installing into /usr/ (for system packages)?
For reference, the Fedora patch is here.

cc @hroncok

[Kixunil]

@encukou looks like I'm right. Putting stuff in /usr breaks FHS and thus all distros.

[encukou]

It's not really about that – FHS is a good guideline with helpful ideas/consderations/rationales, but lately it's not a great standard to follow literally.
Rather, the main reason we do this is to split sudo pip-installed packages from system-installed ones, so you can't break your system as easily with sudo pip. (Even though sudo pip is unsupported and a bad idea, the Internet is full of "tutorials" that recommend it).
System tools (like the package manager) run Python with -I, and end up with /usr/local/ not being on sys.path, so they don't see packages installed/updated with sudo pip. Details here.

[merwok]

For the record, Debian wants to support a local administrator installing compatible packages to /usr/local, so there is an additional tweak where the system Python looks at a dist-packages directory but pip installs go to site-packages, so that pip can’t break system tools.

[Kixunil]

but lately it's not a great standard to follow literally.

Why? What changed?

[pradyunsg]

Every time it's run, apt-installed pip visits the internet and detects the availability of newer pip versions, encouraging the user to install them on top of the system-supplied versions as an overlay

What you're describing is an issue that's been fixed for quite a while in pip itself, but the fix is taking time to propagate through the ecosystem; given the long support cycles that distros have.

If distro repackagers modify INSTALLER file in pip's metadata folder to anything other than pip, which will disable this message. IIUC, every major distro (that I've interacted with directly) does this and the message is indeed suppressed on the newer versions of those.