Restore Debian support

[jaraco]

As reported in #2232, this patch incorporates the patch from Debian to distutils and detects Debian to apply that behavior at run time.

Closes #2232.

[pganssle]

I really don't like putting distro-specific code in upstream like this. If debian wants non-standard behavior they should carry the patch. Otherwise if there is a feature they need they should propose it like anyone else and it should be judged on the normal criteria that we judge features.

[pganssle]

Also this is all stuff that only gets triggered when you do setup.py install instead of installing from a wheel, right? That seems firmly in "unsupported" territory.

[jaraco]

Thank you Paul for the sanity check.

I really don't like putting distro-specific code in upstream like this. If debian wants non-standard behavior they should carry the patch. Otherwise if there is a feature they need they should propose it like anyone else and it should be judged on the normal criteria that we judge features.

I agree with you and had the same reaction initially.

But as I'm thinking about #2232 and what recommendation I could make, I struggled with saying that Setuptools should just break Debian/Ubuntu installs. Debian has already patched distutils, but the patch is bypassed by the fact that Setuptools is carrying an unpatched copy of distutils. I don't see how Debian could reasonably expect to patch an implementation that's typically installed by pip or other package managers outside of their control, across multiple Python versions, and across many (sometimes virtual) environments.

Setuptools' adoption of distutils is currently blocked on this issue, so I'm seeking some solution that can be implemented in the near term and this approach seemed like the most straightforward. In addition to unblocking distutils adoption, it also captures in a single codebase the commonality and divergence of Debian behaviors, and by being in a high velocity project like Setuptools, can be rapidly iterated/improved. In other words, acceptance of this patch should facilitate the broader adoption or deprecation and ultimate convergence of these divergent behaviors. I do hope/imagine that Debian might straight-up adopt pypa/distutils instead of stdlib distutils in order to avoid patching.

I will say that Debian holds some culpability for this issue, but I don't think the Debian users should bear the cost of the somewhat hostile monkeypatch. I don't know the history of this patch in Debian and I wouldn't be surprised if it was the PyPA or similar that recommended or forced Debian's approach.

Also this is all stuff that only gets triggered when you do setup.py install instead of installing from a wheel, right?

Unfortunately, no. This behavior affects editable installs (in particular where scripts are installed).

I can imagine an alternate scenario where Setuptools could first start honoring the the Debian expectation (use unix-local by default), though I don't fully understand the implications of that change. My instinct is that ultimately that's the right thing to do, but I was hoping to defer reconciling that divergence until after the distutils code was adopted and the distutils/setuptools codebases were largely unified.

I do believe the setup.py install case may also be relevant in the situation where the PyPA deprecates distutils, removes it from stdlib, and declares setuptools as the replacement. Hopefully with PEP 517 and a solution for editable installs, that's a non-issue.

Where can we go from here?

[jaraco]

On other small point - I want to avoid the situation where the distutils adoption is blocked again by some other challenging feature (i.e. editable install revamp). This effort stalled once and the work got stale. I'd like to come out of this having distutils in Setuptools in O(weeks) so that the work doesn't grow stale and so that more progress can be made on merging the functionality.

[pganssle]

But as I'm thinking about #2232 and what recommendation I could make, I struggled with saying that Setuptools should just break Debian/Ubuntu installs. Debian has already patched distutils, but the patch is bypassed by the fact that Setuptools is carrying an unpatched copy of distutils. I don't see how Debian could reasonably expect to patch an implementation that's typically installed by pip or other package managers outside of their control, across multiple Python versions, and across many (sometimes virtual) environments.

I don't think they should be patching this in the first place, but the question is why does debian need this at all? Why can't they do what every other distro is doing?

Also, this change only applies to the system python. I realize end users use the system python with non-system-installed packages all the time, but they shouldn't, because it causes all kinds of problems. If they're using system packages, they'll use a Debian-supplied setuptools which carries the patch. I would really hate it if I were forced to use a Debian-derived distro and I couldn't create a normal Python environment by using pyenv etc because setuptools now has some Debian-specific behavior in it.

I will say that Debian holds some culpability for this issue, but I don't think the Debian users should bear the cost of the somewhat hostile monkeypatch. I don't know the history of this patch in Debian and I wouldn't be surprised if it was the PyPA or similar that recommended or forced Debian's approach.

I understand that, but this is what got us into this mess in the first place. Debian wanted some other default behavior and so they patched distutils for whatever reason rather than letting it be fixed in the right place. Now we're going to carry some Debian-specific behavior (which should increase our test matrix to include debian and non-debian flavored tools) instead of fixing it in the right place and in the right way — whatever that is.

Unfortunately, no. This behavior affects editable installs (in particular where scripts are installed).

I'm not certain of the details here, but from what I can tell this will only affect things like sudo pip install -e package, which is just a recipe for breaking your Python installation anyway. If for some reason package managers are using pip install -e, they should probably stop, and again they would use the system package, which can carry whatever patches they want.

At the end of the day, I am not convinced that #2232 needs to be fixed on our end, and it doesn't really need to block our releases.

On other small point - I want to avoid the situation where the distutils adoption is blocked again by some other challenging feature (i.e. editable install revamp). This effort stalled once and the work got stale. I'd like to come out of this having distutils in Setuptools in O(weeks) so that the work doesn't grow stale and so that more progress can be made on merging the functionality.

I don't think you need to worry about editable install revamp blocking anything on the timescale of weeks. I was so thoroughly disappointed by the last round of conversations about this that I'm not totally convinced it will happen at all.

[jaraco]

I would really hate it if I were forced to use a Debian-derived distro and I couldn't create a normal Python environment by using pyenv etc because setuptools now has some Debian-specific behavior in it.

That's a really good point that I hadn't considered, though I would imagine this risk is at least an order of magnitude less problematic than the one at hand.

[jaraco]

After further consideration, I'm going to decline this request. I'm also going to go back to pypa/distutils and reset master to the state without this patch.