Remove bootstrap script

[jaraco]

Summary of changes

Remove bootstrap script and tox hacks and just let pip and PEP 517 do the work. This approach did not work on Python 2.7, but appears to work for the currently-supported Pythons.

Closes #1527.

Pull Request Checklist

• Changes have tests
• News fragment added in changelog.d/.
  (See documentation for details)

[jaraco]

Here's a demo of it building from Github:

~ $ pip-run git+https://github.com/pypa/setuptools@bugfix/1996-no-bootstrap -- -c "import importlib.metadata as md; print('version is', m
d.version('setuptools'))"
Collecting git+https://github.com/pypa/setuptools@bugfix/1996-no-bootstrap
  Cloning https://github.com/pypa/setuptools (to revision bugfix/1996-no-bootstrap) to /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-req-build-1vjydv5h
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Building wheels for collected packages: setuptools
  Building wheel for setuptools (PEP 517) ... done
  Created wheel for setuptools: filename=setuptools-51.3.3.post20210119-py3-none-any.whl size=786456 sha256=5fcbf50e7cb6d0344ad104dbdfd1ce359c010f55889c06493bf213f27bb67422
  Stored in directory: /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-ephem-wheel-cache-myawk3yi/wheels/43/32/cc/060ef20289520951a70cf3deec26d4b68d94732a96fe100811
Successfully built setuptools
Installing collected packages: setuptools
Successfully installed setuptools-51.3.3.post20210119
version is 51.3.3.post20210119

[jaraco]

One thing that occurred to me after submitting the PR - this change will be backward-incompatible to build tools that are explicitly running python -m bootstrap or similar. It may make sense to retain a file to satisfy that use-case and maybe raise a warning, but otherwise do nothing.

[jaraco]

As the 52.0 release already has a few breaking changes, I want to refrain from merging/releasing this change until the package has been stable for at least a week.

[hroncok]

Hello. Fedora maintainer here.

We don't yet have pip when we build setuptools for the first time for a new Python version.

And we cannot build pip without setuptools either.

I can try to work around it by using the pip wheel bundled in ensurepip Python module, but I am not sure it'll work :/

[jaraco]

Indeed. Perhaps what this project needs is an integration test that demonstrates how to install Setuptools from source. Even when you install using pip, there may be a problem that pip will download some other released version of setuptools to build the current version from source. Perhaps there's a better way. As a short-term mitigation, you're welcome to copy the 'bootstrap.py' script and invoke its behavior. Alternatively, if there's sufficient demand, this project can bring back that bootstrap script even if it's not used by Setuptools itself during pip-based workflows. Just confirm, and I'll make that happen quickly while working out a better long-term story to support installation from source.

[hroncok]

As a short-term mitigation, you're welcome to copy the 'bootstrap.py' script and invoke its behavior.

Doing that for now. Will follow #2550 for a long term solution.

[benoit-pierre]

Shouldn't the problem only occur when installing from VCS sources? The PyPI source distribution should work fine (egg-info directory already present), is that not what you're using @hroncok ?

[hroncok]

Right. We do build from PyPI source distribution and we delete the pre-generated egg-info first thing. I suppose I've just shot myself to the leg :D

[encukou]

We do build from PyPI source distribution

Right now we do, but long-term it might not be the best way. Sdists have slightly different purpose than Git sources: for example, there's not much reason to include tests or documentation in them.
Setuptools sdists do currently have both, but removing them it would be a legitimate decision. And in that case, Fedora would need to switch to VCS sources to get tests.

[FRidh]

Nixpkgs maintainer here. We will also keep using the bootstrap script as we do want to keep building from vcs sources.

[jonringer]

Setuptools sdists do currently have both, but removing them it would be a legitimate decision. And in that case, Fedora would need to switch to VCS sources to get tests.

Many packages also filter out tests because of the default behavior of find_packages(exclude=["tests"]). IIRC, you also need to do an include the test directory in the MANIFEST.in for sdist to retain the directory, but building a wheel will remove it. However, many packages do not do this, and usually remove it from both sdist and wheel.

Seems odd to me the python ecosystem can no longer bootstrap from vcs sources. (pip still requires setuptools, and setuptools now requires pip)

I think the removal of the bootstrap.py was a little premature until pip is able to self bootstrap. (If it ever will be able to self-bootstrap).

[jaraco]

Nixpkgs maintainer here. We will also keep using the bootstrap script as we do want to keep building from vcs sources.

Late versions of setuptools include the bootstrap metadata so can build from VCS sources naturally.

Many packages also filter out tests because of the default behavior of find_packages(exclude=["tests"]).

It's best to try to separate thinking of sdist and wheel into two phases, selection of sources for a source distribution, and installation from some source.

find_packages is about finding Python packages to be installed from some source (sdist or vcs checkout), so that will affect the result of what's installed, and wheel represents what is installed.

IIRC, you also need to do an include the test directory in the MANIFEST.in for sdist to retain the directory, but building a wheel will remove it. However, many packages do not do this, and usually remove it from both sdist and wheel.

The manifest is used to find sources, so useful only to the sdist production, so doesn't affect what's installed. There are other ways to affect finding sources (setuptools_scm for example).

Seems odd to me the python ecosystem can no longer bootstrap from vcs sources. (pip still requires setuptools, and setuptools now requires pip)

Pip doesn't require setuptools. It does vendor pkg_resources, but it runs fine without setuptools. I've been running with Setuptools uninstalled in my environments and it works well. When installing packages that implicitly depend on setuptools, I have to pass --use-pep517 to cause pip to implicitly require setuptools to build such packages, but otherwise, it just works.

Furthermore, setuptools does not depend on pip for the primary, long-term-supported use-cases. It does fall back to pip for setup_requires and other installation behavior, but that's for compatibility and slated to be fully deprecated in favor of PEP 517 builds.

I think the removal of the bootstrap.py was a little premature until pip is able to self bootstrap. (If it ever will be able to self-bootstrap).

Pip self bootstraps now through python -m ensurepip and wget https://bootstrap.pypa.io/get-pip.py -O - | python. No setuptools required.

draft $ cat Dockerfile
FROM ubuntu:focal
RUN apt update
RUN apt upgrade -y
RUN apt install -y python3 python3-distutils wget git
RUN wget https://bootstrap.pypa.io/get-pip.py -O - | python3 - --no-setuptools
CMD pip3 install git+https://github.com/pypa/setuptools
draft $ docker run -it @$(docker build -q .) python3 -c "import setuptools"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'setuptools'
draft $ docker run -it @$(docker build -q .)
Collecting git+https://github.com/pypa/setuptools
  Cloning https://github.com/pypa/setuptools to /tmp/pip-req-build-yx2ipnqh
  Running command git clone -q https://github.com/pypa/setuptools /tmp/pip-req-build-yx2ipnqh
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Building wheels for collected packages: setuptools
  Building wheel for setuptools (PEP 517) ... done
  Created wheel for setuptools: filename=setuptools-54.2.0.post20210323-py3-none-any.whl size=785287 sha256=01cdcdfcfb7017c94fd5719d61b55eae398584cecbdd778287ae07adb3e10ac7
  Stored in directory: /tmp/pip-ephem-wheel-cache-pzkpc2k_/wheels/ae/9f/bf/273815a79197c8bbb9027c9145ca5245c95e537b0e9db90af2
Successfully built setuptools
Installing collected packages: setuptools
  Attempting uninstall: setuptools
    Found existing installation: setuptools 54.2.0
    Uninstalling setuptools-54.2.0:
      Successfully uninstalled setuptools-54.2.0
Successfully installed setuptools-54.2.0.post20210323

[hroncok]

Late versions of setuptools include the bootstrap metadata so can build from VCS sources naturally.

Just a quick JFYI we've failed attempting to do this in the Fedora package and I still need investigation (or try harder).

[FRidh]

Late versions of setuptools include the bootstrap metadata so can build from VCS sources naturally.

Indeed, to build a setuptools sdist

python setup.py egg_info
python setup.py sdist --formats=gztar

NixOS/nixpkgs#130250

After that, to build an env with pip, setuptools and wheel

# Give folders a known name
mv pip* pip
mv setuptools* setuptools
mv wheel* wheel
# Set up PYTHONPATH. The above folders need to be on PYTHONPATH
# $out is where we are installing to and takes precedence
export PYTHONPATH="$out/${python.sitePackages}:$(pwd)/pip/src:$(pwd)/setuptools:$(pwd)/setuptools/pkg_resources:$(pwd)/wheel:$PYTHONPATH"

echo "Building setuptools wheel..."
pushd setuptools
rm pyproject.toml
${python.pythonForBuild.interpreter} -m pip install --no-build-isolation --no-index --prefix=$out  --ignore-installed --no-dependencies --no-cache .
popd

echo "Building wheel wheel..."
pushd wheel
${python.pythonForBuild.interpreter} -m pip install --no-build-isolation --no-index --prefix=$out  --ignore-installed --no-dependencies --no-cache .
popd

echo "Building pip wheel..."
pushd pip
${python.pythonForBuild.interpreter} -m pip install --no-build-isolation --no-index --prefix=$out  --ignore-installed --no-dependencies --no-cache .
popd