Re-enable distutils patch by default. #2255

jaraco · 2020-07-12T15:00:22Z

Closes #417
Closes #2232

I plan to release this change no sooner than Tuesday, Jul 14, giving downstream packagers at least some time to pin Setuptools or implement the escape hatch.

Pull Request Checklist

News fragment added in changelog.d. See documentation for details

jaraco · 2020-08-14T02:13:55Z

This change is still failing two tests, both on late versions of pip - one against the current release (whatever virtualenv provides, 20.2.1) and the other against master.

However, when I try to replicate the behavior by repeating the steps the tests are doing, I'm unable to do so. Everything works fine. I'm going to have to dig deeper.

jaraco · 2020-08-14T13:42:52Z

It seems with this behavior enabled by default (or maybe enabled at all?), late versions of pip are unable to install setuptools over setuptools:

setuptools bugfix/2232-adopt-distutils-default $ env/bin/pip install dist/*.whl
Processing ./dist/setuptools-49.5.0.post20200813-py3-none-any.whl
Installing collected packages: setuptools
Successfully installed setuptools-49.5.0.post20200813
WARNING: You are using pip version 20.2.1; however, version 20.2.2 is available.
You should consider upgrading via the '/Users/jaraco/code/public/pypa/setuptools/env/bin/python -m pip install --upgrade pip' command.
setuptools bugfix/2232-adopt-distutils-default $ env/bin/pip install --upgrade dist/*.whl
Processing ./dist/setuptools-49.5.0.post20200813-py3-none-any.whl
Installing collected packages: setuptools
  Attempting uninstall: setuptools
    Found existing installation: setuptools 49.5.0.post20200813
    Uninstalling setuptools-49.5.0.post20200813:
      Successfully uninstalled setuptools-49.5.0.post20200813
  Rolling back uninstall of setuptools
  Moving to /Users/jaraco/code/public/pypa/setuptools/env/bin/easy_install
   from /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-uninstall-cs1344mf/easy_install
  Moving to /Users/jaraco/code/public/pypa/setuptools/env/bin/easy_install-3.8
   from /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-uninstall-cs1344mf/easy_install-3.8
  Moving to /Users/jaraco/code/public/pypa/setuptools/env/lib/python3.8/site-packages/__pycache__/easy_install.cpython-38.pyc
   from /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-uninstall-tqtytie5/easy_install.cpython-38.pyc
  Moving to /Users/jaraco/code/public/pypa/setuptools/env/lib/python3.8/site-packages/_distutils_hack/
   from /Users/jaraco/code/public/pypa/setuptools/env/lib/python3.8/site-packages/~distutils_hack
  Moving to /Users/jaraco/code/public/pypa/setuptools/env/lib/python3.8/site-packages/distutils-precedence.pth
   from /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-uninstall-0mp4q7u8/distutils-precedence.pth
  Moving to /Users/jaraco/code/public/pypa/setuptools/env/lib/python3.8/site-packages/easy_install.py
   from /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-uninstall-0mp4q7u8/easy_install.py
  Moving to /Users/jaraco/code/public/pypa/setuptools/env/lib/python3.8/site-packages/pkg_resources/
   from /Users/jaraco/code/public/pypa/setuptools/env/lib/python3.8/site-packages/~kg_resources
  Moving to /Users/jaraco/code/public/pypa/setuptools/env/lib/python3.8/site-packages/setuptools-49.5.0.post20200813.dist-info/
   from /Users/jaraco/code/public/pypa/setuptools/env/lib/python3.8/site-packages/~etuptools-49.5.0.post20200813.dist-info
  Moving to /Users/jaraco/code/public/pypa/setuptools/env/lib/python3.8/site-packages/setuptools/
   from /Users/jaraco/code/public/pypa/setuptools/env/lib/python3.8/site-packages/~etuptools
ERROR: Exception:
Traceback (most recent call last):
  File "/Users/jaraco/code/public/pypa/setuptools/env/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 216, in _main
    status = self.run(options, args)
  File "/Users/jaraco/code/public/pypa/setuptools/env/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 182, in wrapper
    return func(self, options, args)
  File "/Users/jaraco/code/public/pypa/setuptools/env/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 412, in run
    installed = install_given_reqs(
  File "/Users/jaraco/code/public/pypa/setuptools/env/lib/python3.8/site-packages/pip/_internal/req/__init__.py", line 82, in install_given_reqs
    requirement.install(
  File "/Users/jaraco/code/public/pypa/setuptools/env/lib/python3.8/site-packages/pip/_internal/req/req_install.py", line 778, in install
    scheme = get_scheme(
  File "/Users/jaraco/code/public/pypa/setuptools/env/lib/python3.8/site-packages/pip/_internal/locations.py", line 185, in get_scheme
    scheme = distutils_scheme(
  File "/Users/jaraco/code/public/pypa/setuptools/env/lib/python3.8/site-packages/pip/_internal/locations.py", line 125, in distutils_scheme
    i.finalize_options()
  File "/Users/jaraco/code/public/pypa/setuptools/env/lib/python3.8/site-packages/setuptools/_distutils/command/install.py", line 381, in finalize_options
    self.set_undefined_options('build',
  File "/Users/jaraco/code/public/pypa/setuptools/env/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 286, in set_undefined_options
    src_cmd_obj = self.distribution.get_command_obj(src_cmd)
  File "/Users/jaraco/code/public/pypa/setuptools/env/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 858, in get_command_obj
    klass = self.get_command_class(command)
  File "/Users/jaraco/code/public/pypa/setuptools/env/lib/python3.8/site-packages/setuptools/dist.py", line 771, in get_command_class
    return _Distribution.get_command_class(self, command)
  File "/Users/jaraco/code/public/pypa/setuptools/env/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 844, in get_command_class
    raise DistutilsModuleError("invalid command '%s'" % command)
distutils.errors.DistutilsModuleError: invalid command 'build'
WARNING: You are using pip version 20.2.1; however, version 20.2.2 is available.
You should consider upgrading via the '/Users/jaraco/code/public/pypa/setuptools/env/bin/python -m pip install --upgrade pip' command.

jaraco · 2020-08-14T13:47:26Z

The issue can be replicated in the currently-released version of setuptools by setting SETUPTOOLS_USE_DISTUTILS='local'.

draft $ $SETUPTOOLS_USE_DISTUTILS='local'
draft $ virtualenv env
created virtual environment CPython3.8.5.final.0-64 in 278ms
  creator CPython3Posix(dest=/Users/jaraco/draft/env, clear=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/jaraco/Library/Application Support/virtualenv)
    added seed packages: pip==20.2.1, setuptools==49.2.1, wheel==0.34.2
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
draft $ env/bin/pip install --upgrade https://files.pythonhosted.org/packages/c3/a9/5dc32465951cf4812e9e93b4ad2d314893c2fa6d5f66ce5c057af6e7
6d85/setuptools-49.6.0-py3-none-any.whl
Collecting setuptools==49.6.0
  Using cached setuptools-49.6.0-py3-none-any.whl (803 kB)
Installing collected packages: setuptools
  Attempting uninstall: setuptools
    Found existing installation: setuptools 49.2.1
    Uninstalling setuptools-49.2.1:
      Successfully uninstalled setuptools-49.2.1
Successfully installed setuptools-49.6.0
draft $ env/bin/pip install --upgrade https://files.pythonhosted.org/packages/c3/a9/5dc32465951cf4812e9e93b4ad2d314893c2fa6d5f66ce5c057af6e7
6d85/setuptools-49.6.0-py3-none-any.whl
Collecting setuptools==49.6.0
  Using cached setuptools-49.6.0-py3-none-any.whl (803 kB)
Installing collected packages: setuptools
  Attempting uninstall: setuptools
    Found existing installation: setuptools 49.6.0
    Uninstalling setuptools-49.6.0:
      Successfully uninstalled setuptools-49.6.0
  Rolling back uninstall of setuptools
  Moving to /Users/jaraco/draft/env/bin/easy_install
   from /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-uninstall-kg4fawtu/easy_install
  Moving to /Users/jaraco/draft/env/bin/easy_install-3.8
   from /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-uninstall-kg4fawtu/easy_install-3.8
  Moving to /Users/jaraco/draft/env/lib/python3.8/site-packages/__pycache__/easy_install.cpython-38.pyc
   from /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-uninstall-t335cjpw/easy_install.cpython-38.pyc
  Moving to /Users/jaraco/draft/env/lib/python3.8/site-packages/_distutils_hack/
   from /Users/jaraco/draft/env/lib/python3.8/site-packages/~distutils_hack
  Moving to /Users/jaraco/draft/env/lib/python3.8/site-packages/distutils-precedence.pth
   from /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-uninstall-f8btu2q3/distutils-precedence.pth
  Moving to /Users/jaraco/draft/env/lib/python3.8/site-packages/easy_install.py
   from /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-uninstall-f8btu2q3/easy_install.py
  Moving to /Users/jaraco/draft/env/lib/python3.8/site-packages/pkg_resources/
   from /Users/jaraco/draft/env/lib/python3.8/site-packages/~kg_resources
  Moving to /Users/jaraco/draft/env/lib/python3.8/site-packages/setuptools-49.6.0.dist-info/
   from /Users/jaraco/draft/env/lib/python3.8/site-packages/~etuptools-49.6.0.dist-info
  Moving to /Users/jaraco/draft/env/lib/python3.8/site-packages/setuptools/
   from /Users/jaraco/draft/env/lib/python3.8/site-packages/~etuptools
ERROR: Exception:
Traceback (most recent call last):
  File "/Users/jaraco/draft/env/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 216, in _main
    status = self.run(options, args)
  File "/Users/jaraco/draft/env/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 182, in wrapper
    return func(self, options, args)
  File "/Users/jaraco/draft/env/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 412, in run
    installed = install_given_reqs(
  File "/Users/jaraco/draft/env/lib/python3.8/site-packages/pip/_internal/req/__init__.py", line 82, in install_given_reqs
    requirement.install(
  File "/Users/jaraco/draft/env/lib/python3.8/site-packages/pip/_internal/req/req_install.py", line 778, in install
    scheme = get_scheme(
  File "/Users/jaraco/draft/env/lib/python3.8/site-packages/pip/_internal/locations.py", line 185, in get_scheme
    scheme = distutils_scheme(
  File "/Users/jaraco/draft/env/lib/python3.8/site-packages/pip/_internal/locations.py", line 125, in distutils_scheme
    i.finalize_options()
  File "/Users/jaraco/draft/env/lib/python3.8/site-packages/setuptools/_distutils/command/install.py", line 381, in finalize_options
    self.set_undefined_options('build',
  File "/Users/jaraco/draft/env/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 286, in set_undefined_options
    src_cmd_obj = self.distribution.get_command_obj(src_cmd)
  File "/Users/jaraco/draft/env/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 858, in get_command_obj
    klass = self.get_command_class(command)
  File "/Users/jaraco/draft/env/lib/python3.8/site-packages/setuptools/dist.py", line 771, in get_command_class
    return _Distribution.get_command_class(self, command)
  File "/Users/jaraco/draft/env/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 844, in get_command_class
    raise DistutilsModuleError("invalid command '%s'" % command)
distutils.errors.DistutilsModuleError: invalid command 'build'

…ypa/pip#8761.

jaraco · 2020-08-14T23:51:25Z

In the branch bugfix/2232-just-break-pip-upgrade, I've pushed 6e3aa8d that simply fixes the failing tests and leaves it to users to do the same if they're upgrading setuptools in pip. That approach seems suboptimal, so be7a0a3 takes another approach, bypassing the DistutilsMetaFinder when running under pip. As this approach passes the tests, it should address the issue.

jaraco · 2020-08-14T23:59:27Z

Ugh. The workaround doesn't work on PyPy. :(

jaraco · 2020-08-15T02:36:09Z

I don't understand why it's failing. It's passing for me locally (pypy3 on macOS). I guess I'll try to replicate the failure on an Ubuntu pypy system.

jaraco · 2020-08-15T03:34:47Z

After checking out the right branch, I was able to replicate the failure locally:

setuptools bugfix/2232-adopt-distutils-default $ git-id
be7a0a31a
setuptools bugfix/2232-adopt-distutils-default $ cat Dockerfile
FROM ubuntu:xenial
RUN apt update
RUN apt install -y software-properties-common
RUN add-apt-repository ppa:pypy/ppa
RUN apt update
RUN apt install -y virtualenv pypy3 python3-pip git
RUN pip3 install tox tox-pip-version
RUN virtualenv --python pypy3 /env
setuptools bugfix/2232-adopt-distutils-default $ docker run -v @(os.getcwd()):/setuptools -it @($(docker build -q .).strip()) bash -
c "cd /setuptools && tox --workdir ~/.tox -e pypy3 -- -k pip_upgrade_from_source"
pypy3 create: /root/.tox/pypy3
pypy3: pip_version is pip
pypy3 develop-inst: /setuptools
pypy3 installed: apipkg==1.5,appdirs==1.4.4,attrs==19.3.0,cffi==1.14.0,contextlib2==0.6.0.post1,coverage==5.2.1,distlib==0.3.1,execnet==1.7.1,filelock==3.0.12,flake8==3.8.3,flake8-2020==1.6.0,greenlet==0.4.13,importlib-metadata==1.7.0,importlib-resources==3.0.0,iniconfig==1.0.1,jaraco.envs==2.0.0,mccabe==0.6.1,mock==4.0.2,more-itertools==8.4.0,packaging==20.4,path==15.0.0,path.py==12.5.0,Paver==1.3.4,pip==20.2.2,pluggy==0.13.1,py==1.9.0,pycodestyle==2.6.0,pyflakes==2.2.0,pyparsing==2.4.7,pytest==6.0.1,pytest-cov==2.10.1,pytest-fixture-config==1.7.0,pytest-flake8==1.0.6,pytest-shutil==1.7.0,pytest-virtualenv==1.7.0,readline==6.2.4.1,-e git+gh://pypa/setuptools@be7a0a31a116f6df9bef616ef1adef72b9d0d48d#egg=setuptools,six==1.15.0,termcolor==1.1.0,toml==0.10.1,tox==3.19.0,tox-venv==0.4.0,virtualenv==20.0.30,wheel==0.34.2,zipp==3.1.0
pypy3 run-test-pre: PYTHONHASHSEED='1620508382'
pypy3 run-test: commands[0] | pytest -k pip_upgrade_from_source
======================================================= test session starts ========================================================
platform linux -- Python 3.6.9[pypy-7.3.1-final], pytest-6.0.1, py-1.9.0, pluggy-0.13.1
cachedir: /root/.tox/pypy3/.pytest_cache
rootdir: /setuptools, configfile: pytest.ini
plugins: cov-2.10.1, flake8-1.0.6, virtualenv-1.7.0, shutil-1.7.0
collected 819 items / 813 deselected / 2 skipped / 4 selected                                                                      

setuptools/tests/test_virtualenv.py F....F                                                                                   [100%]

============================================================= FAILURES =============================================================
________________________________________________ test_pip_upgrade_from_source[None] ________________________________________________

pip_version = None, virtualenv = <pytest_virtualenv.VirtualEnv object at 0x00007f707904fd00>

    @pytest.mark.parametrize('pip_version', _get_pip_versions())
    def test_pip_upgrade_from_source(pip_version, virtualenv):
        """
        Check pip can upgrade setuptools from source.
        """
        # Install pip/wheel, and remove setuptools (as it
        # should not be needed for bootstraping from source)
        if pip_version is None:
            upgrade_pip = ()
        else:
            upgrade_pip = ('python -m pip install -U {pip_version} --retries=1',)
        virtualenv.run(' && '.join((
            'pip uninstall -y setuptools',
            'pip install -U wheel',
        ) + upgrade_pip).format(pip_version=pip_version))
        dist_dir = virtualenv.workspace
        # Generate source distribution / wheel.
        virtualenv.run(' && '.join((
            'python setup.py -q sdist -d {dist}',
            'python setup.py -q bdist_wheel -d {dist}',
        )).format(dist=dist_dir), cd=SOURCE_DIR)
        sdist = glob.glob(os.path.join(dist_dir, '*.zip'))[0]
        wheel = glob.glob(os.path.join(dist_dir, '*.whl'))[0]
        # Then update from wheel.
        virtualenv.run('pip install ' + wheel)
        # And finally try to upgrade from source.
>       virtualenv.run('pip install --no-cache-dir --upgrade ' + sdist)

setuptools/tests/test_virtualenv.py:123: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/root/.tox/pypy3/site-packages/pytest_virtualenv.py:151: in run
    return super(VirtualEnv, self).run(args, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <pytest_virtualenv.VirtualEnv object at 0x00007f707904fd00>
cmd = 'pip install --no-cache-dir --upgrade /tmp/tmp1w19zqu3/setuptools-49.5.0.post20200815.zip', capture = False, check_rc = True
cd = Path('/tmp/tmp1w19zqu3'), shell = True
kwargs = {'env': {'COVERAGE_FILE': '/root/.tox/.coverage.pypy3', 'PATH': '/tmp/tmp1w19zqu3/.env/bin:/root/.tox/pypy3/bin:/usr/l...setuptools/tests/test_virtualenv.py::test_pip_upgrade_from_source[None] (setup)', 'PYTHONHASHSEED': '1620508382', ...}}
p = <subprocess.Popen object at 0x00007f707904fe50>, out = None, _ = None
err = CalledProcessError(2, 'pip install --no-cache-dir --upgrade /tmp/tmp1w19zqu3/setuptools-49.5.0.post20200815.zip')

    def run(self, cmd, capture=False, check_rc=True, cd=None, shell=False, **kwargs):
        """
        Run a command relative to a given directory, defaulting to the workspace root
    
        Parameters
        ----------
        cmd : `str` or `list`
            Command string or list. Commands given as a string will be run in a subshell.
        capture : `bool`
            Capture and return output
        check_rc : `bool`
            Assert return code is zero
        cd : `str`
            Path to chdir to, defaults to workspace root
        """
        if isinstance(cmd, string_types):
            shell = True
        else:
            # Some of the command components might be path objects or numbers
            cmd = [str(i) for i in cmd]
    
        if not cd:
            cd = self.workspace
    
        with cmdline.chdir(cd):
            log.debug("run: {0}".format(cmd))
            if capture:
                p = subprocess.Popen(cmd, shell=shell, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, **kwargs)
            else:
                p = subprocess.Popen(cmd, shell=shell, **kwargs)
            (out, _) = p.communicate()
    
            if out is not None and not isinstance(out, string_types):
                out = out.decode('utf-8')
    
            if self.debug and capture:
                log.debug("Stdout/stderr:")
                log.debug(out)
    
            if check_rc and p.returncode != 0:
                err = subprocess.CalledProcessError(p.returncode, cmd)
                err.output = out
                if capture and not self.debug:
                    log.error("Stdout/stderr:")
                    log.error(out)
>               raise err
E               subprocess.CalledProcessError: Command 'pip install --no-cache-dir --upgrade /tmp/tmp1w19zqu3/setuptools-49.5.0.post20200815.zip' returned non-zero exit status 2.

/root/.tox/pypy3/site-packages/pytest_shutil/workspace.py:132: CalledProcessError
------------------------------------------------------ Captured stdout setup -------------------------------------------------------
created virtual environment PyPy3.6.9.final.0-64 in 448ms
  creator PyPy3Posix(dest=/tmp/tmp1w19zqu3/.env, clear=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/root/.local/share/virtualenv)
    added seed packages: pip==20.2.1, setuptools==49.2.1, wheel==0.34.2
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
------------------------------------------------------- Captured stdout call -------------------------------------------------------
Found existing installation: setuptools 49.2.1
Uninstalling setuptools-49.2.1:
  Successfully uninstalled setuptools-49.2.1
Collecting wheel
  Using cached wheel-0.35.1-py2.py3-none-any.whl (33 kB)
Installing collected packages: wheel
  Attempting uninstall: wheel
    Found existing installation: wheel 0.34.2
    Uninstalling wheel-0.34.2:
      Successfully uninstalled wheel-0.34.2
Successfully installed wheel-0.35.1
Processing ./setuptools-49.5.0.post20200815-py3-none-any.whl
Installing collected packages: setuptools
Successfully installed setuptools-49.5.0.post20200815
Processing ./setuptools-49.5.0.post20200815.zip
Building wheels for collected packages: setuptools
  Building wheel for setuptools (setup.py): started
  Building wheel for setuptools (setup.py): finished with status 'done'
  Created wheel for setuptools: filename=setuptools-49.5.0.post20200815-py3-none-any.whl size=803650 sha256=d0e21718f08751074da466fff2239dadbbbc4109069d8146f01a860a71c98136
  Stored in directory: /tmp/pip-ephem-wheel-cache-odk89p0_/wheels/9c/b5/be/d59a9dcb6f2e26db6cf89752c1e74a6f6abb4be7536ebf7304
Successfully built setuptools
Installing collected packages: setuptools
  Attempting uninstall: setuptools
    Found existing installation: setuptools 49.5.0.post20200815
    Uninstalling setuptools-49.5.0.post20200815:
      Successfully uninstalled setuptools-49.5.0.post20200815
  Rolling back uninstall of setuptools
  Moving to /tmp/tmp1w19zqu3/.env/bin/easy_install
   from /tmp/pip-uninstall-whdfuj5i/easy_install
  Moving to /tmp/tmp1w19zqu3/.env/bin/easy_install-3.6
   from /tmp/pip-uninstall-whdfuj5i/easy_install-3.6
  Moving to /tmp/tmp1w19zqu3/.env/site-packages/__pycache__/easy_install.pypy36.pyc
   from /tmp/pip-uninstall-gi7ce6q3/easy_install.pypy36.pyc
  Moving to /tmp/tmp1w19zqu3/.env/site-packages/_distutils_hack/
   from /tmp/tmp1w19zqu3/.env/site-packages/~distutils_hack
  Moving to /tmp/tmp1w19zqu3/.env/site-packages/distutils-precedence.pth
   from /tmp/pip-uninstall-y3ykb1ub/distutils-precedence.pth
  Moving to /tmp/tmp1w19zqu3/.env/site-packages/easy_install.py
   from /tmp/pip-uninstall-y3ykb1ub/easy_install.py
  Moving to /tmp/tmp1w19zqu3/.env/site-packages/pkg_resources/
   from /tmp/tmp1w19zqu3/.env/site-packages/~kg_resources
  Moving to /tmp/tmp1w19zqu3/.env/site-packages/setuptools-49.5.0.post20200815.dist-info/
   from /tmp/tmp1w19zqu3/.env/site-packages/~etuptools-49.5.0.post20200815.dist-info
  Moving to /tmp/tmp1w19zqu3/.env/site-packages/setuptools/
   from /tmp/tmp1w19zqu3/.env/site-packages/~etuptools
------------------------------------------------------- Captured stderr call -------------------------------------------------------
WARNING: You are using pip version 20.2.1; however, version 20.2.2 is available.
You should consider upgrading via the '/tmp/tmp1w19zqu3/.env/bin/pypy3 -m pip install --upgrade pip' command.
/setuptools/_distutils_hack/__init__.py:31: UserWarning: Setuptools is replacing distutils.
  warnings.warn("Setuptools is replacing distutils.")
warning: no files found matching '*.py' under directory 'tests'
/setuptools/_distutils_hack/__init__.py:31: UserWarning: Setuptools is replacing distutils.
  warnings.warn("Setuptools is replacing distutils.")
Distribution option extra_path is deprecated. See issue27919 for details.
warning: no files found matching '*.py' under directory 'tests'
WARNING: You are using pip version 20.2.1; however, version 20.2.2 is available.
You should consider upgrading via the '/tmp/tmp1w19zqu3/.env/bin/pypy3 -m pip install --upgrade pip' command.
ERROR: Exception:
Traceback (most recent call last):
  File "/tmp/tmp1w19zqu3/.env/site-packages/pip/_internal/cli/base_command.py", line 216, in _main
    status = self.run(options, args)
  File "/tmp/tmp1w19zqu3/.env/site-packages/pip/_internal/cli/req_command.py", line 182, in wrapper
    return func(self, options, args)
  File "/tmp/tmp1w19zqu3/.env/site-packages/pip/_internal/commands/install.py", line 421, in run
    pycompile=options.compile,
  File "/tmp/tmp1w19zqu3/.env/site-packages/pip/_internal/req/__init__.py", line 90, in install_given_reqs
    pycompile=pycompile,
  File "/tmp/tmp1w19zqu3/.env/site-packages/pip/_internal/req/req_install.py", line 784, in install
    prefix=prefix,
  File "/tmp/tmp1w19zqu3/.env/site-packages/pip/_internal/locations.py", line 186, in get_scheme
    dist_name, user, home, root, isolated, prefix
  File "/tmp/tmp1w19zqu3/.env/site-packages/pip/_internal/locations.py", line 125, in distutils_scheme
    i.finalize_options()
  File "/tmp/tmp1w19zqu3/.env/site-packages/setuptools/_distutils/command/install.py", line 383, in finalize_options
    ('build_lib', 'build_lib'))
  File "/tmp/tmp1w19zqu3/.env/site-packages/setuptools/_distutils/cmd.py", line 286, in set_undefined_options
    src_cmd_obj = self.distribution.get_command_obj(src_cmd)
  File "/tmp/tmp1w19zqu3/.env/site-packages/setuptools/_distutils/dist.py", line 858, in get_command_obj
    klass = self.get_command_class(command)
  File "/tmp/tmp1w19zqu3/.env/site-packages/setuptools/dist.py", line 771, in get_command_class
    return _Distribution.get_command_class(self, command)
  File "/tmp/tmp1w19zqu3/.env/site-packages/setuptools/_distutils/dist.py", line 844, in get_command_class
    raise DistutilsModuleError("invalid command '%s'" % command)
distutils.errors.DistutilsModuleError: invalid command 'build'
WARNING: You are using pip version 20.2.1; however, version 20.2.2 is available.
You should consider upgrading via the '/tmp/tmp1w19zqu3/.env/bin/pypy3 -m pip install --upgrade pip' command.
___________________________ test_pip_upgrade_from_source[https://github.com/pypa/pip/archive/master.zip] ___________________________

pip_version = 'https://github.com/pypa/pip/archive/master.zip'
virtualenv = <pytest_virtualenv.VirtualEnv object at 0x00007f7078733da8>

    @pytest.mark.parametrize('pip_version', _get_pip_versions())
    def test_pip_upgrade_from_source(pip_version, virtualenv):
        """
        Check pip can upgrade setuptools from source.
        """
        # Install pip/wheel, and remove setuptools (as it
        # should not be needed for bootstraping from source)
        if pip_version is None:
            upgrade_pip = ()
        else:
            upgrade_pip = ('python -m pip install -U {pip_version} --retries=1',)
        virtualenv.run(' && '.join((
            'pip uninstall -y setuptools',
            'pip install -U wheel',
        ) + upgrade_pip).format(pip_version=pip_version))
        dist_dir = virtualenv.workspace
        # Generate source distribution / wheel.
        virtualenv.run(' && '.join((
            'python setup.py -q sdist -d {dist}',
            'python setup.py -q bdist_wheel -d {dist}',
        )).format(dist=dist_dir), cd=SOURCE_DIR)
        sdist = glob.glob(os.path.join(dist_dir, '*.zip'))[0]
        wheel = glob.glob(os.path.join(dist_dir, '*.whl'))[0]
        # Then update from wheel.
        virtualenv.run('pip install ' + wheel)
        # And finally try to upgrade from source.
>       virtualenv.run('pip install --no-cache-dir --upgrade ' + sdist)

setuptools/tests/test_virtualenv.py:123: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/root/.tox/pypy3/site-packages/pytest_virtualenv.py:151: in run
    return super(VirtualEnv, self).run(args, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <pytest_virtualenv.VirtualEnv object at 0x00007f7078733da8>
cmd = 'pip install --no-cache-dir --upgrade /tmp/tmp5__3lnhs/setuptools-49.5.0.post20200815.zip', capture = False, check_rc = True
cd = Path('/tmp/tmp5__3lnhs'), shell = True
kwargs = {'env': {'COVERAGE_FILE': '/root/.tox/.coverage.pypy3', 'PATH': '/tmp/tmp5__3lnhs/.env/bin:/root/.tox/pypy3/bin:/usr/l...pip_upgrade_from_source[https://github.com/pypa/pip/archive/master.zip] (setup)', 'PYTHONHASHSEED': '1620508382', ...}}
p = <subprocess.Popen object at 0x00007f7078733ef8>, out = None, _ = None
err = CalledProcessError(2, 'pip install --no-cache-dir --upgrade /tmp/tmp5__3lnhs/setuptools-49.5.0.post20200815.zip')

    def run(self, cmd, capture=False, check_rc=True, cd=None, shell=False, **kwargs):
        """
        Run a command relative to a given directory, defaulting to the workspace root
    
        Parameters
        ----------
        cmd : `str` or `list`
            Command string or list. Commands given as a string will be run in a subshell.
        capture : `bool`
            Capture and return output
        check_rc : `bool`
            Assert return code is zero
        cd : `str`
            Path to chdir to, defaults to workspace root
        """
        if isinstance(cmd, string_types):
            shell = True
        else:
            # Some of the command components might be path objects or numbers
            cmd = [str(i) for i in cmd]
    
        if not cd:
            cd = self.workspace
    
        with cmdline.chdir(cd):
            log.debug("run: {0}".format(cmd))
            if capture:
                p = subprocess.Popen(cmd, shell=shell, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, **kwargs)
            else:
                p = subprocess.Popen(cmd, shell=shell, **kwargs)
            (out, _) = p.communicate()
    
            if out is not None and not isinstance(out, string_types):
                out = out.decode('utf-8')
    
            if self.debug and capture:
                log.debug("Stdout/stderr:")
                log.debug(out)
    
            if check_rc and p.returncode != 0:
                err = subprocess.CalledProcessError(p.returncode, cmd)
                err.output = out
                if capture and not self.debug:
                    log.error("Stdout/stderr:")
                    log.error(out)
>               raise err
E               subprocess.CalledProcessError: Command 'pip install --no-cache-dir --upgrade /tmp/tmp5__3lnhs/setuptools-49.5.0.post20200815.zip' returned non-zero exit status 2.

/root/.tox/pypy3/site-packages/pytest_shutil/workspace.py:132: CalledProcessError
------------------------------------------------------ Captured stdout setup -------------------------------------------------------
created virtual environment PyPy3.6.9.final.0-64 in 324ms
  creator PyPy3Posix(dest=/tmp/tmp5__3lnhs/.env, clear=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/root/.local/share/virtualenv)
    added seed packages: pip==20.2.1, setuptools==49.2.1, wheel==0.34.2
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
------------------------------------------------------- Captured stdout call -------------------------------------------------------
Found existing installation: setuptools 49.2.1
Uninstalling setuptools-49.2.1:
  Successfully uninstalled setuptools-49.2.1
Collecting wheel
  Using cached wheel-0.35.1-py2.py3-none-any.whl (33 kB)
Installing collected packages: wheel
  Attempting uninstall: wheel
    Found existing installation: wheel 0.34.2
    Uninstalling wheel-0.34.2:
      Successfully uninstalled wheel-0.34.2
Successfully installed wheel-0.35.1
Collecting https://github.com/pypa/pip/archive/master.zip
  Downloading https://github.com/pypa/pip/archive/master.zip
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
Building wheels for collected packages: pip
  Building wheel for pip (PEP 517): started
  Building wheel for pip (PEP 517): finished with status 'done'
  Created wheel for pip: filename=pip-20.3.dev0-py2.py3-none-any.whl size=1504106 sha256=252bbfc009f79db485f7e86ad6a9724e431393c9f6935ab50fa524e9d1c26bbc
  Stored in directory: /tmp/pip-ephem-wheel-cache-e94156d3/wheels/80/3b/f4/b83cf024efdcda3e8b566480490d8113c1484ae9217e3c0f29
Successfully built pip
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 20.2.1
    Uninstalling pip-20.2.1:
      Successfully uninstalled pip-20.2.1
Successfully installed pip-20.3.dev0
Processing ./setuptools-49.5.0.post20200815-py3-none-any.whl
Installing collected packages: setuptools
Successfully installed setuptools-49.5.0.post20200815
Processing ./setuptools-49.5.0.post20200815.zip
Building wheels for collected packages: setuptools
  Building wheel for setuptools (setup.py): started
  Building wheel for setuptools (setup.py): finished with status 'done'
  Created wheel for setuptools: filename=setuptools-49.5.0.post20200815-py3-none-any.whl size=803650 sha256=6700c0ef4b39d90578dc30eabce993053606ff8d8d2aede64d24344d3f45413e
  Stored in directory: /tmp/pip-ephem-wheel-cache-fu7a_r__/wheels/de/aa/bb/2645ed6c4c1c56f5da69b4e012545c98cf755e1a7a1d5c4a20
Successfully built setuptools
Installing collected packages: setuptools
  Attempting uninstall: setuptools
    Found existing installation: setuptools 49.5.0.post20200815
    Uninstalling setuptools-49.5.0.post20200815:
      Successfully uninstalled setuptools-49.5.0.post20200815
  Rolling back uninstall of setuptools
  Moving to /tmp/tmp5__3lnhs/.env/bin/easy_install
   from /tmp/pip-uninstall-x_927ug6/easy_install
  Moving to /tmp/tmp5__3lnhs/.env/bin/easy_install-3.6
   from /tmp/pip-uninstall-x_927ug6/easy_install-3.6
  Moving to /tmp/tmp5__3lnhs/.env/site-packages/__pycache__/easy_install.pypy36.pyc
   from /tmp/pip-uninstall-59vhur_g/easy_install.pypy36.pyc
  Moving to /tmp/tmp5__3lnhs/.env/site-packages/_distutils_hack/
   from /tmp/tmp5__3lnhs/.env/site-packages/~distutils_hack
  Moving to /tmp/tmp5__3lnhs/.env/site-packages/distutils-precedence.pth
   from /tmp/pip-uninstall-k03a921o/distutils-precedence.pth
  Moving to /tmp/tmp5__3lnhs/.env/site-packages/easy_install.py
   from /tmp/pip-uninstall-k03a921o/easy_install.py
  Moving to /tmp/tmp5__3lnhs/.env/site-packages/pkg_resources/
   from /tmp/tmp5__3lnhs/.env/site-packages/~kg_resources
  Moving to /tmp/tmp5__3lnhs/.env/site-packages/setuptools-49.5.0.post20200815.dist-info/
   from /tmp/tmp5__3lnhs/.env/site-packages/~etuptools-49.5.0.post20200815.dist-info
  Moving to /tmp/tmp5__3lnhs/.env/site-packages/setuptools/
   from /tmp/tmp5__3lnhs/.env/site-packages/~etuptools
------------------------------------------------------- Captured stderr call -------------------------------------------------------
WARNING: You are using pip version 20.2.1; however, version 20.2.2 is available.
You should consider upgrading via the '/tmp/tmp5__3lnhs/.env/bin/pypy3 -m pip install --upgrade pip' command.
/setuptools/_distutils_hack/__init__.py:31: UserWarning: Setuptools is replacing distutils.
  warnings.warn("Setuptools is replacing distutils.")
warning: no files found matching '*.py' under directory 'tests'
/setuptools/_distutils_hack/__init__.py:31: UserWarning: Setuptools is replacing distutils.
  warnings.warn("Setuptools is replacing distutils.")
Distribution option extra_path is deprecated. See issue27919 for details.
warning: no files found matching '*.py' under directory 'tests'
ERROR: Exception:
Traceback (most recent call last):
  File "/tmp/tmp5__3lnhs/.env/site-packages/pip/_internal/cli/base_command.py", line 216, in _main
    status = self.run(options, args)
  File "/tmp/tmp5__3lnhs/.env/site-packages/pip/_internal/cli/req_command.py", line 181, in wrapper
    return func(self, options, args)
  File "/tmp/tmp5__3lnhs/.env/site-packages/pip/_internal/commands/install.py", line 420, in run
    pycompile=options.compile,
  File "/tmp/tmp5__3lnhs/.env/site-packages/pip/_internal/req/__init__.py", line 90, in install_given_reqs
    pycompile=pycompile,
  File "/tmp/tmp5__3lnhs/.env/site-packages/pip/_internal/req/req_install.py", line 787, in install
    prefix=prefix,
  File "/tmp/tmp5__3lnhs/.env/site-packages/pip/_internal/locations.py", line 186, in get_scheme
    dist_name, user, home, root, isolated, prefix
  File "/tmp/tmp5__3lnhs/.env/site-packages/pip/_internal/locations.py", line 125, in distutils_scheme
    i.finalize_options()
  File "/tmp/tmp5__3lnhs/.env/site-packages/setuptools/_distutils/command/install.py", line 383, in finalize_options
    ('build_lib', 'build_lib'))
  File "/tmp/tmp5__3lnhs/.env/site-packages/setuptools/_distutils/cmd.py", line 286, in set_undefined_options
    src_cmd_obj = self.distribution.get_command_obj(src_cmd)
  File "/tmp/tmp5__3lnhs/.env/site-packages/setuptools/_distutils/dist.py", line 858, in get_command_obj
    klass = self.get_command_class(command)
  File "/tmp/tmp5__3lnhs/.env/site-packages/setuptools/dist.py", line 771, in get_command_class
    return _Distribution.get_command_class(self, command)
  File "/tmp/tmp5__3lnhs/.env/site-packages/setuptools/_distutils/dist.py", line 844, in get_command_class
    raise DistutilsModuleError("invalid command '%s'" % command)
distutils.errors.DistutilsModuleError: invalid command 'build'
===================================================== short test summary info ======================================================
SKIPPED [2] ../Users/jaraco/code/public/pypa/setuptools/setuptools/tests/test_msvc.py:17: could not import 'distutils.msvc9compiler': No module named 'winreg'
================================ 2 failed, 4 passed, 2 skipped, 813 deselected in 167.10s (0:02:47) ================================
ERROR: InvocationError for command /root/.tox/pypy3/bin/pytest -k pip_upgrade_from_source (exited with code 1)
_____________________________________________________________ summary ______________________________________________________________
ERROR:   pypy3: commands failed

jaraco · 2020-08-15T14:32:09Z

I'm able to replicate the failures on bionic as well.

jaraco · 2020-08-15T15:06:38Z

Well, last night I thought I was able to replicate the issue on bionic, but testing today, tests are passing on bionic :/

jaraco · 2020-08-15T15:24:17Z

This dockerfile seems to be reliably replicating the failure:

FROM ubuntu:bionic
RUN apt update
RUN apt install -y software-properties-common
RUN add-apt-repository ppa:pypy/ppa
RUN apt update
RUN apt install -y virtualenv pypy3 python3-pip git
RUN pip3 install tox tox-pip-version
RUN git clone https://github.com/pypa/setuptools -b bugfix/2232-adopt-distutils-default
WORKDIR /setuptools
RUN tox -e pypy3 --notest
CMD tox -e pypy3 -- -k pip_upgrade

jaraco · 2020-08-15T17:33:05Z

I think I've figured out what's going on.

On macOS, PyPy imports distutils before .pth processing, but on Linux, PyPy imports distutils after .pth processing. That means that on macOS, distutils comes from stdlib but on Linux, distutils resolves to Setuptools' copy. As a result, by the time pip gets around to importing distutils, the selectivity of the importer is no longer relevant - distutils is already resolved during startup time.

I'm beginning to doubt it will be feasible for Setuptools to magically supply distutils or defer to the stdlib at the relevant decision points.

jaraco · 2020-08-15T17:47:41Z

I'm now considering this approach:

In the DistutilsMetaFinder, detect if pip is being imported. If so:

unload any existing distutils
disable the finder

…istutilsMetaFinder.

jaraco · 2020-08-19T01:45:52Z

My plan is to merge and release this change tomorrow or Thursday.

jaraco added 2 commits July 12, 2020 10:56

Re-enable distutils patch by default.

9e58d95

Merge branch 'master' into bugfix/2232-adopt-distutils-default

c85bdd8

jaraco mentioned this pull request Aug 10, 2020

Reconsidering distutils replacement strategy #2259

Closed

jaraco added 2 commits August 13, 2020 20:34

Merge branch 'master' into bugfix/2232-adopt-distutils-default

74e4761

Update tests to reflect new expectation.

4837f21

jaraco mentioned this pull request Aug 14, 2020

Unable to upgrade setuptools with adopted distutils pypa/pip#8761

Closed

Bypass .pth loader when distutils is loaded from pip. Workaround for p…

be7a0a3

…ypa/pip#8761.

jaraco force-pushed the bugfix/2232-adopt-distutils-default branch from 6e3aa8d to be7a0a3 Compare August 14, 2020 23:39

When pip is imported, unload any existing distutils and disable the D…

77eeb77

…istutilsMetaFinder.

jaraco requested a review from pganssle August 15, 2020 19:01

Refactor to use lookups and consolidate behaviors.

3d404fd

jaraco force-pushed the bugfix/2232-adopt-distutils-default branch from 4dd41a2 to 3d404fd Compare August 15, 2020 19:04

jaraco merged commit e20e85e into master Aug 21, 2020

jaraco deleted the bugfix/2232-adopt-distutils-default branch August 21, 2020 01:41

rgommers mentioned this pull request Sep 1, 2020

WIP: BLD: try to fix build with setuptools 50.0 on Windows scipy/scipy#12798

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Re-enable distutils patch by default. #2255

Re-enable distutils patch by default. #2255

jaraco commented Jul 12, 2020

jaraco commented Aug 14, 2020

jaraco commented Aug 14, 2020

jaraco commented Aug 14, 2020

jaraco commented Aug 14, 2020

jaraco commented Aug 14, 2020

jaraco commented Aug 15, 2020

jaraco commented Aug 15, 2020

jaraco commented Aug 15, 2020

jaraco commented Aug 15, 2020

jaraco commented Aug 15, 2020

jaraco commented Aug 15, 2020

jaraco commented Aug 15, 2020

jaraco commented Aug 19, 2020

Re-enable distutils patch by default. #2255

Re-enable distutils patch by default. #2255

Conversation

jaraco commented Jul 12, 2020

Pull Request Checklist

jaraco commented Aug 14, 2020

jaraco commented Aug 14, 2020

jaraco commented Aug 14, 2020

jaraco commented Aug 14, 2020

jaraco commented Aug 14, 2020

jaraco commented Aug 15, 2020

jaraco commented Aug 15, 2020

jaraco commented Aug 15, 2020

jaraco commented Aug 15, 2020

jaraco commented Aug 15, 2020

jaraco commented Aug 15, 2020

jaraco commented Aug 15, 2020

jaraco commented Aug 19, 2020