Compilation issues when SETUPTOOLS_USE_DISTUTILS is local

[matham]

After #2212 I tried setting SETUPTOOLS_USE_DISTUTILS to local, but ran into issues compiling.

Setuptools version is 49.2.0. The first issue was that I got the following error on my system:

Traceback (most recent call last):
   File "setup.py", line 1150, in <module>
     'Topic :: Software Development :: User Interfaces'])
   File "E:\Python\Python37\lib\site-packages\setuptools\__init__.py", line 165, in setup
     return distutils.core.setup(**attrs)
   File "E:\Python\Python37\lib\site-packages\setuptools\_distutils\core.py", line 134, in setup
     ok = dist.parse_command_line()
   File "E:\Python\Python37\lib\site-packages\setuptools\_distutils\dist.py", line 484, in parse_command_line
     args = self._parse_command_opts(parser, args)
   File "E:\Python\Python37\lib\site-packages\setuptools\dist.py", line 929, in _parse_command_opts
     nargs = _Distribution._parse_command_opts(self, parser, args)
   File "E:\Python\Python37\lib\site-packages\setuptools\_distutils\dist.py", line 548, in _parse_command_opts
     "command class %s must subclass Command" % cmd_class)
 distutils.errors.DistutilsClassError: command class <class '__main__.KivyBuildExt'> must subclass Command

Interestingly, it only happened locally and not the on the CI (on the CI the subsequent error happened), I'm not sure why. So, I traced the issue to our setup.py looking as follow:

from distutils.command.build_ext import build_ext
from distutils.version import LooseVersion
from distutils.sysconfig import get_python_inc
from setuptools import setup, Extension, find_packages

I fixed it by importing setuptools before distutils. That's where I ran into the second issue that I cannot fix. Here's the error:

Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Users\runneradmin\kivy\setup.py", line 1150, in <module>
  'Topic :: Software Development :: User Interfaces'])
File "C:\Users\RUNNER~1\AppData\Local\Temp\pip-build-env-zsh0976j\overlay\Lib\site-packages\setuptools\__init__.py", line 165, in setup
  return distutils.core.setup(**attrs)
File "C:\Users\RUNNER~1\AppData\Local\Temp\pip-build-env-zsh0976j\overlay\Lib\site-packages\setuptools\_distutils\core.py", line 148, in setup
  dist.run_commands()
File "C:\Users\RUNNER~1\AppData\Local\Temp\pip-build-env-zsh0976j\overlay\Lib\site-packages\setuptools\_distutils\dist.py", line 967, in run_commands
  self.run_command(cmd)
File "C:\Users\RUNNER~1\AppData\Local\Temp\pip-build-env-zsh0976j\overlay\Lib\site-packages\setuptools\_distutils\dist.py", line 986, in run_command
  cmd_obj.run()
File "C:\Users\RUNNER~1\AppData\Local\Temp\pip-build-env-zsh0976j\overlay\Lib\site-packages\setuptools\command\develop.py", line 38, in run
  self.install_for_development()
File "C:\Users\RUNNER~1\AppData\Local\Temp\pip-build-env-zsh0976j\overlay\Lib\site-packages\setuptools\command\develop.py", line 140, in install_for_development
  self.run_command('build_ext')
File "C:\Users\RUNNER~1\AppData\Local\Temp\pip-build-env-zsh0976j\overlay\Lib\site-packages\setuptools\_distutils\cmd.py", line 313, in run_command
  self.distribution.run_command(command)
File "C:\Users\RUNNER~1\AppData\Local\Temp\pip-build-env-zsh0976j\overlay\Lib\site-packages\setuptools\_distutils\dist.py", line 986, in run_command
  cmd_obj.run()
File "C:\Users\RUNNER~1\AppData\Local\Temp\pip-build-env-zsh0976j\overlay\Lib\site-packages\Cython\Distutils\old_build_ext.py", line 186, in run
  _build_ext.build_ext.run(self)
File "C:\Users\RUNNER~1\AppData\Local\Temp\pip-build-env-zsh0976j\overlay\Lib\site-packages\setuptools\_distutils\command\build_ext.py", line 340, in run
  self.build_extensions()
File "C:\Users\runneradmin\kivy\setup.py", line 331, in build_extensions
  super(KivyBuildExt, self).build_extensions()
File "C:\Users\RUNNER~1\AppData\Local\Temp\pip-build-env-zsh0976j\overlay\Lib\site-packages\Cython\Distutils\old_build_ext.py", line 195, in build_extensions
  _build_ext.build_ext.build_extensions(self)
File "C:\Users\RUNNER~1\AppData\Local\Temp\pip-build-env-zsh0976j\overlay\Lib\site-packages\setuptools\_distutils\command\build_ext.py", line 447, in build_extensions
  self._build_extensions_parallel()
File "C:\Users\RUNNER~1\AppData\Local\Temp\pip-build-env-zsh0976j\overlay\Lib\site-packages\setuptools\_distutils\command\build_ext.py", line 469, in _build_extensions_parallel
  fut.result()
File "C:\hostedtoolcache\windows\Python\3.6.8\x64\lib\concurrent\futures\_base.py", line 432, in result
  return self.__get_result()
File "C:\hostedtoolcache\windows\Python\3.6.8\x64\lib\concurrent\futures\_base.py", line 384, in __get_result
  raise self._exception
File "C:\hostedtoolcache\windows\Python\3.6.8\x64\lib\concurrent\futures\thread.py", line 56, in run
  result = self.fn(*self.args, **self.kwargs)
File "C:\Users\RUNNER~1\AppData\Local\Temp\pip-build-env-zsh0976j\overlay\Lib\site-packages\setuptools\_distutils\command\build_ext.py", line 535, in build_extension
  depends=ext.depends)
File "C:\Users\RUNNER~1\AppData\Local\Temp\pip-build-env-zsh0976j\overlay\Lib\site-packages\setuptools\_distutils\_msvccompiler.py", line 401, in compile
  self.spawn(args)
File "C:\Users\RUNNER~1\AppData\Local\Temp\pip-build-env-zsh0976j\overlay\Lib\site-packages\setuptools\_distutils\_msvccompiler.py", line 505, in spawn
  return super().spawn(cmd, env=env)
TypeError: spawn() got an unexpected keyword argument 'env'

Locally I added the this beofre calling super: print(super().spawn) and that printed <bound method CCompiler.spawn of <distutils._msvccompiler.MSVCCompiler object at 0x00000269AA79CE80>>,

Somehow it still used distutils version of _msvccompiler. Is it possible that the usage of local distutils is patched too late after distutils._msvccompiler is already imported?

[jaraco]

Thank you for reporting this. Yes, it's an issue that the current implementation requires that setuptools be imported before distutils. We're working on another hack that will make that ordering less important in many cases.

Somehow it still used distutils version of _msvccompiler. Is it possible that the usage of local distutils is patched too late after distutils._msvccompiler is already imported?

It's highly unlikely. If setuptools is imported first, it very early ensures that sys.modules['distutils'] is setuptools._distutils. I'll work on wrapping up #2260 and then see if I can replicate the issue. Would you consider putting together for me a minimal reproducer that I can use to replicate the issue on my Windows box?

[matham]

I made a sample project: https://github.com/matham/cython_demo_project.

Master passes: https://github.com/matham/cython_demo_project/actions/runs/183666627.
branch using local fails: https://github.com/matham/cython_demo_project/actions/runs/183665378.

[jaraco]

Thanks Matham for the simple repro. Using it, I was able to replicate the issue with a simple setup.py build invocation (after setting SETUPTOOLS_USE_DISTUTILS=local).

PS C:\Users\jaraco\local\cython_demo_project> $env:SETUPTOOLS_USE_DISTUTILS='local'
PS C:\Users\jaraco\local\cython_demo_project> git clean -fdx
PS C:\Users\jaraco\local\cython_demo_project> pip-run -q cython -- setup.py build
running build
running build_py
creating build
creating build\lib.win-amd64-3.8
creating build\lib.win-amd64-3.8\cython_demo_project
copying cython_demo_project\__init__.py -> build\lib.win-amd64-3.8\cython_demo_project
creating build\lib.win-amd64-3.8\cython_demo_project\tests
copying cython_demo_project\tests\test_numeric.py -> build\lib.win-amd64-3.8\cython_demo_project\tests
copying cython_demo_project\tests\__init__.py -> build\lib.win-amd64-3.8\cython_demo_project\tests
copying cython_demo_project\numeric.pxd -> build\lib.win-amd64-3.8\cython_demo_project
running build_ext
cythoning cython_demo_project\numeric.pyx to cython_demo_project\numeric.c
building 'cython_demo_project.numeric' extension
creating build\temp.win-amd64-3.8
creating build\temp.win-amd64-3.8\Release
creating build\temp.win-amd64-3.8\Release\cython_demo_project
Traceback (most recent call last):
  File "setup.py", line 25, in <module>
    setup(
  File "c:\python38\lib\site-packages\setuptools\__init__.py", line 165, in setup
    return distutils.core.setup(**attrs)
  File "c:\python38\lib\site-packages\setuptools\_distutils\core.py", line 148, in setup
    dist.run_commands()
  File "c:\python38\lib\site-packages\setuptools\_distutils\dist.py", line 967, in run_commands
    self.run_command(cmd)
  File "c:\python38\lib\site-packages\setuptools\_distutils\dist.py", line 986, in run_command
    cmd_obj.run()
  File "c:\python38\lib\site-packages\setuptools\_distutils\command\build.py", line 135, in run
    self.run_command(cmd_name)
  File "c:\python38\lib\site-packages\setuptools\_distutils\cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "c:\python38\lib\site-packages\setuptools\_distutils\dist.py", line 986, in run_command
    cmd_obj.run()
  File "C:\Users\jaraco\AppData\Local\Temp\pip-run-n8i9vs3m\Cython\Distutils\old_build_ext.py", line 186, in run
    _build_ext.build_ext.run(self)
  File "c:\python38\lib\site-packages\setuptools\_distutils\command\build_ext.py", line 340, in run
    self.build_extensions()
  File "C:\Users\jaraco\AppData\Local\Temp\pip-run-n8i9vs3m\Cython\Distutils\old_build_ext.py", line 195, in build_extensions
    _build_ext.build_ext.build_extensions(self)
  File "c:\python38\lib\site-packages\setuptools\_distutils\command\build_ext.py", line 449, in build_extensions
    self._build_extensions_serial()
  File "c:\python38\lib\site-packages\setuptools\_distutils\command\build_ext.py", line 474, in _build_extensions_serial
    self.build_extension(ext)
  File "c:\python38\lib\site-packages\setuptools\_distutils\command\build_ext.py", line 529, in build_extension
    objects = self.compiler.compile(sources,
  File "c:\python38\lib\site-packages\setuptools\_distutils\_msvccompiler.py", line 401, in compile
    self.spawn(args)
  File "c:\python38\lib\site-packages\setuptools\_distutils\_msvccompiler.py", line 505, in spawn
    return super().spawn(cmd, env=env)
TypeError: spawn() got an unexpected keyword argument 'env'

From here, I'm hoping to trace the cause.

[jaraco]

Alright. I've figured out the issue, and it appears it's unrelated to monkey-patching (thank goodness) and is actually just a flaw in the fix for pypa/distutils#5. The flaw is that MSVCCompiler.spawn() calls super().spawn() where super() is distutils.compiler.CCompiler, and indeed that method accepts no env parameter. Although that fix has a test, the test is probably passing because the nature of threads is they can throw exceptions silently.

[jaraco]

I merged the distutils changes at c2fcb01 and cut 49.2.1 with the fixes. Please report back if that doesn't fix the issue properly.