numpy.distutils is deprecated in NumPy 1.23 and issues with setuptools>=65.

[stuartarchibald]

Introduction:

In trying to get to the root cause of this backtrace: #8352 (comment) some things of note:

1. setuptools >= 65 has removed the msvc compiler exports that Numba was using via NumPy (see pin setuptools to 64 for doc builds #8352 (comment))
2. NumPy 1.23 has deprecated its numpy.distutils module with removal scheduled for Python >= 3.12. To trigger the deprecation notice, using NumPy 1.23, run:

   python -c "import numpy.distutils.misc_util"
   

3. The change in 1. leads to NumPy's distutils compilation chain being broken. MWR that doesn't involve Numba at all:

   import numpy.distutils.misc_util
   from cffi import FFI
   ffi = FFI()
   ffi.set_source('tmp', "int foo(void) { return 1; }")
   ffi.compile()

   this triggers the backtrace noted above.

I think that the impact of the setuptools change is that numba.distutils for all NumPy versions will be broken.

---

Numba is impacted by this as follows:

• This manifests as an issue in Numba's build as the extension modules creation in setup.py has this:

  numba/setup.py

  Lines 136 to 143 in f5af7b6

  	def get_ext_modules():
  	"""
  	Return a list of Extension instances for the setup() call.
  	"""
  	# Note we don't import Numpy at the toplevel, since setup.py
  	# should be able to run without Numpy for pip to discover the
  	# build dependencies
  	import numpy.distutils.misc_util as np_misc

• This manifests as issues in Numba's testing as numba.tests.support, which is imported in pretty much every single Numba test case to get the custom TestCase class, has this:
  

  numba/numba/tests/support.py

  Line 42 in f5af7b6

  from numba.pycc.platform import _external_compiler_ok

  details of the call chain follows.

  The pycc module imports numpy.distutils.misc_util to use the external compilers bound by numpy.distutil.

  numba/numba/pycc/platform.py

  Line 6 in f5af7b6

  import numpy.distutils.misc_util as np_misc

  
  which is imported via:
  

  numba/numba/pycc/cc.py

  Line 12 in f5af7b6

  from numba.pycc.platform import Toolchain

  
  via:
  

  numba/numba/pycc/__init__.py

  Line 11 in f5af7b6

  from .cc import CC

  
  which leads to any import where Numba imports something from pycc to trigger the use of numpy.distutils which then hits the problem mentioned in above.

---

Fixing it:

• Suggested short term fix is to require setuptools < 60 as suggested by the NumPy 1.23 deprecation warning.

• Suggested long term fix it to move away from numpy.distutils.

[esc]

#8356 has now been merged. @stuartarchibald do we leave this open as it still requires a long term fix?

[esc]

@stuartarchibald could this be a potential long-term fix: #8366

[stuartarchibald]

@esc #8544 should fix the issue with numba.distutils.misc_util as it isn't needed any more. Once implemented correctly #8476 should remove the reliance on setuptools at runtime.

[stuartarchibald]

Am going to close this as the issue is quite specific and the general issues surrouding setuptools/distutils are address in the PRs mentioned in #8355 (comment).