BUG: Cython 0.29.31 release breaks building SciPy from source

[oscarbenjamin]

Describe your issue.

This comes from failures seen in SymPy CI:
sympy/sympy#23847

The release of Cython 0.29.31 means that building SciPy from source with pip fails both with the latest release SciPy 1.8.1 and also the main branch.

It looks like pyproject.toml on the 1.8.x branch needs to have a better upper bound for the Cython version since it only requires <3.0 which does not prevent new releases in Cython's 0.29.x series from breaking the build:

scipy/pyproject.toml

Lines 1 to 14 in b76137f

	# Note that in maintenance branches, all build dependencies should
	# have an upper bound equal to the most recent already-released version
	# of the dependency. This to prevent that a future backwards-incompatible
	# release will break the source build of a SciPy release.
	# Do accept micro (bug-fix) releases. So for example, if pybind11 2.4.3 is
	# the most recent version on PyPI:
	#
	# "pybind11>=2.4.3,<2.5.0",
	[build-system]
	requires = [
	"wheel<0.38.0",
	"setuptools<60.0.0",
	"Cython>=0.29.18,<3.0",

Reproducing Code Example

$ pip install -U pip setuptools wheel cython numpy
Collecting pip
  Downloading pip-22.2.1-py3-none-any.whl (2.0 MB)
     |████████████████████████████████| 2.0 MB 4.8 MB/s 
Collecting setuptools
  Downloading setuptools-63.2.0-py3-none-any.whl (1.2 MB)
     |████████████████████████████████| 1.2 MB 28.2 MB/s 
Collecting wheel
  Using cached wheel-0.37.1-py2.py3-none-any.whl (35 kB)
Collecting cython
  Downloading Cython-0.29.31-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (1.9 MB)
     |████████████████████████████████| 1.9 MB 2.8 MB/s 
Collecting numpy
  Using cached numpy-1.23.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.1 MB)
Installing collected packages: pip, setuptools, wheel, cython, numpy
  Attempting uninstall: pip
    Found existing installation: pip 20.2.3
    Uninstalling pip-20.2.3:
      Successfully uninstalled pip-20.2.3
  Attempting uninstall: setuptools
    Found existing installation: setuptools 49.2.1
    Uninstalling setuptools-49.2.1:
      Successfully uninstalled setuptools-49.2.1
Successfully installed cython-0.29.31 numpy-1.23.1 pip-22.2.1 setuptools-63.2.0 wheel-0.37.1
$ pip install --no-binary scipy scipy
Collecting scipy
  Downloading scipy-1.8.1.tar.gz (38.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 38.2/38.2 MB 8.0 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error

Error message

$ pip install --no-binary scipy scipy
Collecting scipy
  Downloading scipy-1.8.1.tar.gz (38.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 38.2/38.2 MB 8.0 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [102 lines of output]
      Running scipy/special/_generate_pyx.py
      Running scipy/stats/_generate_pyx.py
      Running scipy/linalg/_generate_pyx.py
      Processing scipy/special/_ufuncs.pyx
      Processing scipy/special/_test_round.pyx
      Processing scipy/special/_ufuncs_cxx.pyx
      Processing scipy/special/_ellip_harm_2.pyx
      Processing scipy/special/cython_special.pyx
      Processing scipy/special/_comb.pyx
      Processing scipy/spatial/_voronoi.pyx
      Processing scipy/spatial/_hausdorff.pyx
      Processing scipy/spatial/_qhull.pyx
      Processing scipy/spatial/_ckdtree.pyx
      Processing scipy/spatial/transform/_rotation.pyx
      Processing scipy/interpolate/_ppoly.pyx
      Processing scipy/interpolate/_bspl.pyx
      Processing scipy/interpolate/interpnd.pyx
      Processing scipy/cluster/_vq.pyx
      Processing scipy/cluster/_hierarchy.pyx
      Processing scipy/cluster/_optimal_leaf_ordering.pyx
      Processing scipy/_lib/messagestream.pyx
      Processing scipy/_lib/_test_deprecation_call.pyx
      Processing scipy/_lib/_test_deprecation_def.pyx
      Processing scipy/_lib/_ccallback_c.pyx
      Processing scipy/signal/_upfirdn_apply.pyx
      Processing scipy/signal/_max_len_seq_inner.pyx
      Processing scipy/signal/_sosfilt.pyx
      Processing scipy/signal/_spectral.pyx
      Processing scipy/signal/_peak_finding_utils.pyx
      Processing scipy/io/matlab/_streams.pyx
      Processing scipy/io/matlab/_mio5_utils.pyx
      Processing scipy/io/matlab/_mio_utils.pyx
      Processing scipy/stats/_stats.pyx
      Processing scipy/stats/_qmc_cy.pyx
      Processing scipy/stats/_biasedurn.pyx
      Processing scipy/stats/_sobol.pyx
      Processing scipy/stats/_boost/src/nbinom_ufunc.pyx
      Processing scipy/stats/_boost/src/beta_ufunc.pyx
      
      Error compiling Cython file:
      ------------------------------------------------------------
      ...
          return np.array(result, dtype=np.int64)
      
      
      @cython.wraparound(False)
      @cython.boundscheck(False)
      def _weightedrankedtau(ordered[:] x, ordered[:] y, intp_t[:] rank, weigher, bool additive):
      ^
      ------------------------------------------------------------
      
      _stats.pyx:173:0: Referring to a memoryview typed argument directly in a nested closure function is not supported in Cython 0.x. Either upgrade to Cython 3, or assign the argument to a local variable and use that in the nested function.
      Processing scipy/stats/_boost/src/binom_ufunc.pyx
      Traceback (most recent call last):
        File "/tmp/pip-install-wnazhy0r/scipy_3ee8a876e4f54aa7bfa3d9fdba592754/tools/cythonize.py", line 317, in <module>
          main()
        File "/tmp/pip-install-wnazhy0r/scipy_3ee8a876e4f54aa7bfa3d9fdba592754/tools/cythonize.py", line 313, in main
          find_process_files(root_dir)
        File "/tmp/pip-install-wnazhy0r/scipy_3ee8a876e4f54aa7bfa3d9fdba592754/tools/cythonize.py", line 302, in find_process_files
          for result in pool.imap_unordered(lambda args: process(*args), jobs):
        File "/usr/lib/python3.8/multiprocessing/pool.py", line 868, in next
          raise value
        File "/usr/lib/python3.8/multiprocessing/pool.py", line 125, in worker
          result = (True, func(*args, **kwds))
        File "/tmp/pip-install-wnazhy0r/scipy_3ee8a876e4f54aa7bfa3d9fdba592754/tools/cythonize.py", line 302, in <lambda>
          for result in pool.imap_unordered(lambda args: process(*args), jobs):
        File "/tmp/pip-install-wnazhy0r/scipy_3ee8a876e4f54aa7bfa3d9fdba592754/tools/cythonize.py", line 236, in process
          processor_function(fromfile, tofile, cwd=path)
        File "/tmp/pip-install-wnazhy0r/scipy_3ee8a876e4f54aa7bfa3d9fdba592754/tools/cythonize.py", line 102, in process_pyx
          raise Exception('Cython failed')
      Exception: Cython failed
      setup.py:486: UserWarning: Unrecognized setuptools command ('dist_info --egg-base /tmp/pip-modern-metadata-tcs7khbn'), proceeding with generating Cython sources and expanding templates
        warnings.warn("Unrecognized setuptools command ('{}'), proceeding with "
      Running from SciPy source directory.
      Cythonizing sources
      Traceback (most recent call last):
        File "setup.py", line 350, in generate_cython
          import pip
      ModuleNotFoundError: No module named 'pip'
      
      During handling of the above exception, another exception occurred:
      
      Traceback (most recent call last):
        File "/home/oscar/current/sympy/sympy.git/tmpvenv/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
          main()
        File "/home/oscar/current/sympy/sympy.git/tmpvenv/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/home/oscar/current/sympy/sympy.git/tmpvenv/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 164, in prepare_metadata_for_build_wheel
          return hook(metadata_directory, config_settings)
        File "/tmp/pip-build-env-nvuzm8iv/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 174, in prepare_metadata_for_build_wheel
          self.run_setup()
        File "/tmp/pip-build-env-nvuzm8iv/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 267, in run_setup
          super(_BuildMetaLegacyBackend,
        File "/tmp/pip-build-env-nvuzm8iv/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 158, in run_setup
          exec(compile(code, __file__, 'exec'), locals())
        File "setup.py", line 628, in <module>
          setup_package()
        File "setup.py", line 612, in setup_package
          generate_cython()
        File "setup.py", line 352, in generate_cython
          raise RuntimeError("Running cythonize failed!")
      RuntimeError: Running cythonize failed!
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

SciPy/NumPy/Python version information

SciPy 1.8.1 and current main, numpy 1.23.1, Any version of Python or PyPy (I think), Cython 0.29.31

[oscarbenjamin]

Actually the SymPy CI job which runs with Cython master and SciPy main now seems to have passed. I'm not sure if that is because of a change/fix in either Cython or SciPy. It does look there are other issues in the Cython tracker related to the 0.29.31 release breaking some code that uses memoryviews so maybe this is a Cython bug.

Probably though SciPy should pin the Cython version more tightly. I don't immediately see a way to force SciPy 1.8.1 to build with cython != 0.29.31.

[oscarbenjamin]

I've opened a Cython issue: cython/cython#4928

[mattip]

I think this is a duplicate of #16718 and fixed in #16719

[oscarbenjamin]

Thanks. Maybe this can be closed then. Two questions though:

1. Should there be a tighter version pin for Cython in SciPy releases?
2. Is there a way to manually pin the version of Cython that SciPy uses to build itself via pip?

[oscarbenjamin]

The Cython issue itself looks like it will be fixed by cython/cython#4929 which I guess will go into a Cython 0.29.32 release.

[rgommers]

Should there be a tighter version pin for Cython in SciPy releases?

We'll backport the fix to 1.9.0 which will come out in the next week.

Is there a way to manually pin the version of Cython that SciPy uses to build itself via pip?

Unfortunately, not in an isolated build. So pip install scipy from source is broken right now. When adding --no-build-isolation, you have control over the build environment, so can control what Cython version to use.

[oscarbenjamin]

Unfortunately, not in an isolated build

The best I've found is:

$ echo 'cython!=0.29.31' > constraints.txt
$ PIP_CONSTRAINT=constraints.txt pip install scipy

You have to use the environment variable (rather than a command line argument) so that it passes through to the subprocess that actually installs these in the isolated build.

[rgommers]

Nice - that is better than nothing:)

[h-vetinari]

Given that 0.29.32 was released, is this still necessary?

[rgommers]

This is fixed on both ends of the line now, so can be closed.