"AttributeError: cython_sources" with Cython 3.0.0a10

[The-Compiler]

When attempting to install PyYAML from sources with Cython 3.0.0a10, e.g. like so:

pip install --pre -v git+https://github.com/yaml/pyyaml.git

the install fails with:

Using pip 21.3.1 from /home/florian/tmp/toxtest/.venv/lib/python3.10/site-packages/pip (python 3.10)
Collecting git+https://github.com/yaml/pyyaml.git
  Cloning https://github.com/yaml/pyyaml.git to /tmp/pip-req-build-rdv2sw86
  Running command git version
  git version 2.34.1
  Running command git clone --filter=blob:none -q https://github.com/yaml/pyyaml.git /tmp/pip-req-build-rdv2sw86
  Running command git rev-parse HEAD
  8cdff2c80573b8be8e8ad28929264a913a63aa33
  Resolved https://github.com/yaml/pyyaml.git to commit 8cdff2c80573b8be8e8ad28929264a913a63aa33
  Running command /home/florian/tmp/toxtest/.venv/bin/python3 /tmp/pip-standalone-pip-7sp3xayz/__env_pip__.zip/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-az2k859x/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple --pre -- setuptools wheel Cython
  Collecting setuptools
    Using cached setuptools-60.5.0-py3-none-any.whl (958 kB)
  Collecting wheel
    Using cached wheel-0.37.1-py2.py3-none-any.whl (35 kB)
  Collecting Cython
    Using cached Cython-3.0.0a10-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (2.2 MB)
  Installing collected packages: wheel, setuptools, Cython
  Successfully installed Cython-3.0.0a10 setuptools-60.5.0 wheel-0.37.1
  Installing build dependencies ... done
  Running command /home/florian/tmp/toxtest/.venv/bin/python3 /home/florian/tmp/toxtest/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmpqirb9t5l
  running egg_info
  creating lib/PyYAML.egg-info
  writing manifest file 'lib/PyYAML.egg-info/SOURCES.txt'
  Traceback (most recent call last):
    File "/home/florian/tmp/toxtest/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
      main()
    File "/home/florian/tmp/toxtest/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/home/florian/tmp/toxtest/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 130, in get_requires_for_build_wheel
      return hook(config_settings)
    File "/tmp/pip-build-env-az2k859x/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 162, in get_requires_for_build_wheel
      return self._get_build_requires(
    File "/tmp/pip-build-env-az2k859x/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 143, in _get_build_requires
      self.run_setup()
    File "/tmp/pip-build-env-az2k859x/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 158, in run_setup
      exec(compile(code, __file__, 'exec'), locals())
    File "setup.py", line 288, in <module>
      setup(
    File "/tmp/pip-build-env-az2k859x/overlay/lib/python3.10/site-packages/setuptools/__init__.py", line 155, in setup
      return distutils.core.setup(**attrs)
    File "/tmp/pip-build-env-az2k859x/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 148, in setup
      return run_commands(dist)
    File "/tmp/pip-build-env-az2k859x/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 163, in run_commands
      dist.run_commands()
    File "/tmp/pip-build-env-az2k859x/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 967, in run_commands
      self.run_command(cmd)
    File "/tmp/pip-build-env-az2k859x/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
      cmd_obj.run()
    File "/tmp/pip-build-env-az2k859x/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 299, in run
      self.find_sources()
    File "/tmp/pip-build-env-az2k859x/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 306, in find_sources
      mm.run()
    File "/tmp/pip-build-env-az2k859x/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 541, in run
      self.add_defaults()
    File "/tmp/pip-build-env-az2k859x/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 578, in add_defaults
      sdist.add_defaults(self)
    File "/tmp/pip-build-env-az2k859x/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/sdist.py", line 228, in add_defaults
      self._add_defaults_ext()
    File "/tmp/pip-build-env-az2k859x/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/sdist.py", line 312, in _add_defaults_ext
      self.filelist.extend(build_ext.get_source_files())
    File "setup.py", line 204, in get_source_files
      self.cython_sources(ext.sources, ext)
    File "/tmp/pip-build-env-az2k859x/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 103, in __getattr__
      raise AttributeError(attr)
  AttributeError: cython_sources
  Getting requirements to build wheel ... error
WARNING: Discarding git+https://github.com/yaml/pyyaml.git. Command errored out with exit status 1: /home/florian/tmp/toxtest/.venv/bin/python3 /home/florian/tmp/toxtest/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmpqirb9t5l Check the logs for full command output.
ERROR: Command errored out with exit status 1: /home/florian/tmp/toxtest/.venv/bin/python3 /home/florian/tmp/toxtest/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmpqirb9t5l Check the logs for full command output.

I'm not sure if this is an intended change in Cython 3, or a bug - but given that "pyyaml setup is still relying on a bunch of ancient custom distutils bits" (#584) I thought I'd report it here first. Please let me know if you think I should rather report it there.

[nitzmahone]

It's definitely a regression between Cython 3.0.0a9 and a10; every previous 3.0.0 alpha has been able to build the pyyaml extension fine. This PR (included in a10) swapped out the base class on Cython's custom build_ext distutils command (which pyyaml further subclasses to customize some stuff). I haven't dug down all the way yet to see if it's something we should try to address in pyyaml's customization or if it's just a bug in the new base class.

Thanks for the report- regardless, we'll want to get this sorted one way or another, so I'll figure out where the problem is and work with the Cython/setuptools folks if necessary to get it working.

[tomplex]

I encountered this error when attempting to lock my dependencies with pipenv; I was able to fix it in my case by setting

[pipenv]
allow_prereleases = false

just figured I'd share this in case it helps someone else.

[nitzmahone]

allow_prereleases = false

That'll only work until Cython 3.0 releases, unless we're able to get a change made over there (which it sounds like they're amenable to, if it's not too gnarly)

[RobertoPrevato]

I don't know if this can be of interest for anyone reading here, but I had the same issue in a GitHub Workflow, only happening on Python 3.10. The failed build can be seen here: https://github.com/Neoteroi/BlackSheep/runs/5098546591?check_suite_focus=true

In my case, PyYAML is a dependency and I was testing another wheel with this command:

pip install --pre blacksheep -f "file:///${GITHUB_WORKSPACE}/dist"

In my case, removing the --pre option fixed the problem.

pip install blacksheep -f "file:///${GITHUB_WORKSPACE}/dist"

[nitzmahone]

Yep, unfortunately --pre seems to affect all deps and build deps as well under pip.

[mgorny]

Ping. Cython 3.0.0 has already reached beta releases, so we need some solution sooner than later.

[nitzmahone]

Cool- we've been poking at doing another release, and since we sat on it long enough, setuptools and pip now fully support PEP517/518 config passthru from the CLI, which should make the replacement of the dynamic bits a whole lot easier.

[jmakov]

Stumbled upon this bug while trying to install Pycaret:

Collecting PyYAML~=5.1                                                                                                                   
#0 30.24   Downloading PyYAML-5.4.1.tar.gz (175 kB)                                                                                               #0 30.24      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 175.1/175.1 kB 34.3 MB/s eta 0:00:00
#0 30.32   Installing build dependencies: started                                                                                                 
#0 33.08   Installing build dependencies: finished with status 'done'                                                                             
#0 33.08   Getting requirements to build wheel: started                  
#0 33.19   Getting requirements to build wheel: finished with status 'error'                                                                      #0 33.20   error: subprocess-exited-with-error                                                                                                    
#0 33.20                                                                                                                                          
#0 33.20   × Getting requirements to build wheel did not run successfully.
#0 33.20   │ exit code: 1                                                                                                                         
#0 33.20   ╰─> [50 lines of output]                                      
#0 33.20       /tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/config/setupcfg.py:516: SetuptoolsDeprecationWarning: T
he license_file parameter is deprecated, use license_files instead.
#0 33.20         warnings.warn(msg, warning_class)                                                                                                
#0 33.20       running egg_info                                          
#0 33.20       writing lib3/PyYAML.egg-info/PKG-INFO                                                                                              
#0 33.20       writing dependency_links to lib3/PyYAML.egg-info/dependency_links.txt                                                              
#0 33.20       writing top-level names to lib3/PyYAML.egg-info/top_level.txt                                                                      
#0 33.20       Traceback (most recent call last):
#0 33.20         File "/opt/conda/envs/rapids/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <
module>
#0 33.20           main()
#0 33.20         File "/opt/conda/envs/rapids/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in m
ain
#0 33.20           json_out['return_val'] = hook(**hook_input['kwargs'])
#0 33.20         File "/opt/conda/envs/rapids/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in g
et_requires_for_build_wheel
#0 33.20           return hook(config_settings)
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 338, in get_requires_for_b
uild_wheel
#0 33.20           return self._get_build_requires(config_settings, requirements=['wheel'])
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 320, in _get_build_require
s
#0 33.20           self.run_setup()
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 335, in run_setup
#0 33.20           exec(code, locals())
#0 33.20         File "<string>", line 271, in <module>
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/__init__.py", line 108, in setup
#0 33.20           return distutils.core.setup(**attrs)
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 185, in setup
#0 33.20           return run_commands(dist)
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
#0 33.20           dist.run_commands()
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
#0 33.20           self.run_command(cmd)
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 1221, in run_command
#0 33.20           super().run_command(command)
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
#0 33.20           cmd_obj.run()
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 318, in run
#0 33.20           self.find_sources()
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 326, in find_sources
#0 33.20           mm.run()
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 570, in run
#0 33.20           self.add_defaults()
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 608, in add_defaults
#0 33.20           sdist.add_defaults(self)
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/command/sdist.py", line 106, in add_defaults
#0 33.20           super().add_defaults()
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/sdist.py", line 251, in add_defaults
#0 33.20           self._add_defaults_ext()
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/sdist.py", line 336, in _add_defaults_ext
#0 33.20           self.filelist.extend(build_ext.get_source_files())
#0 33.20         File "<string>", line 201, in get_source_files
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 107, in __getattr__
#0 33.20           raise AttributeError(attr)
#0 33.20       AttributeError: cython_sources
#0 33.20       [end of output]

[nitzmahone]

@jmakov the likely short-term fix will be to pin the Cython PEP518 build deps to pre-3.0 Cython in a 6.x release (see #702), but we generally consider the 5.x release branch dead, so hadn't planned on re-publishing the build change back there. You might want to try and get the Pycaret folks (or whoever's ultimately pinning PyYAML to ~=5.1) to update their deps.

[jmakov]

@nitzmahone thank you so much for pointing out the solution! Will link your answer to an issue in their repo.

[nitzmahone]

@jmakov after looking at the issue you just linked, now I think I see how you ended up building PyYAML with an unreleased Cython 3.x; IIRC pip's --pre option globally allows pre-releases to be considered for installation (so for the package you asked for, as well as its entire dep tree and any build requirements pip installs).

If you just ask pip for for a specific pre-release version of Pycaret (without using the --pre flag, eg pip install pycaret==3.0.0rc8), the normal runtime and build-dep rules will be applied and PyYAML shouldn't try to build with an unreleased Cython.

[MagicalTux]

Managed to get PyYAML compile with python 3.12 by adding Cython 0.29.37 at the beginning of PYTHONPATH.

[nanonyme]

It's not necessarily to document workarounds. You can take pip or pypa/build and keep build isolation on. (which is the default) This will install compatible dependencies into virtualenv/venv to produce a wheel. You can then install wheel outside virtualenv/venv. This issue is about fixing compatibility with new Cython.

[benorgera]

Any update on this?