reloc/build_reloc.sh: `AttributeError: cython_sources` when building PyYAML #78

tchaikov · 2024-04-24T10:25:26Z

$ cat /etc/fedora-release 
Fedora release 40 (Forty)
$ cython --version
Cython version 3.0.9
$ reloc/build_reloc.sh --version scylla-5.5-1234 --nodeps         
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
poetry 1.7.1 requires build<2.0.0,>=1.0.3, but you have build 0.10.0 which is incompatible.      
* Creating virtualenv isolated environment...
* Installing packages in isolated environment... (Cython, setuptools>=42, setuptools_scm[toml]>=6.2, wheel)
* Getting build dependencies for sdist...
Compiling cqlshlib/copyutil.py because it changed.                                                                     
[1/1] Cythonizing cqlshlib/copyutil.py
/tmp/build-env-fi88_xxy/lib/python3.12/site-packages/setuptools_scm/version.py:102: UserWarning: tag 'v6.0.16-scylla' will be stripped of its suffix '-scylla'
  warnings.warn(f"tag {tag!r} will be stripped of its suffix {suffix!r}")
* Building sdist...                                                                                                    
/tmp/build-env-fi88_xxy/lib/python3.12/site-packages/setuptools_scm/version.py:102: UserWarning: tag 'v6.0.16-scylla' will be stripped of its suffix '-scylla'
  warnings.warn(f"tag {tag!r} will be stripped of its suffix {suffix!r}")                              
'cqlshlib' not a regular file -- skipping
Successfully built scylla_cqlsh-6.0.17.dev5+g3baa678.tar.gz                                                            
  error: subprocess-exited-with-error
                                                                                                                                                                                                                                              
  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1                                           
  ╰─> [55 lines of output]                            
      running dist_info                                                                                                
      creating /tmp/pip-modern-metadata-yjpfnhok/PyYAML.egg-info
      writing /tmp/pip-modern-metadata-yjpfnhok/PyYAML.egg-info/PKG-INFO
      writing dependency_links to /tmp/pip-modern-metadata-yjpfnhok/PyYAML.egg-info/dependency_links.txt
      writing top-level names to /tmp/pip-modern-metadata-yjpfnhok/PyYAML.egg-info/top_level.txt
      writing manifest file '/tmp/pip-modern-metadata-yjpfnhok/PyYAML.egg-info/SOURCES.txt'
      Traceback (most recent call last):
        File "/usr/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()                                      
        File "/usr/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^  
        File "/usr/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 149, in prepare_metadata_for_build_wheel
          return hook(metadata_directory, config_settings)                                                             
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/usr/lib/python3.12/site-packages/setuptools/build_meta.py", line 366, in prepare_metadata_for_build_wheel
          self.run_setup()
        File "/usr/lib/python3.12/site-packages/setuptools/build_meta.py", line 311, in run_setup
          exec(code, locals())
        File "<string>", line 288, in <module>
        File "/usr/lib/python3.12/site-packages/setuptools/__init__.py", line 103, in setup
          return distutils.core.setup(**attrs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/usr/lib/python3.12/site-packages/setuptools/_distutils/core.py", line 185, in setup
          return run_commands(dist)
                 ^^^^^^^^^^^^^^^^^^
        File "/usr/lib/python3.12/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
          dist.run_commands()
        File "/usr/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "/usr/lib/python3.12/site-packages/setuptools/dist.py", line 963, in run_command
          super().run_command(command)
        File "/usr/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/usr/lib/python3.12/site-packages/setuptools/command/dist_info.py", line 91, in run
          self.egg_info.run()
        File "/usr/lib/python3.12/site-packages/setuptools/command/egg_info.py", line 321, in run
          self.find_sources()
        File "/usr/lib/python3.12/site-packages/setuptools/command/egg_info.py", line 329, in find_sources
          mm.run()
        File "/usr/lib/python3.12/site-packages/setuptools/command/egg_info.py", line 551, in run
          self.add_defaults()
        File "/usr/lib/python3.12/site-packages/setuptools/command/egg_info.py", line 589, in add_defaults
          sdist.add_defaults(self)
        File "/usr/lib/python3.12/site-packages/setuptools/command/sdist.py", line 112, in add_defaults
          super().add_defaults()
        File "/usr/lib/python3.12/site-packages/setuptools/_distutils/command/sdist.py", line 251, in add_defaults
          self._add_defaults_ext()
        File "/usr/lib/python3.12/site-packages/setuptools/_distutils/command/sdist.py", line 336, in _add_defaults_ext 
          self.filelist.extend(build_ext.get_source_files())
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "<string>", line 204, in get_source_files
        File "/usr/lib/python3.12/site-packages/setuptools/_distutils/cmd.py", line 107, in __getattr__
          raise AttributeError(attr)
      AttributeError: cython_sources
      [end of output]
   
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

looks like yaml/pyyaml#601 . but i still have the same error after unpinning or bumping pyyaml to 6.0.1 in requirement.txt.

The text was updated successfully, but these errors were encountered:

fruch · 2024-05-09T12:20:38Z

@tchaikov it's need to be pinned also in the pyproject.toml

tchaikov · 2024-05-09T12:27:10Z

$ git diff
diff --git a/pyproject.toml b/pyproject.toml
index ad9691e..154a301 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -1,5 +1,6 @@
 [build-system]
 requires = [
+    "PyYAML==6.0.1",
     "setuptools>=42",
     "wheel",
     "Cython",
@@ -9,4 +10,4 @@ build-backend = "setuptools.build_meta"

 [tool.setuptools_scm]
 write_to = "pylib/cqlshlib/_version.py"
-tag_regex = '^(?P<prefix>v)?(?P<version>[^\+-]+)(?P<suffix>-scylla)?$'
\ No newline at end of file
+tag_regex = '^(?P<prefix>v)?(?P<version>[^\+-]+)(?P<suffix>-scylla)?$'
diff --git a/requirements.txt b/requirements.txt
index 3b3c264..e2e12ab 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,5 +1,5 @@
 scylla-driver==3.26.5
 geomet==0.2.1.post1
-PyYAML==6.0
+PyYAML==6.0.1
 click==8.1.3
$ reloc/build_reloc.sh --version scylla-5.5-1234 --nodeps
# ...
        File "/usr/lib/python3.12/site-packages/setuptools/_distutils/command/sdist.py", line 251, in add_defaults
          self._add_defaults_ext()
        File "/usr/lib/python3.12/site-packages/setuptools/_distutils/command/sdist.py", line 336, in _add_defaults_ext
          self.filelist.extend(build_ext.get_source_files())
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "<string>", line 205, in get_source_files
        File "/usr/lib/python3.12/site-packages/setuptools/_distutils/cmd.py", line 107, in __getattr__
          raise AttributeError(attr)
      AttributeError: cython_sources
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

tchaikov · 2024-05-09T12:28:19Z

i must be missing something?

fruch · 2024-05-09T12:36:53Z

i must be missing something?

I've managed to reproduce it

when uninstalling cython completely it worked
also when doing pip install 'cython<3' it worked

so seems like we need to make sure the env we are running doesn't have cython or have one lower then 3

tchaikov · 2024-05-09T12:45:30Z

yeah, but IMHO, to remove Cython from system is the recipe to bad performance for the applications which use Cython. not to mention some of them rely on Cython to build. i knew, there are some workaround in this regard. but i always prefer using the packaged Cython.

and use Cython < 3, is exactly one of the workarounds proposed by yaml/pyyaml#601 .

anyway, we need to be prepared for this.

fruch · 2024-05-09T12:46:26Z

one more detail the failing command is not the build of the package, but the download command
that need to fetch all dependencies

it uses --no-build-isolation which mean it doesn't download it's own set of versions, and uses what there.
the reason it was used since we are using --no-binary :all: to download the source packages

but that's affect also the build dependency which would make it extremely slow, compiled cython from source

for downloading the source packages dependecies, we use the `pip download` command and we uses `--no-build-isolation` which mean it doesn't download it's own set of versions, with using using `--no-binary :all:` to download the source packages but that's affect also the build dependency which would make it extremely slow, compile cython from source. so first pin the cython in the build dependecies, and force the command download cython always as wheel to avoid building it from source Fixes: scylladb#78

for downloading the source packages dependecies, we use the `pip download` command and we uses `--no-build-isolation` which mean it doesn't download it's own set of versions, with using using `--no-binary :all:` to download the source packages but that's affect also the build dependency which would make it extremely slow, compile cython from source. so first pin the cython in the build dependecies, and force the command download cython always as wheel to avoid building it from source Fixes: #78

fruch mentioned this issue May 9, 2024

reloc/build_reloc.sh: don't use --no-build-isolation #86

Merged

fruch closed this as completed in #86 May 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

reloc/build_reloc.sh: `AttributeError: cython_sources` when building PyYAML #78

reloc/build_reloc.sh: `AttributeError: cython_sources` when building PyYAML #78

tchaikov commented Apr 24, 2024

fruch commented May 9, 2024

tchaikov commented May 9, 2024

tchaikov commented May 9, 2024

fruch commented May 9, 2024

tchaikov commented May 9, 2024

fruch commented May 9, 2024

reloc/build_reloc.sh: AttributeError: cython_sources when building PyYAML #78

reloc/build_reloc.sh: AttributeError: cython_sources when building PyYAML #78

Comments

tchaikov commented Apr 24, 2024

fruch commented May 9, 2024

tchaikov commented May 9, 2024

tchaikov commented May 9, 2024

fruch commented May 9, 2024

tchaikov commented May 9, 2024

fruch commented May 9, 2024

reloc/build_reloc.sh: `AttributeError: cython_sources` when building PyYAML #78

reloc/build_reloc.sh: `AttributeError: cython_sources` when building PyYAML #78