Python requirements installation fails due to new Cython3 release

[jnasselle]

Description

Related issues:

• Error installing Pyyaml==5.4, Cython_sources yaml/pyyaml#724
• "AttributeError: cython_sources" with Cython 3.0.0a10 yaml/pyyaml#601

Current behavior

The integration test's Python requirements (located in requirements.txt) are failing due to the new Cython3 version released https://pypi.org/project/Cython/3.0.0/#history

15:12:44  fatal: [ID2_B41137_Test_integration_ubuntu_agent]: FAILED! => {
15:12:44      "changed": true,
15:12:44      "cmd": [
15:12:44          "/usr/bin/python3",
15:12:44          "-m",
15:12:44          "pip",
15:12:44          "install",
15:12:44          "-r",
15:12:44          "requirements.txt",
15:12:44          "--no-cache-dir",
15:12:44          "--upgrade",
15:12:44          "--only-binary=:cryptography,grpcio:"
15:12:44      ],
15:12:44      "delta": "0:00:09.799239",
15:12:44      "end": "2023-07-17 18:12:44.908280",
15:12:44      "invocation": {
15:12:44          "module_args": {
15:12:44              "_raw_params": "/usr/bin/python3 -m pip install -r requirements.txt --no-cache-dir --upgrade --only-binary=:cryptography,grpcio:",
15:12:44              "_uses_shell": false,
15:12:44              "argv": null,
15:12:44              "chdir": "/tmp/Test_integration_B41137_20230717185851",
15:12:44              "creates": null,
15:12:44              "executable": null,
15:12:44              "removes": null,
15:12:44              "stdin": null,
15:12:44              "stdin_add_newline": true,
15:12:44              "strip_empty_ends": true,
15:12:44              "warn": true
15:12:44          }
15:12:44      },
15:12:44      "rc": 1,
15:12:44      "start": "2023-07-17 18:12:35.109041"
15:12:44  }
15:12:44  
15:12:44  STDOUT:
15:12:44  
15:12:44  Ignoring pytest: markers 'python_version <= "3.9"' don't match your environment
15:12:44  Ignoring jq: markers '(platform_system == "Linux" or platform_system == "Darwin") and python_version <= "3.9"' don't match your environment
15:12:44  Ignoring wmi: markers 'platform_system == "Windows"' don't match your environment
15:12:44  Ignoring libcst: markers 'python_version <= "3.6"' don't match your environment
15:12:44  Requirement already satisfied: configobj==5.0.6 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 1)) (5.0.6)
15:12:44  Collecting certifi==2020.12.5 (from -r requirements.txt (line 2))
15:12:44    Downloading certifi-2020.12.5-py2.py3-none-any.whl (147 kB)
15:12:44       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 147.5/147.5 kB 2.7 MB/s eta 0:00:00
15:12:44  Collecting cffi>=1.14.0 (from -r requirements.txt (line 3))
15:12:44    Downloading cffi-1.15.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (441 kB)
15:12:44       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 441.8/441.8 kB 39.8 MB/s eta 0:00:00
15:12:44  Collecting cycler>=0.10 (from -r requirements.txt (line 4))
15:12:44    Downloading cycler-0.11.0-py3-none-any.whl (6.4 kB)
15:12:44  Requirement already satisfied: bcrypt==3.2.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 5)) (3.2.0)
15:12:44  Collecting distro==1.3.0 (from -r requirements.txt (line 6))
15:12:44    Downloading distro-1.3.0-py2.py3-none-any.whl (16 kB)
15:12:44  Collecting filetype==1.0.7 (from -r requirements.txt (line 7))
15:12:44    Downloading filetype-1.0.7-py2.py3-none-any.whl (15 kB)
15:12:44  Collecting freezegun==0.3.15 (from -r requirements.txt (line 8))
15:12:44    Downloading freezegun-0.3.15-py2.py3-none-any.whl (14 kB)
15:12:44  Collecting grpcio>=1.27.2 (from -r requirements.txt (line 9))
15:12:44    Obtaining dependency information for grpcio>=1.27.2 from [https://files.pythonhosted.org/packages/a9/6e/fa9785a95e70d60b9b2b75944e422b06c62eaed183abdfcdc0b7812b1e21/grpcio-1.56.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata](https://files.pythonhosted.org/packages/a9/6e/fa9785a95e70d60b9b2b75944e422b06c62eaed183abdfcdc0b7812b1e21/grpcio-1.56.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata%1B[0m)
15:12:44    Downloading grpcio-1.56.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.0 kB)
15:12:44  Collecting google-cloud-pubsub==2.3.0 (from -r requirements.txt (line 10))
15:12:44    Downloading google_cloud_pubsub-2.3.0-py2.py3-none-any.whl (210 kB)
15:12:44       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 210.2/210.2 kB 204.0 MB/s eta 0:00:00
15:12:44  Requirement already satisfied: jsonschema==3.2.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 11)) (3.2.0)
15:12:44  Collecting kiwisolver>=1.0.1 (from -r requirements.txt (line 12))
15:12:44    Downloading kiwisolver-1.4.4-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.6 MB)
15:12:44       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.6/1.6 MB 156.3 MB/s eta 0:00:00
15:12:44  Collecting lockfile==0.12.2 (from -r requirements.txt (line 13))
15:12:44    Downloading lockfile-0.12.2-py2.py3-none-any.whl (13 kB)
15:12:44  Collecting matplotlib>=3.3.4 (from -r requirements.txt (line 14))
15:12:44    Obtaining dependency information for matplotlib>=3.3.4 from [https://files.pythonhosted.org/packages/c2/da/a5622266952ab05dc3995d77689cba600e49ea9d6c51d469c077695cb719/matplotlib-3.7.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata](https://files.pythonhosted.org/packages/c2/da/a5622266952ab05dc3995d77689cba600e49ea9d6c51d469c077695cb719/matplotlib-3.7.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata%1B[0m)
15:12:44    Downloading matplotlib-3.7.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.6 kB)
15:12:44  Collecting netifaces==0.10.9 (from -r requirements.txt (line 15))
15:12:44    Downloading netifaces-0.10.9.tar.gz (28 kB)
15:12:44    Preparing metadata (setup.py): started
15:12:44    Preparing metadata (setup.py): finished with status 'done'
15:12:44  Collecting numpy>=1.18.1 (from -r requirements.txt (line 16))
15:12:44    Obtaining dependency information for numpy>=1.18.1 from [https://files.pythonhosted.org/packages/d0/55/559e6f455a066e12058330377259a106b7fefa41c15dbdb1b71070cec429/numpy-1.25.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata](https://files.pythonhosted.org/packages/d0/55/559e6f455a066e12058330377259a106b7fefa41c15dbdb1b71070cec429/numpy-1.25.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata%1B[0m)
15:12:44    Downloading numpy-1.25.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.6 kB)
15:12:44  Collecting pandas>=1.1.5 (from -r requirements.txt (line 17))
15:12:44    Obtaining dependency information for pandas>=1.1.5 from [https://files.pythonhosted.org/packages/e3/59/35a2892bf09ded9c1bf3804461efe772836a5261ef5dfb4e264ce813ff99/pandas-2.0.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata](https://files.pythonhosted.org/packages/e3/59/35a2892bf09ded9c1bf3804461efe772836a5261ef5dfb4e264ce813ff99/pandas-2.0.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata%1B[0m)
15:12:44    Downloading pandas-2.0.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (18 kB)
15:12:44  Collecting pillow>=6.2.0 (from -r requirements.txt (line 18))
15:12:44    Obtaining dependency information for pillow>=6.2.0 from [https://files.pythonhosted.org/packages/3d/36/e78f09d510354977e10102dd811e928666021d9c451e05df962d56477772/Pillow-10.0.0-cp310-cp310-manylinux_2_28_x86_64.whl.metadata](https://files.pythonhosted.org/packages/3d/36/e78f09d510354977e10102dd811e928666021d9c451e05df962d56477772/Pillow-10.0.0-cp310-cp310-manylinux_2_28_x86_64.whl.metadata%1B[0m)
15:12:44    Downloading Pillow-10.0.0-cp310-cp310-manylinux_2_28_x86_64.whl.metadata (9.5 kB)
15:12:44  Collecting psutil>=5.6.6 (from -r requirements.txt (line 19))
15:12:44    Downloading psutil-5.9.5-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (282 kB)
15:12:44       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 282.1/282.1 kB 208.4 MB/s eta 0:00:00
15:12:44  Collecting py~=1.10.0 (from -r requirements.txt (line 20))
15:12:44    Downloading py-1.10.0-py2.py3-none-any.whl (97 kB)
15:12:44       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 97.6/97.6 kB 181.2 MB/s eta 0:00:00
15:12:44  Collecting pycryptodome>=3.9.8 (from -r requirements.txt (line 21))
15:12:44    Obtaining dependency information for pycryptodome>=3.9.8 from [https://files.pythonhosted.org/packages/40/88/89623388754162ddb82c62814ccbfdbfcc8ed9bd6d5f7412d2479bdca3a7/pycryptodome-3.18.0-cp35-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata](https://files.pythonhosted.org/packages/40/88/89623388754162ddb82c62814ccbfdbfcc8ed9bd6d5f7412d2479bdca3a7/pycryptodome-3.18.0-cp35-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata%1B[0m)
15:12:44    Downloading pycryptodome-3.18.0-cp35-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.4 kB)
15:12:44  Collecting pyOpenSSL==19.1.0 (from -r requirements.txt (line 22))
15:12:44    Downloading pyOpenSSL-19.1.0-py2.py3-none-any.whl (53 kB)
15:12:44       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 53.7/53.7 kB 167.3 MB/s eta 0:00:00
15:12:44  Collecting pytest-html==3.1.1 (from -r requirements.txt (line 23))
15:12:44    Downloading pytest_html-3.1.1-py3-none-any.whl (14 kB)
15:12:44  Collecting pytest==7.1.2 (from -r requirements.txt (line 25))
15:12:44    Downloading pytest-7.1.2-py3-none-any.whl (297 kB)
15:12:44       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 297.0/297.0 kB 201.8 MB/s eta 0:00:00
15:12:44  Collecting pyyaml==5.4 (from -r requirements.txt (line 26))
15:12:44    Downloading PyYAML-5.4.tar.gz (174 kB)
15:12:44       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 174.8/174.8 kB 201.2 MB/s eta 0:00:00
15:12:44    Installing build dependencies: started
15:12:44    Installing build dependencies: finished with status 'done'
15:12:44    Getting requirements to build wheel: started
15:12:44    Getting requirements to build wheel: finished with status 'error'
15:12:44  
15:12:44  
15:12:44  STDERR:
15:12:44  
15:12:44    error: subprocess-exited-with-error
15:12:44    
15:12:44    × Getting requirements to build wheel did not run successfully.
15:12:44    │ exit code: 1
15:12:44    ╰─> [62 lines of output]
15:12:44        /tmp/pip-build-env-zd5qrofd/overlay/local/lib/python3.10/dist-packages/setuptools/config/setupcfg.py:293: _DeprecatedConfig: Deprecated config in `setup.cfg`
15:12:44        !!
15:12:44        
15:12:44                ********************************************************************************
15:12:44                The license_file parameter is deprecated, use license_files instead.
15:12:44        
15:12:44                By 2023-Oct-30, you need to update your project and remove deprecated calls
15:12:44                or your builds will no longer be supported.
15:12:44        
15:12:44                See https://setuptools.pypa.io/en/latest/userguide/declarative_config.html for details.
15:12:44                ********************************************************************************
15:12:44        
15:12:44        !!
15:12:44          parsed = self.parsers.get(option_name, lambda x: x)(value)
15:12:44        running egg_info
15:12:44        writing lib3/PyYAML.egg-info/PKG-INFO
15:12:44        writing dependency_links to lib3/PyYAML.egg-info/dependency_links.txt
15:12:44        writing top-level names to lib3/PyYAML.egg-info/top_level.txt
15:12:44        Traceback (most recent call last):
15:12:44          File "/usr/local/lib/python3.10/dist-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
15:12:44            main()
15:12:44          File "/usr/local/lib/python3.10/dist-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
15:12:44            json_out['return_val'] = hook(**hook_input['kwargs'])
15:12:44          File "/usr/local/lib/python3.10/dist-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
15:12:44            return hook(config_settings)
15:12:44          File "/tmp/pip-build-env-zd5qrofd/overlay/local/lib/python3.10/dist-packages/setuptools/build_meta.py", line 341, in get_requires_for_build_wheel
15:12:44            return self._get_build_requires(config_settings, requirements=['wheel'])
15:12:44          File "/tmp/pip-build-env-zd5qrofd/overlay/local/lib/python3.10/dist-packages/setuptools/build_meta.py", line 323, in _get_build_requires
15:12:44            self.run_setup()
15:12:44          File "/tmp/pip-build-env-zd5qrofd/overlay/local/lib/python3.10/dist-packages/setuptools/build_meta.py", line 338, in run_setup
15:12:44            exec(code, locals())
15:12:44          File "<string>", line 271, in <module>
15:12:44          File "/tmp/pip-build-env-zd5qrofd/overlay/local/lib/python3.10/dist-packages/setuptools/__init__.py", line 107, in setup
15:12:44            return distutils.core.setup(**attrs)
15:12:44          File "/tmp/pip-build-env-zd5qrofd/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/core.py", line 185, in setup
15:12:44            return run_commands(dist)
15:12:44          File "/tmp/pip-build-env-zd5qrofd/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/core.py", line 201, in run_commands
15:12:44            dist.run_commands()
15:12:44          File "/tmp/pip-build-env-zd5qrofd/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/dist.py", line 969, in run_commands
15:12:44            self.run_command(cmd)
15:12:44          File "/tmp/pip-build-env-zd5qrofd/overlay/local/lib/python3.10/dist-packages/setuptools/dist.py", line 1234, in run_command
15:12:44            super().run_command(command)
15:12:44          File "/tmp/pip-build-env-zd5qrofd/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/dist.py", line 988, in run_command
15:12:44            cmd_obj.run()
15:12:44          File "/tmp/pip-build-env-zd5qrofd/overlay/local/lib/python3.10/dist-packages/setuptools/command/egg_info.py", line 314, in run
15:12:44            self.find_sources()
15:12:44          File "/tmp/pip-build-env-zd5qrofd/overlay/local/lib/python3.10/dist-packages/setuptools/command/egg_info.py", line 322, in find_sources
15:12:44            mm.run()
15:12:44          File "/tmp/pip-build-env-zd5qrofd/overlay/local/lib/python3.10/dist-packages/setuptools/command/egg_info.py", line 551, in run
15:12:44            self.add_defaults()
15:12:44          File "/tmp/pip-build-env-zd5qrofd/overlay/local/lib/python3.10/dist-packages/setuptools/command/egg_info.py", line 589, in add_defaults
15:12:44            sdist.add_defaults(self)
15:12:44          File "/tmp/pip-build-env-zd5qrofd/overlay/local/lib/python3.10/dist-packages/setuptools/command/sdist.py", line 104, in add_defaults
15:12:44            super().add_defaults()
15:12:44          File "/tmp/pip-build-env-zd5qrofd/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/command/sdist.py", line 251, in add_defaults
15:12:44            self._add_defaults_ext()
15:12:44          File "/tmp/pip-build-env-zd5qrofd/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/command/sdist.py", line 336, in _add_defaults_ext
15:12:44            self.filelist.extend(build_ext.get_source_files())
15:12:44          File "<string>", line 201, in get_source_files
15:12:44          File "/tmp/pip-build-env-zd5qrofd/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/cmd.py", line 107, in __getattr__
15:12:44            raise AttributeError(attr)
15:12:44        AttributeError: cython_sources
15:12:44        [end of output]
15:12:44    
15:12:44    note: This error originates from a subprocess, and is likely not a problem with pip.
15:12:44  error: subprocess-exited-with-error
15:12:44  
15:12:44  × Getting requirements to build wheel did not run successfully.
15:12:44  │ exit code: 1
15:12:44  ╰─> See above for output.
15:12:44  
15:12:44  note: This error originates from a subprocess, and is likely not a problem with pip.
15:12:44  
15:12:44  
15:12:44  MSG:
15:12:44  
15:12:44  non-zero return code

Expected behavior

Overcome this situation in order to execute integration tests, both locally or using Jenkins

[jnasselle]

Update

• Solution comes with pyyaml 6.0.1 https://github.com/yaml/pyyaml/blob/6.0.1/CHANGES
• After checking pyyaml API changes from 5.4 (current used version) and 6.0.1, the only thing was yaml.load(s, Loader=...) requires Loader= argument always yaml/pyyaml#561, but the code already meet this new constraint

Proposal

• Upgrade pyyaml to 6.0.1

Change traceability

• wazuh/wazuh-qa
  • https://github.com/wazuh/wazuh-qa/blob/4.5.0/requirements.txt
  • https://github.com/wazuh/wazuh-qa/blob/4.5.0/tests/system/requirements.txt
  • https://github.com/wazuh/wazuh-qa/blob/4.5.0/tests/system/test_jwt_invalidation/requirements.txt
• wazuh/wazuh-jenkins
  • https://github.com/wazuh/wazuh-jenkins/blob/4.5.0/ansible-playbooks/packages/unattended/provision.yml#L28
  • https://github.com/wazuh/wazuh-jenkins/blob/4.5.0/quality/tests/performance/fim/roles/provision/tasks/main.yml#L66
  • https://github.com/wazuh/wazuh-jenkins/blob/4.5.0/scripts/StackCheckFiles/requirements.txt#L3
• wazuh/wazuh
  • https://github.com/wazuh/wazuh/blob/4.5.0/framework/requirements.txt#L73