Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[PyOV] Enable Python 3.10 on Azure CI #12578

Merged
merged 258 commits into from Nov 8, 2022

Conversation

p-wysocki
Copy link
Contributor

@p-wysocki p-wysocki commented Aug 16, 2022

Details:

  • Enable Python 3.10 on Azure CI
  • Enable building openvino_dev wheel on Python 3.10

Tickets:

  • 89605

tar -xzf Python-"$PYTHON_VERSION".tgz
cd Python-"$PYTHON_VERSION" || exit
./configure --enable-optimizations --prefix=/usr/local --enable-shared LDFLAGS="-Wl,-rpath /usr/local/lib"
make python
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

here and maybe in other places we are just wasting machine time with building python each time.
I still suppose we need to enable python 3.10 only on Jenkins, while on Azure we should use default python version for the system (for Windows we can install 3.10 since Windows does not have python by default)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Answered in an internal channel.

@ilya-lavrenov
Copy link
Contributor

ilya-lavrenov commented Nov 6, 2022

Please, have a look at pip conflicts issue:

[2022-11-06T00:14:29.375Z] [2022-11-06 00:14:29,189] [140019797665536] install 3.9_pip_22.2.2 nncf INFO: Installing ninja-1.11.1-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl to /home/jenkins/agent/workspace/private-ci/ie/e2e-tests-linux-ubuntu20-pip-conflicts/b/tmp/check_pkgs_conflicts_venv_3.9_pip_22.2.2/lib/python3.9/site-packages
[2022-11-06T00:14:29.375Z] [2022-11-06 00:14:29,224] [140019797665536] install 3.9_pip_22.2.2 nncf INFO: Adding ninja 1.11.1 to easy-install.pth file
[2022-11-06T00:14:29.375Z] [2022-11-06 00:14:29,231] [140019797665536] install 3.9_pip_22.2.2 nncf INFO: Installing ninja script to /home/jenkins/agent/workspace/private-ci/ie/e2e-tests-linux-ubuntu20-pip-conflicts/b/tmp/check_pkgs_conflicts_venv_3.9_pip_22.2.2/bin
[2022-11-06T00:14:29.375Z] [2022-11-06 00:14:29,231] [140019797665536] install 3.9_pip_22.2.2 nncf INFO: 
[2022-11-06T00:14:29.375Z] [2022-11-06 00:14:29,231] [140019797665536] install 3.9_pip_22.2.2 nncf INFO: Installed /home/jenkins/agent/workspace/private-ci/ie/e2e-tests-linux-ubuntu20-pip-conflicts/b/tmp/check_pkgs_conflicts_venv_3.9_pip_22.2.2/lib/python3.9/site-packages/ninja-1.11.1-py3.9-linux-x86_64.egg
[2022-11-06T00:14:29.375Z] [2022-11-06 00:14:29,270] [140019797665536] install 3.9_pip_22.2.2 nncf INFO: error: numpy 1.23.1 is installed but numpy<1.23.0,>=1.16.5 is required by {'scipy'}
[2022-11-06T00:14:29.375Z] [2022-11-06 00:14:29,295] [140019797665536] install 3.9_pip_22.2.2 nncf INFO: ========== Completed. Exit code: 1, elapsed time: 9.486 sec, 0:00:09
[2022-11-06T00:14:29.375Z] [2022-11-06 00:14:29,296] [140019797665536] install 3.9_pip_22.2.2 nncf INFO: 
[2022-11-06T00:14:29.375Z] [2022-11-06 00:14:29,305] [140019797665536] install 3.9_pip_22.2.2 nncf ERROR: ========== ERRORS were found in the output:
[2022-11-06T00:14:29.375Z] [2022-11-06 00:14:29,305] [140019797665536] install 3.9_pip_22.2.2 nncf ERROR: error: numpy 1.23.1 is installed but numpy<1.23.0,>=1.16.5 is required by {'scipy'}
[2022-11-06T00:14:29.375Z] [2022-11-06 00:14:29,305] [140019797665536] install 3.9_pip_22.2.2 nncf WARNING: ========== See full log for details regarding the errors
[2022-11-06T00:14:29.375Z] [2022-11-06 00:14:29,306] [140019844339520] TaskTreeExec ERROR: Task 'install 3.9_pip_22.2.2 nncf' generated an exception: Task 'install 3.9_pip_22.2.2 nncf' - FAILED. Exit code: 1
[2022-11-06T00:14:29.375Z] Traceback (most recent call last):

And resolve merge conflicts.

@p-wysocki
Copy link
Contributor Author

The pip conflicts issue is fixed by bumping NNCF commit on CI, the PR is currently waiting for review and merge. I'll resolve conflicts later to not waste CI's time.

attrs==19.1.0
pytest==4.0.1; python_version < '3.10'
pytest==7.2.0; python_version >= '3.10'
py>=1.9.0
Copy link
Contributor

@eaidova eaidova Nov 6, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are you sure that py package requires to install separately?
as I know pytest 7.2.0 removes dependency on it
pytest-dev/pytest#10396
P.S. this package has vulnerability issue, from product perspective, it can lead to problems if we will require it

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to support a range of pytest versions due to multiple different Python versions used (and other repositories' dependencies). The reason it's added is there were CIs failing due to this package not being installed in the environment. Adding it helped with the issue.

Copy link
Contributor

@jiwaszki jiwaszki Nov 6, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@eaidova BTW it is still open how to score this problem and other additional packages are still using py if you look at some comments like: pytest-dev/py#287 (comment) or here tox-dev/tox#2524

It is still on-going effort to drop it everywhere and some of checks still show the issue.
I think we should be dealing with it in separate issue - not here.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not insist, just highlight potential problem which these changes bring

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

However if py is a problem, project will face it eventually, even without this change. This package is already a dependency of others... probably a new task is coming.

Thanks for noticing it!

@ilya-lavrenov ilya-lavrenov merged commit d2871fd into openvinotoolkit:master Nov 8, 2022
@p-wysocki p-wysocki deleted the enable_python3.10 branch March 31, 2023 12:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

7 participants