Upgrade Python version to 3.10 #2057

tenzen-y · 2022-12-11T02:09:46Z

Signed-off-by: tenzen-y yuki.iwai.tz@gmail.com

What this PR does / why we need it:
I upgraded Python version to 3.10 since Python 3.9 no longer fixes bugs, according to this.

3.9 | PEP 596 | security | 2020-10-05 | 2025-10 | Łukasz Langa

Also, I upgraded pytest version to avoid the error, E TypeError: required field "lineno" missing from alias.

ref: pytest-dev/pytest#8540

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

Checklist:

Docs included if any changes are user facing

tenzen-y · 2022-12-11T13:38:32Z

blocked by #2058
/hold

tenzen-y · 2022-12-11T15:21:14Z

cmd/metricscollector/v1beta1/tfevent-metricscollector/requirements.txt

@@ -1,4 +1,4 @@
-psutil==5.8.0
+psutil==5.9.4


To resolve the following error, we must upgrade the psutil version.

... #0 98.35 C compiler or Python headers are not installed on this system. Try to run: #0 98.35 sudo apt-get install gcc python3-dev #0 98.35 error: command 'gcc' failed: No such file or directory #0 98.35 [end of output] #0 98.35 #0 98.35 note: This error originates from a subprocess, and is likely not a problem with pip. #0 98.36 error: legacy-install-failure #0 98.36 #0 98.36 × Encountered error while trying to install package. #0 98.36 ╰─> psutil ...

ref: giampaolo/psutil#2165

tenzen-y · 2022-12-11T15:24:09Z

test/unit/v1beta1/suggestion/test_chocolate_service.py

+if sys.version_info <= (3, 9):
+    from pkg.suggestion.v1beta1.chocolate.service import ChocolateService

 import utils


+@unittest.skipIf(sys.version_info >= (3, 10), reason="Chocolate supports only Python 3.9 or lower")


We can not work chocolate on Python 3.10.

ref: #2058

tenzen-y · 2022-12-11T16:50:23Z

cmd/suggestion/chocolate/v1beta1/Dockerfile

-    if [ "${TARGETARCH}" = "ppc64le" ] || [ "${TARGETARCH}" = "arm64" ]; then \
    apt-get -y install gfortran libopenblas-dev liblapack-dev g++; \
-    fi && \


To avoid the following error, we must install these packages for all platforms.

... #0 73.28 gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/usr/local/include/python3.9 -c src/Halton.cpp -o build/temp.linux-x86_64-3.9/src/Halton.o #0 73.28 error: command 'gcc' failed: No such file or directory #0 73.28 [end of output] #0 73.28 #0 73.28 note: This error originates from a subprocess, and is likely not a problem with pip. #0 73.29 error: legacy-install-failure #0 73.29 #0 73.29 × Encountered error while trying to install package. #0 73.29 ╰─> ghalton ...

tenzen-y · 2022-12-12T01:49:30Z

test/unit/v1beta1/suggestion/test_chocolate_service.py

+if sys.version_info < (3, 10):
+    from pkg.suggestion.v1beta1.chocolate.service import ChocolateService

 import utils


+@unittest.skipIf(sys.version_info >= (3, 10), reason="Chocolate supports only Python 3.9 or lower")


We can not work chocolate on Python 3.10.

ref: #2058

Are we removing chocolate in this release itself? Else, removing tests will lead to untested releases

@johnugeorge
IIUC, we can run tests for chocolate in CI since I added Python 3.9 to CI so that we can verify the chocolate suggestion service as shown below:

katib/.github/workflows/test-python.yaml

Lines 29 to 33 in ff6f913

# TODO (tenzen-y): Remove tests for Python 3.9 once we remove the Chocolate Suggestion Service from Katib.

strategy:

fail-fast: false

matrix:

python-version: ['3.9', '3.10']

Although, I'm ok with removing chocolate in this release.
wdyt?

I think removing chocolate is better, since there is no much reason to support it as you mentioned at #2058

I agree, we should remove Chocolate first.
We want to reduce number of actions that we run since we have 20 concurrent GitHub Actions jobs limitation.

Ok.
First, I will remove Chocolate Suggestion Service in another PR.

anencore94

/lgtm
looks good to me!

johnugeorge · 2022-12-12T10:28:20Z

LGTM

/assign @andreyvelich

andreyvelich · 2022-12-12T18:01:56Z

Thank you for updating the version @tenzen-y!
@tenzen-y @johnugeorge @anencore94 What should we do with our Katib SDK Python version ?
Should we update this version carefully with proper user awareness in advance?

tenzen-y · 2022-12-13T05:18:43Z

@andreyvelich

@tenzen-y @johnugeorge @anencore94 What should we do with our Katib SDK Python version ?
Should we update this version carefully with proper user awareness in advance?

It might be better to use our Python SDK in our E2E, as you mentioned in #2024, and use multiple Python versions.
For example, run tests for the E2E Test with darts-cnn-cifar10 with Python 3.7, and run tests for the E2E Test with mxnet-mnist with Python 3.8.

tenzen-y · 2022-12-14T15:33:35Z

blocked by #2064

anencore94 · 2022-12-15T01:53:12Z

@andreyvelich

@tenzen-y @johnugeorge @anencore94 What should we do with our Katib SDK Python version ?
Should we update this version carefully with proper user awareness in advance?

It might be better to use our Python SDK in our E2E, as you mentioned in #2024, and use multiple Python versions. For example, run tests for the E2E Test with darts-cnn-cifar10 with Python 3.7, and run tests for the E2E Test with mxnet-mnist with Python 3.8.

I agree with you too @tenzen-y . SDK should care about backward compatibility of python version. And notice to users at release note if supported python version will be changed.
Thus we need to test sdk with multiple version of python

Signed-off-by: tenzen-y <yuki.iwai.tz@gmail.com> Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com>

tenzen-y · 2023-01-06T07:09:41Z

@andreyvelich @johnugeorge @anencore94 I have rebased this. PTAL.
/hold cancel

google-oss-prow · 2023-01-06T08:56:55Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: johnugeorge, tenzen-y

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [johnugeorge,tenzen-y]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

johnugeorge · 2023-01-06T09:02:47Z

/lgtm

google-oss-prow bot added do-not-merge/work-in-progress size/M approved labels Dec 11, 2022

google-oss-prow bot requested review from andreyvelich, anencore94 and johnugeorge December 11, 2022 02:10

google-oss-prow bot added the do-not-merge/hold label Dec 11, 2022

tenzen-y changed the title ~~[WIP] Upgrade python version to v3.10~~ [WIP] Upgrade Python version to 3.10 Dec 11, 2022

tenzen-y force-pushed the upgrade-python-version-to-3.10 branch from 801e8f9 to 58d63a8 Compare December 11, 2022 15:19

tenzen-y commented Dec 11, 2022

View reviewed changes

tenzen-y force-pushed the upgrade-python-version-to-3.10 branch 4 times, most recently from cff7b18 to ff6f913 Compare December 11, 2022 16:48

tenzen-y commented Dec 11, 2022

View reviewed changes

tenzen-y changed the title ~~[WIP] Upgrade Python version to 3.10~~ Upgrade Python version to 3.10 Dec 11, 2022

google-oss-prow bot removed the do-not-merge/work-in-progress label Dec 11, 2022

tenzen-y commented Dec 12, 2022

View reviewed changes

anencore94 reviewed Dec 12, 2022

View reviewed changes

google-oss-prow bot assigned anencore94 Dec 12, 2022

google-oss-prow bot added the lgtm label Dec 12, 2022

google-oss-prow bot assigned andreyvelich Dec 12, 2022

tenzen-y force-pushed the upgrade-python-version-to-3.10 branch from ff6f913 to 473c633 Compare January 6, 2023 07:07

google-oss-prow bot removed the lgtm label Jan 6, 2023

Upgrade Python version to 3.10

83afcfc

Signed-off-by: tenzen-y <yuki.iwai.tz@gmail.com> Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com>

tenzen-y force-pushed the upgrade-python-version-to-3.10 branch from 473c633 to 83afcfc Compare January 6, 2023 07:08

google-oss-prow bot removed the do-not-merge/hold label Jan 6, 2023

johnugeorge approved these changes Jan 6, 2023

View reviewed changes

google-oss-prow bot assigned johnugeorge Jan 6, 2023

google-oss-prow bot added the lgtm label Jan 6, 2023

google-oss-prow bot merged commit 45a4744 into kubeflow:master Jan 6, 2023

tenzen-y deleted the upgrade-python-version-to-3.10 branch January 6, 2023 09:06

tenzen-y mentioned this pull request Jan 6, 2023

[SDK] Use Katib SDK for E2E Tests #2075

Merged

tenzen-y mentioned this pull request Jan 20, 2023

Run e2e with various Python versions to verify Python SDK #2092

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade Python version to 3.10 #2057

Upgrade Python version to 3.10 #2057

tenzen-y commented Dec 11, 2022 •

edited

tenzen-y commented Dec 11, 2022

tenzen-y Dec 11, 2022 •

edited

tenzen-y Dec 11, 2022

tenzen-y Dec 11, 2022

tenzen-y Dec 12, 2022

johnugeorge Dec 12, 2022

tenzen-y Dec 12, 2022 •

edited

anencore94 Dec 12, 2022

andreyvelich Dec 12, 2022

tenzen-y Dec 13, 2022

anencore94 left a comment

johnugeorge commented Dec 12, 2022

andreyvelich commented Dec 12, 2022

tenzen-y commented Dec 13, 2022

tenzen-y commented Dec 14, 2022

anencore94 commented Dec 15, 2022

tenzen-y commented Jan 6, 2023

google-oss-prow bot commented Jan 6, 2023

johnugeorge commented Jan 6, 2023

	# TODO (tenzen-y): Remove tests for Python 3.9 once we remove the Chocolate Suggestion Service from Katib.
	strategy:
	fail-fast: false
	matrix:
	python-version: ['3.9', '3.10']

		@@ -1,4 +1,4 @@
		psutil==5.8.0
		psutil==5.9.4

Upgrade Python version to 3.10 #2057

Upgrade Python version to 3.10 #2057

Conversation

tenzen-y commented Dec 11, 2022 • edited

tenzen-y commented Dec 11, 2022

tenzen-y Dec 11, 2022 • edited

Choose a reason for hiding this comment

tenzen-y Dec 11, 2022

Choose a reason for hiding this comment

tenzen-y Dec 11, 2022

Choose a reason for hiding this comment

tenzen-y Dec 12, 2022

Choose a reason for hiding this comment

johnugeorge Dec 12, 2022

Choose a reason for hiding this comment

tenzen-y Dec 12, 2022 • edited

Choose a reason for hiding this comment

anencore94 Dec 12, 2022

Choose a reason for hiding this comment

andreyvelich Dec 12, 2022

Choose a reason for hiding this comment

tenzen-y Dec 13, 2022

Choose a reason for hiding this comment

anencore94 left a comment

Choose a reason for hiding this comment

johnugeorge commented Dec 12, 2022

andreyvelich commented Dec 12, 2022

tenzen-y commented Dec 13, 2022

tenzen-y commented Dec 14, 2022

anencore94 commented Dec 15, 2022

tenzen-y commented Jan 6, 2023

google-oss-prow bot commented Jan 6, 2023

johnugeorge commented Jan 6, 2023

tenzen-y commented Dec 11, 2022 •

edited

tenzen-y Dec 11, 2022 •

edited

tenzen-y Dec 12, 2022 •

edited