Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

com.hazelcast.jet.python.PythonServiceTest [HZ-1776] [HZ-4730] #22885

Open
sumnerib opened this issue Nov 18, 2022 · 8 comments
Open

com.hazelcast.jet.python.PythonServiceTest [HZ-1776] [HZ-4730] #22885

sumnerib opened this issue Nov 18, 2022 · 8 comments
Assignees
@Patras3
Labels
Module: Jet Issues/PRs for Jet Source: Internal PR or issue was opened by an employee Team: Integration to-jira Type: Test-Failure
Milestone
Backlog

Comments

@sumnerib
Copy link
Contributor

sumnerib commented Nov 18, 2022 •
edited

master (commit 1cb1e05)

Failed on IbmJDK8-fips: https://jenkins.hazelcast.com/job/Hazelcast-master-IbmJDK8-fips-nightly/482/testReport/junit/com.hazelcast.jet.python/PythonServiceTest

Also fails for: https://jenkins.hazelcast.com/job/Hazelcast-master-IbmJDK8-fips-nightly/482/testReport/junit/com.hazelcast.jet.python/PythonInitCleanupTest/

Stacktrace: 
java.util.concurrent.CompletionException: com.hazelcast.jet.JetException: PythonService initialization failed: java.lang.Exception: Initialization script finished with non-zero exit code: 1
	at java.util.concurrent.CompletableFuture.reportJoin(CompletableFuture.java:386)
	at java.util.concurrent.CompletableFuture.join(CompletableFuture.java:1958)
	at com.hazelcast.jet.Job.join(Job.java:78)
	at com.hazelcast.jet.python.PythonServiceTest.batchStage_mapUsingPython_onAllMembers(PythonServiceTest.java:126)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
	at java.lang.reflect.Method.invoke(Method.java:508)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at com.hazelcast.test.FailOnTimeoutStatement$CallableStatement.call(FailOnTimeoutStatement.java:115)
	at com.hazelcast.test.FailOnTimeoutStatement$CallableStatement.call(FailOnTimeoutStatement.java:107)
	at java.util.concurrent.FutureTask.run(FutureTask.java:277)
	at java.lang.Thread.run(Thread.java:825)
Caused by: com.hazelcast.jet.JetException: PythonService initialization failed: java.lang.Exception: Initialization script finished with non-zero exit code: 1
	at com.hazelcast.jet.python.PythonServiceContext.<init>(PythonServiceContext.java:114)
	at com.hazelcast.jet.python.PythonService.createContextWithRetry(PythonService.java:109)
	at com.hazelcast.jet.python.PythonService.lambda$factory$9a0f8f23$1(PythonService.java:89)
	at com.hazelcast.function.FunctionEx.apply(FunctionEx.java:49)
	at com.hazelcast.jet.impl.processor.ProcessorSupplierWithService.init(ProcessorSupplierWithService.java:59)
	at com.hazelcast.jet.impl.execution.init.ExecutionPlan.lambda$null$9c5eb802$1(ExecutionPlan.java:456)
	at com.hazelcast.jet.function.RunnableEx.run(RunnableEx.java:31)
	at com.hazelcast.jet.impl.util.Util.doWithClassLoader(Util.java:532)
	at com.hazelcast.jet.impl.execution.init.ExecutionPlan.lambda$initProcSuppliers$10beb6ff$1(ExecutionPlan.java:455)
	at com.hazelcast.jet.function.RunnableEx.run(RunnableEx.java:31)
	at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1651)
	at com.hazelcast.internal.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:217)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.lang.Thread.run(Thread.java:825)
	at com.hazelcast.internal.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76)
	at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102)
Caused by: java.lang.Exception: Initialization script finished with non-zero exit code: 1
	at com.hazelcast.jet.python.PythonServiceContext.<init>(PythonServiceContext.java:105)
	... 16 more

Standard output can be found here - https://s3.console.aws.amazon.com/s3/buckets/j-artifacts/Hazelcast-master-IbmJDK8-fips-nightly/482/
@sumnerib sumnerib added Type: Test-Failure Source: Internal PR or issue was opened by an employee Module: Jet Issues/PRs for Jet Team: Integration labels Nov 18, 2022
@sumnerib sumnerib added this to the 5.3 Backlog milestone Nov 18, 2022
@sumnerib sumnerib changed the title com.hazelcast.jet.python.PythonServiceTest.batchStage_mapUsingPython_onAllMembers com.hazelcast.jet.python.PythonServiceTest Nov 18, 2022
@kwart
Copy link
Member

kwart commented Nov 22, 2022

Hopefully will be fixed by Python upgrade (3.6->3.8) in https://github.com/hazelcast/DevOps/commit/ee62c5068d5071497a19f3b46047a86218b3afc2

Verification run: https://jenkins.hazelcast.com/job/Hazelcast-master-IbmJDK8-fips-nightly/484/

@kwart kwart added the to-jira label Nov 22, 2022
@github-actions github-actions bot changed the title com.hazelcast.jet.python.PythonServiceTest com.hazelcast.jet.python.PythonServiceTest [HZ-1776] Nov 22, 2022
@github-actions
Copy link
Contributor

Internal Jira issue: HZ-1776

@AyberkSorgun AyberkSorgun modified the milestones: 5.3 Backlog, 5.3.0 Nov 22, 2022
@kwart
Copy link
Member

kwart commented Nov 23, 2022

The PythonServiceTest in the verification run passed. Closing.

@kwart kwart closed this as completed Nov 23, 2022
@sumnerib
Copy link
Contributor Author

Reopening as issue occurred again on: master (commit f853b8a782d995a038f242914f5632eac018545c)
with CorrettoJDK17: https://jenkins.hazelcast.com/job/Hazelcast-master-CorrettoJDK17-arm64-nightly/12/testReport/junit/com.hazelcast.jet.python/PythonServiceTest/

Stacktrace: 
java.util.concurrent.CompletionException: com.hazelcast.jet.JetException: PythonService initialization failed: java.lang.Exception: Initialization script finished with non-zero exit code: 1
	at java.base/java.util.concurrent.CompletableFuture.reportJoin(CompletableFuture.java:413)
	at java.base/java.util.concurrent.CompletableFuture.join(CompletableFuture.java:2118)
	at com.hazelcast.jet.Job.join(Job.java:80)
	at com.hazelcast.jet.python.PythonServiceTest.streamStage_mapUsingPython_withCleanupScript(PythonServiceTest.java:250)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at com.hazelcast.test.FailOnTimeoutStatement$CallableStatement.call(FailOnTimeoutStatement.java:115)
	at com.hazelcast.test.FailOnTimeoutStatement$CallableStatement.call(FailOnTimeoutStatement.java:107)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: com.hazelcast.jet.JetException: PythonService initialization failed: java.lang.Exception: Initialization script finished with non-zero exit code: 1
	at com.hazelcast.jet.python.PythonServiceContext.<init>(PythonServiceContext.java:113)
	at com.hazelcast.jet.python.PythonService.createContextWithRetry(PythonService.java:109)
	at com.hazelcast.jet.python.PythonService.lambda$factory$9a0f8f23$1(PythonService.java:89)
	at com.hazelcast.function.FunctionEx.apply(FunctionEx.java:47)
	at com.hazelcast.jet.impl.processor.ProcessorSupplierWithService.init(ProcessorSupplierWithService.java:59)
	at com.hazelcast.jet.impl.execution.init.ExecutionPlan.lambda$initProcSuppliers$3d2e4f48$1(ExecutionPlan.java:460)
	at com.hazelcast.jet.function.RunnableEx.run(RunnableEx.java:31)
	at com.hazelcast.jet.impl.util.Util.doWithClassLoader(Util.java:581)
	at com.hazelcast.jet.impl.execution.init.ExecutionPlan.lambda$initProcSuppliers$2325813f$1(ExecutionPlan.java:459)
	at com.hazelcast.jet.function.RunnableEx.run(RunnableEx.java:31)
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)
	at com.hazelcast.internal.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:220)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)
	at com.hazelcast.internal.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76)
	at com.hazelcast.internal.util.executor.PoolExecutorThreadFactory$ManagedThread.executeRun(PoolExecutorThreadFactory.java:74)
	at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:111)
Caused by: java.lang.Exception: Initialization script finished with non-zero exit code: 1
	at com.hazelcast.jet.python.PythonServiceContext.<init>(PythonServiceContext.java:104)
	... 17 more

Standard output can be found here - https://s3.console.aws.amazon.com/s3/buckets/j-artifacts/Hazelcast-master-CorrettoJDK17-arm64-nightly/12/

@sumnerib sumnerib reopened this Apr 18, 2024
@sumnerib sumnerib modified the milestones: 5.3.0, 5.5.0 Apr 18, 2024
@Patras3
Copy link
Contributor

Patras3 commented Apr 26, 2024 •
edited

Failed again on master with:

@Patras3 Patras3 modified the milestones: 5.5.0, Backlog Apr 26, 2024
@Patras3
Copy link
Contributor

Patras3 commented May 9, 2024 •
edited

The same issue was found in soaks tests for devel.
⬇️ Created a new issue in Jira (previous was closed, and the reason is probably different).

@Patras3 Patras3 added to-jira and removed to-jira labels May 9, 2024
@github-actions github-actions bot changed the title com.hazelcast.jet.python.PythonServiceTest [HZ-1776] com.hazelcast.jet.python.PythonServiceTest [HZ-1776] [HZ-4730] May 9, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented May 9, 2024

Internal Jira issue: HZ-4730

@kwart kwart assigned Patras3 and unassigned kwart and Patras3 May 13, 2024
@kwart
Copy link
Member

kwart commented May 13, 2024

Reassigned to Patryk, who manages the ARM64 environment for the Corretto17 tests.
The issue is caused by the old Python version (3.7.16) in the AMI (arm64-mvn-corretto17-cmake). Our Python Protobuf dependency requires at least Python 3.8.

(Currently, we have the Python version 3.10.12 in the JDK compatibility Docker image).

Test output contains:

02:11:01,550 DEBUG || - [python] python-init-logger_29449 - [127.0.0.1]:5701 [dev] [5.5.0-SNAPSHOT] + flock --exclusive /home/ec2-user/.jet-pip.lock python3 -m pip install protobuf==4.25.3 grpcio==1.60.0
02:11:02,198 DEBUG || - [python] python-init-logger_29449 - [127.0.0.1]:5701 [dev] [5.5.0-SNAPSHOT] ERROR: Ignored the following yanked versions: 3.18.0, 4.0.0rc1, 4.0.0rc2, 4.21.0
02:11:02,198 DEBUG || - [python] python-init-logger_29449 - [127.0.0.1]:5701 [dev] [5.5.0-SNAPSHOT] ERROR: Ignored the following versions that require a different python version: 4.25.0 Requires-Python >=3.8; 4.25.0rc1 Requires-Python >=3.8; 4.25.0rc2 Requires-Python >=3.8; 4.25.1 Requires-Python >=3.8; 4.25.2 Requires-Python >=3.8; 4.25.3 Requires-Python >=3.8; 5.26.0 Requires-Python >=3.8; 5.26.0rc1 Requires-Python >=3.8; 5.26.0rc2 Requires-Python >=3.8; 5.26.0rc3 Requires-Python >=3.8; 5.26.1 Requires-Python >=3.8

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Patras3 Patras3

Labels
Module: Jet Issues/PRs for Jet Source: Internal PR or issue was opened by an employee Team: Integration to-jira Type: Test-Failure
Projects
None yet
Milestone
Backlog
Development

No branches or pull requests
6 participants
@kwart @olukas @frant-hartm @AyberkSorgun @sumnerib @Patras3