Assess the health of external test group #3187

smlambert · 2021-12-15T13:23:32Z

For each of the tests listed in https://github.com/adoptium/aqa-tests/tree/master/external we should assess their current health and status for all versions and implementations.

smlambert · 2022-01-04T21:39:00Z

Based on the work done to date and the notes in the spreadsheet, here is a recent activity update:

Wycheproof - a prereq for this test is bazel which is no longer being correctly added to the auto-generated Dockerfile (dockerfile_functions.sh and likely wycheproof/test.properties needs to be updated)
Scala - a prereq is sbt and it is out of date and no longer correctly being added to the auto-generatated Dockerfile (dockerfile_functions.sh and scala/test.properties needs to be updated
example-test is no longer needed and can be removed (see Remove external/example-test and update relating documentation #3181)
thorntail-mp-tck is no longer needed and can be removed (see Remove external/thorntail-mp-tck #3182)

sophia-guo · 2022-02-16T04:27:47Z

Scala: #3356

LizyBbethy · 2022-02-18T13:46:34Z

currently re assessment of fails with fresher grinder console outputs is on going

For each of the tests listed in https://github.com/adoptium/aqa-tests/tree/master/external we should assess their current health and status for all versions and implementations.

Hotspot spreadsheet

OpenJ9 spreadsheet

sophia-guo · 2022-02-23T22:28:18Z

Using the change https://github.com/adoptium/aqa-tests/compare/master...sophia-guo:health?expand=1 running target

sanity.external: https://ci.adoptopenjdk.net/view/work-in-progress/job/grinder_sandbox_new/436/
extended.external: https://ci.adoptopenjdk.net/view/work-in-progress/job/grinder_sandbox_new/437/

waiting for the result

Update: failed at build stage as sbt version, which is fixed by #3373

sophia-guo · 2022-02-24T17:25:45Z

rerun:
8: sanity.external: https://ci.adoptopenjdk.net/view/work-in-progress/job/grinder_sandbox_new/438/:
11: sanity.external: https://ci.adoptopenjdk.net/view/work-in-progress/job/grinder_sandbox_new/440/console

openliberty_microprofile_tck_jdk8_hs failed with:

openliberty_microprofile_tck_jdk8_hs
:01  BUILD SUCCESSFUL
11:12:01  Total time: 1 minute 37 seconds
11:12:26  Error: No such container:path: openliberty-mp-tck-test:/testResults/surefire-reports
11:12:26  
11:12:26  openliberty_microprofile_tck_jdk8_hs_0_FAILED

issue opened #3377
8: extended.external: https://ci.adoptopenjdk.net/view/work-in-progress/job/grinder_sandbox_new/439/

sophia-guo · 2022-02-25T14:16:30Z

11: sanity.external https://ci.adoptopenjdk.net/view/work-in-progress/job/grinder_sandbox_new/440/console
openliberty_microprofile_tck failed. Issue raised #3380

13:15:04  16 errors
13:15:04  
13:15:04  FAILURE: Build failed with an exception.
13:15:04  
13:15:04  * What went wrong:
13:15:04  Execution failed for task ':com.ibm.ws.componenttest:compileJava'.
13:15:04  > Compilation failed; see the compiler error output for details.
13:15:04  
13:15:04  * Try:
13:15:04  Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

17: sanity.external https://ci.adoptopenjdk.net/view/work-in-progress/job/grinder_sandbox_new/442/console

openliberty_microprofile_tck_0 failed. Issue raised #3381

20:00:41  Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8 -Djava.security.egd=file:/dev/./urandom
20:00:57  java.lang.NoClassDefFoundError: Could not initialize class org.codehaus.groovy.vmplugin.v7.Java7
20:00:57  	at org.codehaus.groovy.vmplugin.VMPluginFactory.<clinit>(VMPluginFactory.java:43)
20:00:57  	at org.codehaus.groovy.reflection.GroovyClassValueFactory.<clinit>(GroovyClassValueFactory.java:35)
20:00:57  	at org.codehaus.groovy.reflection.ClassInfo.<clinit>(ClassInfo.java:107)
20:00:57  	at org.codehaus.groovy.reflection.ReflectionCache.getCachedClass(ReflectionCache.java:95)
20:00:57  	at org.codehaus.groovy.reflection.ReflectionCache.<clinit>(ReflectionCache.java:39)
20:00:57  	at org.codehaus.groovy.runtime.metaclass.MetaClassRegistryImpl.registerMethods(MetaClassRegistryImpl.java:210)
...
20:00:57  FAILURE: Build failed with an exception.
20:00:57  
20:00:57  * What went wrong:
20:00:57  Could not initialize class org.codehaus.groovy.reflection.ReflectionCache
20:00:57  
20:00:57  * Try:
20:00:57  Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
20:00:57  
20:00:57  * Get more help at https://help.gradle.org/
20:00:57  
20:00:57  BUILD FAILED in 14s
20:00:57  
20:00:57  openliberty_microprofile_tck_0_FAILED

quarkus_quickstarts_test_0 failed. Issue raised #3382

20:00:35  /quarkus-quickstarts
20:00:35  Compile and run quarkus_quickstarts tests
20:00:35  Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8 -Djava.security.egd=file:/dev/./urandom
20:00:38  [�[1;31mERROR�[m] Error executing Maven.
20:00:38  [�[1;31mERROR�[m] java.lang.IllegalStateException: Unable to load cache item
20:00:38  [�[1;31mERROR�[m] Caused by: Unable to load cache item
20:00:38  [�[1;31mERROR�[m] Caused by: Could not initialize class com.google.inject.internal.cglib.core.$MethodWrapper
20:00:38  Build quarkus_quickstarts completed

sophia-guo · 2022-02-25T19:08:43Z

Closed by wrong PR. Reopen

sophia-guo · 2022-02-28T18:53:43Z

extended.external, the console output is too big to load. The summary shows most tests failed. Suggest to run each failed application separately in grinder to see the detailed failure message.

jdk8: https://ci.adoptopenjdk.net/view/work-in-progress/job/grinder_sandbox_new/439/
Stoped due to timeout. No summary. Same suggestion, run each application separately in grinder to see the status.

jdk11: https://ci.adoptopenjdk.net/view/work-in-progress/job/grinder_sandbox_new/441/console

19:52:57  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
19:52:57  DISABLED test targets:
19:52:57  	quarkus_native_test_0
19:52:57  
19:52:57  PASSED test targets:
19:52:57  	jacoco_test_0
19:52:57  	quarkus_test_0
19:52:57  	lucene_solr_nightly_smoketest_0
19:52:57  
19:52:57  FAILED test targets:
19:52:57  	tomcat_test_0
19:52:57  	netty_test_0
19:52:57  	wycheproof_test_0
19:52:57  	scala_test_0
19:52:57  	camel_test_0
19:52:57  	kafka_test_0
19:52:57  	derby_test_junit_all_0
19:52:57  	jenkins_test_0
19:52:57  	external_custom_0
19:52:57  	spring_test_0
19:52:57  	wildfly_test_0
19:52:57  
19:52:57  TOTAL: 15   EXECUTED: 14   PASSED: 3   FAILED: 11   DISABLED: 1   SKIPPED: 0

jdk17: https://ci.adoptopenjdk.net/view/work-in-progress/job/grinder_sandbox_new/444/console

19:50:07  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
19:50:07  DISABLED test targets:
19:50:07  	quarkus_native_test_0
19:50:07  
19:50:07  PASSED test targets:
19:50:07  	spring_test_0
19:50:07  
19:50:07  FAILED test targets:
19:50:07  	jacoco_test_0
19:50:07  	tomcat_test_0
19:50:07  	quarkus_test_0
19:50:07  	netty_test_0
19:50:07  	wycheproof_test_0
19:50:07  	elasticsearch_test_hotspot_0
19:50:07  	scala_test_0
19:50:07  	lucene_solr_nightly_smoketest_0
19:50:07  	camel_test_0
19:50:07  	kafka_test_0
19:50:07  	derby_test_junit_all_0
19:50:07  	jenkins_test_0
19:50:07  	wildfly_test_0
19:50:07  
19:50:07  TOTAL: 15   EXECUTED: 14   PASSED: 1   FAILED: 13   DISABLED: 1   SKIPPED: 0
19:50:07  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

sophia-guo · 2022-03-01T17:39:17Z

Tests have different behaviors with different jdk version.
passed with jdk11 and failed with jdk17.
jacoco_test : Grinder: https://ci.adoptopenjdk.net/view/work-in-progress/job/grinder_sandbox_new/449/ issue created #3390
quarkus_test: Grinder https://ci.adoptopenjdk.net/view/work-in-progress/job/grinder_sandbox_new/451 issue created #3392
lucene_solr_nightly_smoketest: Grinder https://ci.adoptopenjdk.net/view/work-in-progress/job/grinder_sandbox_new/450/
issue created #3393

Failed with jdk11 and passed with jdk17:
spring_test: Grinder https://ci.adoptopenjdk.net/view/work-in-progress/job/grinder_sandbox_new/452 issue created #3389

sophia-guo · 2022-03-03T04:59:26Z

sophia-guo · 2022-03-11T22:03:57Z

Update health of sanity
sanity.extennal sheet

sophia-guo · 2022-04-07T15:46:11Z

extended.external sheet

sophia-guo · 2022-04-13T17:06:04Z

extended.external sheet updated. Close this one.

smlambert added the epic label Dec 15, 2021

smlambert added the type:external 3rd party application tests (invoked by the make target 'external') label Feb 5, 2022

smlambert mentioned this issue Feb 5, 2022

Some external projects (wycheproof, scala) Dockefile can't be generated correctly. #3320

Closed

This was referenced Feb 25, 2022

Target sanity.external(extended.external) with hotspot and no parallel build get the build error with external_custom project #3319

Closed

Skip compile external_custom project if EXTERNAL_CUSTOM_REPO is not set #3383

Merged

smlambert closed this as completed in #3383 Feb 25, 2022

sophia-guo reopened this Feb 25, 2022

This was referenced Feb 26, 2022

Lucene junit reports do not include failed tests #467

Closed

type:external Reported Upstream issues #587

Closed

This was referenced Feb 28, 2022

sanity.external_s390x_linux_openliberty-mp-tck tests fail to compile #2191

Closed

jdk14 x86-64_linux_openliberty-mp-tck build failure #1858

Closed

jdk8, jdk11 x86-64_linux_openliberty-mp-tck build failure #1857

Closed

sophia-guo mentioned this issue Apr 8, 2022

Remove or replace unnecessary/deprecated external tests #3188

Closed

sophia-guo closed this as completed Apr 13, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Assess the health of external test group #3187

Assess the health of external test group #3187

smlambert commented Dec 15, 2021 •

edited

smlambert commented Jan 4, 2022

sophia-guo commented Feb 16, 2022

LizyBbethy commented Feb 18, 2022

sophia-guo commented Feb 23, 2022 •

edited

sophia-guo commented Feb 24, 2022 •

edited

sophia-guo commented Feb 25, 2022

sophia-guo commented Feb 25, 2022

sophia-guo commented Feb 28, 2022

sophia-guo commented Mar 1, 2022 •

edited

sophia-guo commented Mar 3, 2022 •

edited

sophia-guo commented Mar 11, 2022 •

edited

sophia-guo commented Apr 7, 2022

sophia-guo commented Apr 13, 2022

Assess the health of external test group #3187

Assess the health of external test group #3187

Comments

smlambert commented Dec 15, 2021 • edited

smlambert commented Jan 4, 2022

sophia-guo commented Feb 16, 2022

LizyBbethy commented Feb 18, 2022

sophia-guo commented Feb 23, 2022 • edited

sophia-guo commented Feb 24, 2022 • edited

sophia-guo commented Feb 25, 2022

sophia-guo commented Feb 25, 2022

sophia-guo commented Feb 28, 2022

sophia-guo commented Mar 1, 2022 • edited

sophia-guo commented Mar 3, 2022 • edited

sophia-guo commented Mar 11, 2022 • edited

sophia-guo commented Apr 7, 2022

sophia-guo commented Apr 13, 2022

smlambert commented Dec 15, 2021 •

edited

sophia-guo commented Feb 23, 2022 •

edited

sophia-guo commented Feb 24, 2022 •

edited

sophia-guo commented Mar 1, 2022 •

edited

sophia-guo commented Mar 3, 2022 •

edited

sophia-guo commented Mar 11, 2022 •

edited