[SPARK-41522][BUILD] Pin `versions-maven-plugin` to 2.13.0 to recover `test-dependencies.sh` #39067

LuciferYang · 2022-12-15T05:22:12Z

What changes were proposed in this pull request?

This pr aims to pin versions-maven-plugin to 2.13.0 to recover test-dependencies.sh and make GA Dependencies test task pass , this pr should revert after we know how to use version 2.14.0.

Why are the changes needed?

dev/test-dependencies.sh always use latest versions-maven-plugin version, and versions-maven-plugin 2.14.0 has not set the version of the sub-module.

Run:

build/mvn -q versions:set -DnewVersion=spark-928034 -DgenerateBackupPoms=false

2.14.0

+ git status
On branch test-ci
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
  modified:   assembly/pom.xml
  modified:   core/pom.xml
  modified:   examples/pom.xml
  modified:   graphx/pom.xml
  modified:   hadoop-cloud/pom.xml
  modified:   launcher/pom.xml
  modified:   mllib-local/pom.xml
  modified:   mllib/pom.xml
  modified:   pom.xml
  modified:   repl/pom.xml
  modified:   streaming/pom.xml
  modified:   tools/pom.xml

2.13.0

+ git status
On branch test-ci
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
  modified:   assembly/pom.xml
  modified:   common/kvstore/pom.xml
  modified:   common/network-common/pom.xml
  modified:   common/network-shuffle/pom.xml
  modified:   common/network-yarn/pom.xml
  modified:   common/sketch/pom.xml
  modified:   common/tags/pom.xml
  modified:   common/unsafe/pom.xml
  modified:   connector/avro/pom.xml
  modified:   connector/connect/common/pom.xml
  modified:   connector/connect/server/pom.xml
  modified:   connector/docker-integration-tests/pom.xml
  modified:   connector/kafka-0-10-assembly/pom.xml
  modified:   connector/kafka-0-10-sql/pom.xml
  modified:   connector/kafka-0-10-token-provider/pom.xml
  modified:   connector/kafka-0-10/pom.xml
  modified:   connector/kinesis-asl-assembly/pom.xml
  modified:   connector/kinesis-asl/pom.xml
  modified:   connector/protobuf/pom.xml
  modified:   connector/spark-ganglia-lgpl/pom.xml
  modified:   core/pom.xml
  modified:   dev/test-dependencies.sh
  modified:   examples/pom.xml
  modified:   graphx/pom.xml
  modified:   hadoop-cloud/pom.xml
  modified:   launcher/pom.xml
  modified:   mllib-local/pom.xml
  modified:   mllib/pom.xml
  modified:   pom.xml
  modified:   repl/pom.xml
  modified:   resource-managers/kubernetes/core/pom.xml
  modified:   resource-managers/kubernetes/integration-tests/pom.xml
  modified:   resource-managers/mesos/pom.xml
  modified:   resource-managers/yarn/pom.xml
  modified:   sql/catalyst/pom.xml
  modified:   sql/core/pom.xml
  modified:   sql/hive-thriftserver/pom.xml
  modified:   sql/hive/pom.xml
  modified:   streaming/pom.xml
  modified:   tools/pom.xml

Therefore, the following compilation error will occur when using 2.14.0.

2022-12-15T02:37:35.5536924Z [ERROR] [ERROR] Some problems were encountered while processing the POMs:
2022-12-15T02:37:35.5538469Z [FATAL] Non-resolvable parent POM for org.apache.spark:spark-sketch_2.12:3.4.0-SNAPSHOT: Could not find artifact org.apache.spark:spark-parent_2.12:pom:3.4.0-SNAPSHOT and 'parent.relativePath' points at wrong local POM @ line 22, column 11

Does this PR introduce any user-facing change?

No

How was this patch tested?

Pass GitHub Actions

dongjoon-hyun · 2022-12-15T05:40:36Z

Thank you for investigating this, @LuciferYang .

LuciferYang · 2022-12-15T06:03:42Z

Different from the local error, further investigation is required

LuciferYang · 2022-12-15T08:18:44Z

@dongjoon-hyun @HyukjinKwon This pr provides a workaround solution. We can restore GA first. I will give another pr when I know how to use the latest versions-maven-plugin version

dongjoon-hyun · 2022-12-15T08:24:16Z

dev/test-dependencies.sh

@@ -76,11 +76,13 @@ function reset_version {
  find "$HOME/.m2/" | grep "$TEMP_VERSION" | xargs rm -rf

  # Restore the original version number:
-  $MVN -q versions:set -DnewVersion=$OLD_VERSION -DgenerateBackupPoms=false > /dev/null
+  # SPARK-41522: ping versions-maven-plugin version for GA pass


ping -> pin?

dongjoon-hyun · 2022-12-15T08:24:33Z

dev/test-dependencies.sh

 }
 trap reset_version EXIT

-$MVN -q versions:set -DnewVersion=$TEMP_VERSION -DgenerateBackupPoms=false > /dev/null
+# SPARK-41522: ping versions-maven-plugin version for GA pass


ping -> pin?

LuciferYang · 2022-12-15T08:25:16Z

A simple GA Task to verify dev/test-dependencies.sh has been executed successfully

https://github.com/LuciferYang/make-distribution.sh/blob/master/.github/workflows/blank.yml

https://github.com/LuciferYang/make-distribution.sh/actions/runs/3702150582/jobs/6272109950

dongjoon-hyun · 2022-12-15T08:25:37Z

I agree with this workaround, @LuciferYang . I left two typo comments.

LuciferYang · 2022-12-15T08:27:31Z

98d2d21 fix typos

dongjoon-hyun · 2022-12-15T08:29:30Z

BTW, this PR recovers local test failures too because dev/test-dependencies.sh fails locally in the same way.
Could you revise the code, PR title, and description? Something like to recover test-dependencies.sh?

dongjoon-hyun

+1, LGTM. Merged to master/3.3/3.2 to recover CI.
Thank you so much, @LuciferYang .

… `test-dependencies.sh` ### What changes were proposed in this pull request? This pr aims to pin `versions-maven-plugin` to 2.13.0 to recover `test-dependencies.sh` and make GA pass , this pr should revert after we know how to use version 2.14.0. ### Why are the changes needed? `dev/test-dependencies.sh` always use latest `versions-maven-plugin` version, and `versions-maven-plugin` 2.14.0 has not set the version of the sub-module. Run: ``` build/mvn -q versions:set -DnewVersion=spark-928034 -DgenerateBackupPoms=false ``` **2.14.0** ``` + git status On branch test-ci Changes not staged for commit: (use "git add <file>..." to update what will be committed) (use "git restore <file>..." to discard changes in working directory) modified: assembly/pom.xml modified: core/pom.xml modified: examples/pom.xml modified: graphx/pom.xml modified: hadoop-cloud/pom.xml modified: launcher/pom.xml modified: mllib-local/pom.xml modified: mllib/pom.xml modified: pom.xml modified: repl/pom.xml modified: streaming/pom.xml modified: tools/pom.xml ``` **2.13.0** ``` + git status On branch test-ci Changes not staged for commit: (use "git add <file>..." to update what will be committed) (use "git restore <file>..." to discard changes in working directory) modified: assembly/pom.xml modified: common/kvstore/pom.xml modified: common/network-common/pom.xml modified: common/network-shuffle/pom.xml modified: common/network-yarn/pom.xml modified: common/sketch/pom.xml modified: common/tags/pom.xml modified: common/unsafe/pom.xml modified: connector/avro/pom.xml modified: connector/connect/common/pom.xml modified: connector/connect/server/pom.xml modified: connector/docker-integration-tests/pom.xml modified: connector/kafka-0-10-assembly/pom.xml modified: connector/kafka-0-10-sql/pom.xml modified: connector/kafka-0-10-token-provider/pom.xml modified: connector/kafka-0-10/pom.xml modified: connector/kinesis-asl-assembly/pom.xml modified: connector/kinesis-asl/pom.xml modified: connector/protobuf/pom.xml modified: connector/spark-ganglia-lgpl/pom.xml modified: core/pom.xml modified: dev/test-dependencies.sh modified: examples/pom.xml modified: graphx/pom.xml modified: hadoop-cloud/pom.xml modified: launcher/pom.xml modified: mllib-local/pom.xml modified: mllib/pom.xml modified: pom.xml modified: repl/pom.xml modified: resource-managers/kubernetes/core/pom.xml modified: resource-managers/kubernetes/integration-tests/pom.xml modified: resource-managers/mesos/pom.xml modified: resource-managers/yarn/pom.xml modified: sql/catalyst/pom.xml modified: sql/core/pom.xml modified: sql/hive-thriftserver/pom.xml modified: sql/hive/pom.xml modified: streaming/pom.xml modified: tools/pom.xml ``` Therefore, the following compilation error will occur when using 2.14.0. ``` 2022-12-15T02:37:35.5536924Z [ERROR] [ERROR] Some problems were encountered while processing the POMs: 2022-12-15T02:37:35.5538469Z [FATAL] Non-resolvable parent POM for org.apache.spark:spark-sketch_2.12:3.4.0-SNAPSHOT: Could not find artifact org.apache.spark:spark-parent_2.12:pom:3.4.0-SNAPSHOT and 'parent.relativePath' points at wrong local POM line 22, column 11 ``` ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Pass GitHub Actions Closes #39067 from LuciferYang/test-ci. Authored-by: yangjie01 <yangjie01@baidu.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit bfe1af9) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

LuciferYang · 2022-12-15T08:43:06Z

Thanks @dongjoon-hyun

HyukjinKwon

LGTM2

LuciferYang · 2022-12-15T12:35:10Z

Further feedback, I found that this is a bug of versions-maven-plugin 2.14.0, nested modules cannot be updated version,

and an issue is tracking this：

Regression in 2.14.0 mojohaus/versions#848

we need wait 2.14.1 for fix, @dongjoon-hyun @HyukjinKwon

… `test-dependencies.sh` ### What changes were proposed in this pull request? This pr aims to pin `versions-maven-plugin` to 2.13.0 to recover `test-dependencies.sh` and make GA pass , this pr should revert after we know how to use version 2.14.0. ### Why are the changes needed? `dev/test-dependencies.sh` always use latest `versions-maven-plugin` version, and `versions-maven-plugin` 2.14.0 has not set the version of the sub-module. Run: ``` build/mvn -q versions:set -DnewVersion=spark-928034 -DgenerateBackupPoms=false ``` **2.14.0** ``` + git status On branch test-ci Changes not staged for commit: (use "git add <file>..." to update what will be committed) (use "git restore <file>..." to discard changes in working directory) modified: assembly/pom.xml modified: core/pom.xml modified: examples/pom.xml modified: graphx/pom.xml modified: hadoop-cloud/pom.xml modified: launcher/pom.xml modified: mllib-local/pom.xml modified: mllib/pom.xml modified: pom.xml modified: repl/pom.xml modified: streaming/pom.xml modified: tools/pom.xml ``` **2.13.0** ``` + git status On branch test-ci Changes not staged for commit: (use "git add <file>..." to update what will be committed) (use "git restore <file>..." to discard changes in working directory) modified: assembly/pom.xml modified: common/kvstore/pom.xml modified: common/network-common/pom.xml modified: common/network-shuffle/pom.xml modified: common/network-yarn/pom.xml modified: common/sketch/pom.xml modified: common/tags/pom.xml modified: common/unsafe/pom.xml modified: connector/avro/pom.xml modified: connector/connect/common/pom.xml modified: connector/connect/server/pom.xml modified: connector/docker-integration-tests/pom.xml modified: connector/kafka-0-10-assembly/pom.xml modified: connector/kafka-0-10-sql/pom.xml modified: connector/kafka-0-10-token-provider/pom.xml modified: connector/kafka-0-10/pom.xml modified: connector/kinesis-asl-assembly/pom.xml modified: connector/kinesis-asl/pom.xml modified: connector/protobuf/pom.xml modified: connector/spark-ganglia-lgpl/pom.xml modified: core/pom.xml modified: dev/test-dependencies.sh modified: examples/pom.xml modified: graphx/pom.xml modified: hadoop-cloud/pom.xml modified: launcher/pom.xml modified: mllib-local/pom.xml modified: mllib/pom.xml modified: pom.xml modified: repl/pom.xml modified: resource-managers/kubernetes/core/pom.xml modified: resource-managers/kubernetes/integration-tests/pom.xml modified: resource-managers/mesos/pom.xml modified: resource-managers/yarn/pom.xml modified: sql/catalyst/pom.xml modified: sql/core/pom.xml modified: sql/hive-thriftserver/pom.xml modified: sql/hive/pom.xml modified: streaming/pom.xml modified: tools/pom.xml ``` Therefore, the following compilation error will occur when using 2.14.0. ``` 2022-12-15T02:37:35.5536924Z [ERROR] [ERROR] Some problems were encountered while processing the POMs: 2022-12-15T02:37:35.5538469Z [FATAL] Non-resolvable parent POM for org.apache.spark:spark-sketch_2.12:3.4.0-SNAPSHOT: Could not find artifact org.apache.spark:spark-parent_2.12:pom:3.4.0-SNAPSHOT and 'parent.relativePath' points at wrong local POM line 22, column 11 ``` ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Pass GitHub Actions Closes apache#39067 from LuciferYang/test-ci. Authored-by: yangjie01 <yangjie01@baidu.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit bfe1af9) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

github-actions bot added BUILD CONNECT PROTOBUF labels Dec 15, 2022

work around

f7d3629

LuciferYang force-pushed the test-ci branch from d6b5f6b to f7d3629 Compare December 15, 2022 07:55

github-actions bot removed PROTOBUF CONNECT labels Dec 15, 2022

LuciferYang changed the title ~~[DON'T MERGE] Test test-dependencies~~ [SPARK-41522][BUILD] Ping versions-maven-plugin to 2.13.0 for GA pass Dec 15, 2022

dongjoon-hyun reviewed Dec 15, 2022

View reviewed changes

dongjoon-hyun changed the title ~~[SPARK-41522][BUILD] Ping versions-maven-plugin to 2.13.0 for GA pass~~ [SPARK-41522][BUILD] Pin versions-maven-plugin to 2.13.0 to recover GitHub Action Dec 15, 2022

ping -> pin

98d2d21

LuciferYang changed the title ~~[SPARK-41522][BUILD] Pin versions-maven-plugin to 2.13.0 to recover GitHub Action~~ [SPARK-41522][BUILD] Pin versions-maven-plugin to 2.13.0 to recover test-dependencies.sh Dec 15, 2022

LuciferYang changed the title ~~[SPARK-41522][BUILD] Pin versions-maven-plugin to 2.13.0 to recover test-dependencies.sh~~ [SPARK-41522][BUILD] Pin versions-maven-plugin to 2.13.0 to recover test-dependencies.sh and make GA pass Dec 15, 2022

fix coments

1d4ca7c

dongjoon-hyun changed the title ~~[SPARK-41522][BUILD] Pin versions-maven-plugin to 2.13.0 to recover test-dependencies.sh and make GA pass~~ [SPARK-41522][BUILD] Pin versions-maven-plugin to 2.13.0 to recover test-dependencies.sh Dec 15, 2022

dongjoon-hyun approved these changes Dec 15, 2022

View reviewed changes

dongjoon-hyun closed this in bfe1af9 Dec 15, 2022

HyukjinKwon reviewed Dec 15, 2022

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-41522][BUILD] Pin `versions-maven-plugin` to 2.13.0 to recover `test-dependencies.sh` #39067

[SPARK-41522][BUILD] Pin `versions-maven-plugin` to 2.13.0 to recover `test-dependencies.sh` #39067

LuciferYang commented Dec 15, 2022 •

edited

dongjoon-hyun commented Dec 15, 2022

LuciferYang commented Dec 15, 2022

LuciferYang commented Dec 15, 2022

dongjoon-hyun Dec 15, 2022

dongjoon-hyun Dec 15, 2022

LuciferYang Dec 15, 2022

LuciferYang commented Dec 15, 2022

dongjoon-hyun commented Dec 15, 2022

LuciferYang commented Dec 15, 2022

dongjoon-hyun commented Dec 15, 2022 •

edited

dongjoon-hyun left a comment •

edited

LuciferYang commented Dec 15, 2022

HyukjinKwon left a comment

LuciferYang commented Dec 15, 2022 •

edited

[SPARK-41522][BUILD] Pin versions-maven-plugin to 2.13.0 to recover test-dependencies.sh #39067

[SPARK-41522][BUILD] Pin versions-maven-plugin to 2.13.0 to recover test-dependencies.sh #39067

Conversation

LuciferYang commented Dec 15, 2022 • edited

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

dongjoon-hyun commented Dec 15, 2022

LuciferYang commented Dec 15, 2022

LuciferYang commented Dec 15, 2022

dongjoon-hyun Dec 15, 2022

Choose a reason for hiding this comment

dongjoon-hyun Dec 15, 2022

Choose a reason for hiding this comment

LuciferYang Dec 15, 2022

Choose a reason for hiding this comment

LuciferYang commented Dec 15, 2022

dongjoon-hyun commented Dec 15, 2022

LuciferYang commented Dec 15, 2022

dongjoon-hyun commented Dec 15, 2022 • edited

dongjoon-hyun left a comment • edited

Choose a reason for hiding this comment

LuciferYang commented Dec 15, 2022

HyukjinKwon left a comment

Choose a reason for hiding this comment

LuciferYang commented Dec 15, 2022 • edited

[SPARK-41522][BUILD] Pin `versions-maven-plugin` to 2.13.0 to recover `test-dependencies.sh` #39067

[SPARK-41522][BUILD] Pin `versions-maven-plugin` to 2.13.0 to recover `test-dependencies.sh` #39067

LuciferYang commented Dec 15, 2022 •

edited

dongjoon-hyun commented Dec 15, 2022 •

edited

dongjoon-hyun left a comment •

edited

LuciferYang commented Dec 15, 2022 •

edited