Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-41522][BUILD] Pin versions-maven-plugin to 2.13.0 to recover test-dependencies.sh #39067

Closed
wants to merge 3 commits into from

Conversation

LuciferYang
Copy link
Contributor

@LuciferYang LuciferYang commented Dec 15, 2022

What changes were proposed in this pull request?

This pr aims to pin versions-maven-plugin to 2.13.0 to recover test-dependencies.sh and make GA Dependencies test task pass , this pr should revert after we know how to use version 2.14.0.

Why are the changes needed?

dev/test-dependencies.sh always use latest versions-maven-plugin version, and versions-maven-plugin 2.14.0 has not set the version of the sub-module.

Run:

build/mvn -q versions:set -DnewVersion=spark-928034 -DgenerateBackupPoms=false

2.14.0

+ git status
On branch test-ci
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
  modified:   assembly/pom.xml
  modified:   core/pom.xml
  modified:   examples/pom.xml
  modified:   graphx/pom.xml
  modified:   hadoop-cloud/pom.xml
  modified:   launcher/pom.xml
  modified:   mllib-local/pom.xml
  modified:   mllib/pom.xml
  modified:   pom.xml
  modified:   repl/pom.xml
  modified:   streaming/pom.xml
  modified:   tools/pom.xml

2.13.0

+ git status
On branch test-ci
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
  modified:   assembly/pom.xml
  modified:   common/kvstore/pom.xml
  modified:   common/network-common/pom.xml
  modified:   common/network-shuffle/pom.xml
  modified:   common/network-yarn/pom.xml
  modified:   common/sketch/pom.xml
  modified:   common/tags/pom.xml
  modified:   common/unsafe/pom.xml
  modified:   connector/avro/pom.xml
  modified:   connector/connect/common/pom.xml
  modified:   connector/connect/server/pom.xml
  modified:   connector/docker-integration-tests/pom.xml
  modified:   connector/kafka-0-10-assembly/pom.xml
  modified:   connector/kafka-0-10-sql/pom.xml
  modified:   connector/kafka-0-10-token-provider/pom.xml
  modified:   connector/kafka-0-10/pom.xml
  modified:   connector/kinesis-asl-assembly/pom.xml
  modified:   connector/kinesis-asl/pom.xml
  modified:   connector/protobuf/pom.xml
  modified:   connector/spark-ganglia-lgpl/pom.xml
  modified:   core/pom.xml
  modified:   dev/test-dependencies.sh
  modified:   examples/pom.xml
  modified:   graphx/pom.xml
  modified:   hadoop-cloud/pom.xml
  modified:   launcher/pom.xml
  modified:   mllib-local/pom.xml
  modified:   mllib/pom.xml
  modified:   pom.xml
  modified:   repl/pom.xml
  modified:   resource-managers/kubernetes/core/pom.xml
  modified:   resource-managers/kubernetes/integration-tests/pom.xml
  modified:   resource-managers/mesos/pom.xml
  modified:   resource-managers/yarn/pom.xml
  modified:   sql/catalyst/pom.xml
  modified:   sql/core/pom.xml
  modified:   sql/hive-thriftserver/pom.xml
  modified:   sql/hive/pom.xml
  modified:   streaming/pom.xml
  modified:   tools/pom.xml

Therefore, the following compilation error will occur when using 2.14.0.

2022-12-15T02:37:35.5536924Z [ERROR] [ERROR] Some problems were encountered while processing the POMs:
2022-12-15T02:37:35.5538469Z [FATAL] Non-resolvable parent POM for org.apache.spark:spark-sketch_2.12:3.4.0-SNAPSHOT: Could not find artifact org.apache.spark:spark-parent_2.12:pom:3.4.0-SNAPSHOT and 'parent.relativePath' points at wrong local POM @ line 22, column 11

Does this PR introduce any user-facing change?

No

How was this patch tested?

Pass GitHub Actions

@dongjoon-hyun
Copy link
Member

Thank you for investigating this, @LuciferYang .

@LuciferYang
Copy link
Contributor Author

Different from the local error, further investigation is required

@LuciferYang LuciferYang changed the title [DON'T MERGE] Test test-dependencies [SPARK-41522][BUILD] Ping versions-maven-plugin to 2.13.0 for GA pass Dec 15, 2022
@LuciferYang
Copy link
Contributor Author

@dongjoon-hyun @HyukjinKwon This pr provides a workaround solution. We can restore GA first. I will give another pr when I know how to use the latest versions-maven-plugin version

@@ -76,11 +76,13 @@ function reset_version {
find "$HOME/.m2/" | grep "$TEMP_VERSION" | xargs rm -rf

# Restore the original version number:
$MVN -q versions:set -DnewVersion=$OLD_VERSION -DgenerateBackupPoms=false > /dev/null
# SPARK-41522: ping versions-maven-plugin version for GA pass
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ping -> pin?

}
trap reset_version EXIT

$MVN -q versions:set -DnewVersion=$TEMP_VERSION -DgenerateBackupPoms=false > /dev/null
# SPARK-41522: ping versions-maven-plugin version for GA pass
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ping -> pin?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks

@dongjoon-hyun dongjoon-hyun changed the title [SPARK-41522][BUILD] Ping versions-maven-plugin to 2.13.0 for GA pass [SPARK-41522][BUILD] Pin versions-maven-plugin to 2.13.0 to recover GitHub Action Dec 15, 2022
@LuciferYang
Copy link
Contributor Author

@dongjoon-hyun
Copy link
Member

I agree with this workaround, @LuciferYang . I left two typo comments.

@LuciferYang
Copy link
Contributor Author

98d2d21 fix typos

@dongjoon-hyun
Copy link
Member

dongjoon-hyun commented Dec 15, 2022

BTW, this PR recovers local test failures too because dev/test-dependencies.sh fails locally in the same way.
Could you revise the code, PR title, and description? Something like to recover test-dependencies.sh?

@LuciferYang LuciferYang changed the title [SPARK-41522][BUILD] Pin versions-maven-plugin to 2.13.0 to recover GitHub Action [SPARK-41522][BUILD] Pin versions-maven-plugin to 2.13.0 to recover test-dependencies.sh Dec 15, 2022
@LuciferYang LuciferYang changed the title [SPARK-41522][BUILD] Pin versions-maven-plugin to 2.13.0 to recover test-dependencies.sh [SPARK-41522][BUILD] Pin versions-maven-plugin to 2.13.0 to recover test-dependencies.sh and make GA pass Dec 15, 2022
@dongjoon-hyun dongjoon-hyun changed the title [SPARK-41522][BUILD] Pin versions-maven-plugin to 2.13.0 to recover test-dependencies.sh and make GA pass [SPARK-41522][BUILD] Pin versions-maven-plugin to 2.13.0 to recover test-dependencies.sh Dec 15, 2022
Copy link
Member

@dongjoon-hyun dongjoon-hyun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, LGTM. Merged to master/3.3/3.2 to recover CI.
Thank you so much, @LuciferYang .

dongjoon-hyun pushed a commit that referenced this pull request Dec 15, 2022
… `test-dependencies.sh`

### What changes were proposed in this pull request?
This pr aims to pin `versions-maven-plugin` to 2.13.0 to recover `test-dependencies.sh` and make GA pass , this pr should revert after we know how to use version 2.14.0.

### Why are the changes needed?
`dev/test-dependencies.sh` always use latest `versions-maven-plugin` version, and `versions-maven-plugin` 2.14.0 has not set the version of the sub-module.

Run:

```
build/mvn -q versions:set -DnewVersion=spark-928034 -DgenerateBackupPoms=false
```

**2.14.0**

```
+ git status
On branch test-ci
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
  modified:   assembly/pom.xml
  modified:   core/pom.xml
  modified:   examples/pom.xml
  modified:   graphx/pom.xml
  modified:   hadoop-cloud/pom.xml
  modified:   launcher/pom.xml
  modified:   mllib-local/pom.xml
  modified:   mllib/pom.xml
  modified:   pom.xml
  modified:   repl/pom.xml
  modified:   streaming/pom.xml
  modified:   tools/pom.xml

```

**2.13.0**

```
+ git status
On branch test-ci
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
  modified:   assembly/pom.xml
  modified:   common/kvstore/pom.xml
  modified:   common/network-common/pom.xml
  modified:   common/network-shuffle/pom.xml
  modified:   common/network-yarn/pom.xml
  modified:   common/sketch/pom.xml
  modified:   common/tags/pom.xml
  modified:   common/unsafe/pom.xml
  modified:   connector/avro/pom.xml
  modified:   connector/connect/common/pom.xml
  modified:   connector/connect/server/pom.xml
  modified:   connector/docker-integration-tests/pom.xml
  modified:   connector/kafka-0-10-assembly/pom.xml
  modified:   connector/kafka-0-10-sql/pom.xml
  modified:   connector/kafka-0-10-token-provider/pom.xml
  modified:   connector/kafka-0-10/pom.xml
  modified:   connector/kinesis-asl-assembly/pom.xml
  modified:   connector/kinesis-asl/pom.xml
  modified:   connector/protobuf/pom.xml
  modified:   connector/spark-ganglia-lgpl/pom.xml
  modified:   core/pom.xml
  modified:   dev/test-dependencies.sh
  modified:   examples/pom.xml
  modified:   graphx/pom.xml
  modified:   hadoop-cloud/pom.xml
  modified:   launcher/pom.xml
  modified:   mllib-local/pom.xml
  modified:   mllib/pom.xml
  modified:   pom.xml
  modified:   repl/pom.xml
  modified:   resource-managers/kubernetes/core/pom.xml
  modified:   resource-managers/kubernetes/integration-tests/pom.xml
  modified:   resource-managers/mesos/pom.xml
  modified:   resource-managers/yarn/pom.xml
  modified:   sql/catalyst/pom.xml
  modified:   sql/core/pom.xml
  modified:   sql/hive-thriftserver/pom.xml
  modified:   sql/hive/pom.xml
  modified:   streaming/pom.xml
  modified:   tools/pom.xml
```

Therefore, the following compilation error will occur when using 2.14.0.

```
2022-12-15T02:37:35.5536924Z [ERROR] [ERROR] Some problems were encountered while processing the POMs:
2022-12-15T02:37:35.5538469Z [FATAL] Non-resolvable parent POM for org.apache.spark:spark-sketch_2.12:3.4.0-SNAPSHOT: Could not find artifact org.apache.spark:spark-parent_2.12:pom:3.4.0-SNAPSHOT and 'parent.relativePath' points at wrong local POM  line 22, column 11
```

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
Pass GitHub Actions

Closes #39067 from LuciferYang/test-ci.

Authored-by: yangjie01 <yangjie01@baidu.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
(cherry picked from commit bfe1af9)
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
dongjoon-hyun pushed a commit that referenced this pull request Dec 15, 2022
… `test-dependencies.sh`

### What changes were proposed in this pull request?
This pr aims to pin `versions-maven-plugin` to 2.13.0 to recover `test-dependencies.sh` and make GA pass , this pr should revert after we know how to use version 2.14.0.

### Why are the changes needed?
`dev/test-dependencies.sh` always use latest `versions-maven-plugin` version, and `versions-maven-plugin` 2.14.0 has not set the version of the sub-module.

Run:

```
build/mvn -q versions:set -DnewVersion=spark-928034 -DgenerateBackupPoms=false
```

**2.14.0**

```
+ git status
On branch test-ci
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
  modified:   assembly/pom.xml
  modified:   core/pom.xml
  modified:   examples/pom.xml
  modified:   graphx/pom.xml
  modified:   hadoop-cloud/pom.xml
  modified:   launcher/pom.xml
  modified:   mllib-local/pom.xml
  modified:   mllib/pom.xml
  modified:   pom.xml
  modified:   repl/pom.xml
  modified:   streaming/pom.xml
  modified:   tools/pom.xml

```

**2.13.0**

```
+ git status
On branch test-ci
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
  modified:   assembly/pom.xml
  modified:   common/kvstore/pom.xml
  modified:   common/network-common/pom.xml
  modified:   common/network-shuffle/pom.xml
  modified:   common/network-yarn/pom.xml
  modified:   common/sketch/pom.xml
  modified:   common/tags/pom.xml
  modified:   common/unsafe/pom.xml
  modified:   connector/avro/pom.xml
  modified:   connector/connect/common/pom.xml
  modified:   connector/connect/server/pom.xml
  modified:   connector/docker-integration-tests/pom.xml
  modified:   connector/kafka-0-10-assembly/pom.xml
  modified:   connector/kafka-0-10-sql/pom.xml
  modified:   connector/kafka-0-10-token-provider/pom.xml
  modified:   connector/kafka-0-10/pom.xml
  modified:   connector/kinesis-asl-assembly/pom.xml
  modified:   connector/kinesis-asl/pom.xml
  modified:   connector/protobuf/pom.xml
  modified:   connector/spark-ganglia-lgpl/pom.xml
  modified:   core/pom.xml
  modified:   dev/test-dependencies.sh
  modified:   examples/pom.xml
  modified:   graphx/pom.xml
  modified:   hadoop-cloud/pom.xml
  modified:   launcher/pom.xml
  modified:   mllib-local/pom.xml
  modified:   mllib/pom.xml
  modified:   pom.xml
  modified:   repl/pom.xml
  modified:   resource-managers/kubernetes/core/pom.xml
  modified:   resource-managers/kubernetes/integration-tests/pom.xml
  modified:   resource-managers/mesos/pom.xml
  modified:   resource-managers/yarn/pom.xml
  modified:   sql/catalyst/pom.xml
  modified:   sql/core/pom.xml
  modified:   sql/hive-thriftserver/pom.xml
  modified:   sql/hive/pom.xml
  modified:   streaming/pom.xml
  modified:   tools/pom.xml
```

Therefore, the following compilation error will occur when using 2.14.0.

```
2022-12-15T02:37:35.5536924Z [ERROR] [ERROR] Some problems were encountered while processing the POMs:
2022-12-15T02:37:35.5538469Z [FATAL] Non-resolvable parent POM for org.apache.spark:spark-sketch_2.12:3.4.0-SNAPSHOT: Could not find artifact org.apache.spark:spark-parent_2.12:pom:3.4.0-SNAPSHOT and 'parent.relativePath' points at wrong local POM  line 22, column 11
```

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
Pass GitHub Actions

Closes #39067 from LuciferYang/test-ci.

Authored-by: yangjie01 <yangjie01@baidu.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
(cherry picked from commit bfe1af9)
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
@LuciferYang
Copy link
Contributor Author

Thanks @dongjoon-hyun

Copy link
Member

@HyukjinKwon HyukjinKwon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM2

@LuciferYang
Copy link
Contributor Author

LuciferYang commented Dec 15, 2022

Further feedback, I found that this is a bug of versions-maven-plugin 2.14.0, nested modules cannot be updated version,

and an issue is tracking this:

we need wait 2.14.1 for fix, @dongjoon-hyun @HyukjinKwon

sunchao pushed a commit to sunchao/spark that referenced this pull request Jun 2, 2023
… `test-dependencies.sh`

### What changes were proposed in this pull request?
This pr aims to pin `versions-maven-plugin` to 2.13.0 to recover `test-dependencies.sh` and make GA pass , this pr should revert after we know how to use version 2.14.0.

### Why are the changes needed?
`dev/test-dependencies.sh` always use latest `versions-maven-plugin` version, and `versions-maven-plugin` 2.14.0 has not set the version of the sub-module.

Run:

```
build/mvn -q versions:set -DnewVersion=spark-928034 -DgenerateBackupPoms=false
```

**2.14.0**

```
+ git status
On branch test-ci
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
  modified:   assembly/pom.xml
  modified:   core/pom.xml
  modified:   examples/pom.xml
  modified:   graphx/pom.xml
  modified:   hadoop-cloud/pom.xml
  modified:   launcher/pom.xml
  modified:   mllib-local/pom.xml
  modified:   mllib/pom.xml
  modified:   pom.xml
  modified:   repl/pom.xml
  modified:   streaming/pom.xml
  modified:   tools/pom.xml

```

**2.13.0**

```
+ git status
On branch test-ci
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
  modified:   assembly/pom.xml
  modified:   common/kvstore/pom.xml
  modified:   common/network-common/pom.xml
  modified:   common/network-shuffle/pom.xml
  modified:   common/network-yarn/pom.xml
  modified:   common/sketch/pom.xml
  modified:   common/tags/pom.xml
  modified:   common/unsafe/pom.xml
  modified:   connector/avro/pom.xml
  modified:   connector/connect/common/pom.xml
  modified:   connector/connect/server/pom.xml
  modified:   connector/docker-integration-tests/pom.xml
  modified:   connector/kafka-0-10-assembly/pom.xml
  modified:   connector/kafka-0-10-sql/pom.xml
  modified:   connector/kafka-0-10-token-provider/pom.xml
  modified:   connector/kafka-0-10/pom.xml
  modified:   connector/kinesis-asl-assembly/pom.xml
  modified:   connector/kinesis-asl/pom.xml
  modified:   connector/protobuf/pom.xml
  modified:   connector/spark-ganglia-lgpl/pom.xml
  modified:   core/pom.xml
  modified:   dev/test-dependencies.sh
  modified:   examples/pom.xml
  modified:   graphx/pom.xml
  modified:   hadoop-cloud/pom.xml
  modified:   launcher/pom.xml
  modified:   mllib-local/pom.xml
  modified:   mllib/pom.xml
  modified:   pom.xml
  modified:   repl/pom.xml
  modified:   resource-managers/kubernetes/core/pom.xml
  modified:   resource-managers/kubernetes/integration-tests/pom.xml
  modified:   resource-managers/mesos/pom.xml
  modified:   resource-managers/yarn/pom.xml
  modified:   sql/catalyst/pom.xml
  modified:   sql/core/pom.xml
  modified:   sql/hive-thriftserver/pom.xml
  modified:   sql/hive/pom.xml
  modified:   streaming/pom.xml
  modified:   tools/pom.xml
```

Therefore, the following compilation error will occur when using 2.14.0.

```
2022-12-15T02:37:35.5536924Z [ERROR] [ERROR] Some problems were encountered while processing the POMs:
2022-12-15T02:37:35.5538469Z [FATAL] Non-resolvable parent POM for org.apache.spark:spark-sketch_2.12:3.4.0-SNAPSHOT: Could not find artifact org.apache.spark:spark-parent_2.12:pom:3.4.0-SNAPSHOT and 'parent.relativePath' points at wrong local POM  line 22, column 11
```

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
Pass GitHub Actions

Closes apache#39067 from LuciferYang/test-ci.

Authored-by: yangjie01 <yangjie01@baidu.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
(cherry picked from commit bfe1af9)
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
3 participants