Skip to content

Commit

Permalink
[SPARK-41522][BUILD] Pin versions-maven-plugin to 2.13.0 to recover…
Browse files Browse the repository at this point in the history
… `test-dependencies.sh`

### What changes were proposed in this pull request?
This pr aims to pin `versions-maven-plugin` to 2.13.0 to recover `test-dependencies.sh` and make GA pass , this pr should revert after we know how to use version 2.14.0.

### Why are the changes needed?
`dev/test-dependencies.sh` always use latest `versions-maven-plugin` version, and `versions-maven-plugin` 2.14.0 has not set the version of the sub-module.

Run:

```
build/mvn -q versions:set -DnewVersion=spark-928034 -DgenerateBackupPoms=false
```

**2.14.0**

```
+ git status
On branch test-ci
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
  modified:   assembly/pom.xml
  modified:   core/pom.xml
  modified:   examples/pom.xml
  modified:   graphx/pom.xml
  modified:   hadoop-cloud/pom.xml
  modified:   launcher/pom.xml
  modified:   mllib-local/pom.xml
  modified:   mllib/pom.xml
  modified:   pom.xml
  modified:   repl/pom.xml
  modified:   streaming/pom.xml
  modified:   tools/pom.xml

```

**2.13.0**

```
+ git status
On branch test-ci
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
  modified:   assembly/pom.xml
  modified:   common/kvstore/pom.xml
  modified:   common/network-common/pom.xml
  modified:   common/network-shuffle/pom.xml
  modified:   common/network-yarn/pom.xml
  modified:   common/sketch/pom.xml
  modified:   common/tags/pom.xml
  modified:   common/unsafe/pom.xml
  modified:   connector/avro/pom.xml
  modified:   connector/connect/common/pom.xml
  modified:   connector/connect/server/pom.xml
  modified:   connector/docker-integration-tests/pom.xml
  modified:   connector/kafka-0-10-assembly/pom.xml
  modified:   connector/kafka-0-10-sql/pom.xml
  modified:   connector/kafka-0-10-token-provider/pom.xml
  modified:   connector/kafka-0-10/pom.xml
  modified:   connector/kinesis-asl-assembly/pom.xml
  modified:   connector/kinesis-asl/pom.xml
  modified:   connector/protobuf/pom.xml
  modified:   connector/spark-ganglia-lgpl/pom.xml
  modified:   core/pom.xml
  modified:   dev/test-dependencies.sh
  modified:   examples/pom.xml
  modified:   graphx/pom.xml
  modified:   hadoop-cloud/pom.xml
  modified:   launcher/pom.xml
  modified:   mllib-local/pom.xml
  modified:   mllib/pom.xml
  modified:   pom.xml
  modified:   repl/pom.xml
  modified:   resource-managers/kubernetes/core/pom.xml
  modified:   resource-managers/kubernetes/integration-tests/pom.xml
  modified:   resource-managers/mesos/pom.xml
  modified:   resource-managers/yarn/pom.xml
  modified:   sql/catalyst/pom.xml
  modified:   sql/core/pom.xml
  modified:   sql/hive-thriftserver/pom.xml
  modified:   sql/hive/pom.xml
  modified:   streaming/pom.xml
  modified:   tools/pom.xml
```

Therefore, the following compilation error will occur when using 2.14.0.

```
2022-12-15T02:37:35.5536924Z [ERROR] [ERROR] Some problems were encountered while processing the POMs:
2022-12-15T02:37:35.5538469Z [FATAL] Non-resolvable parent POM for org.apache.spark:spark-sketch_2.12:3.4.0-SNAPSHOT: Could not find artifact org.apache.spark:spark-parent_2.12:pom:3.4.0-SNAPSHOT and 'parent.relativePath' points at wrong local POM  line 22, column 11
```

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
Pass GitHub Actions

Closes #39067 from LuciferYang/test-ci.

Authored-by: yangjie01 <yangjie01@baidu.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
(cherry picked from commit bfe1af9)
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
  • Loading branch information
LuciferYang authored and dongjoon-hyun committed Dec 15, 2022
1 parent c6bd820 commit 54a1c0f
Showing 1 changed file with 4 additions and 2 deletions.
6 changes: 4 additions & 2 deletions dev/test-dependencies.sh
Expand Up @@ -74,11 +74,13 @@ function reset_version {
find "$HOME/.m2/" | grep "$TEMP_VERSION" | xargs rm -rf

# Restore the original version number:
$MVN -q versions:set -DnewVersion=$OLD_VERSION -DgenerateBackupPoms=false > /dev/null
# SPARK-41522: pin versions-maven-plugin version to recover test-dependencies.sh and make GA task pass.
$MVN -q org.codehaus.mojo:versions-maven-plugin:2.13.0:set -DnewVersion=$OLD_VERSION -DgenerateBackupPoms=false > /dev/null
}
trap reset_version EXIT

$MVN -q versions:set -DnewVersion=$TEMP_VERSION -DgenerateBackupPoms=false > /dev/null
# SPARK-41522: pin versions-maven-plugin version to recover test-dependencies.sh and make GA task pass.
$MVN -q org.codehaus.mojo:versions-maven-plugin:2.13.0:set -DnewVersion=$TEMP_VERSION -DgenerateBackupPoms=false > /dev/null

# Generate manifests for each Hadoop profile:
for HADOOP_HIVE_PROFILE in "${HADOOP_HIVE_PROFILES[@]}"; do
Expand Down

0 comments on commit 54a1c0f

Please sign in to comment.