forked from apache/spark
-
Notifications
You must be signed in to change notification settings - Fork 17
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* Spark jars should have a dedicated artifact id * Use python 3 instead of python 2. This requires to make /usr/bin/python point to python3 * Add -l option to docker useradd command to prevent a bug using long uid moby/moby#5419 * Remove \\\ in the maven settings.xml file in the container otherwise the variables criteo.repo.username|password are not correctly used and lead to 401 issues when uploading files to nexus * Fix hive dependency. Groupid has to be org.spark-project.hive and artifactid version 1.2.1.spark2 * use set -e in the build script to fail immediatly when an error occur in any command * Spark scala is a profile and cannot be activated with -D java option but rather with -P maven option * Use --no-transfer-progress in maven commands to make output readable * All mvn commands were using \\ instead of \ leading to bad command interpreation by bash * Fix tar command used to build the 'jar-only' tgz * Fix mvn jar:jar deploy:deploy parameters which are not exactly the same as the ones of mvn deploy:deploy-file * Upgrade pip in the venv otherwise pyarrow cannot install * Fix altDeploymentRepository declaration * Set pypandoc version to 1.5 as some functions have been removed in newer versions
- Loading branch information
w.montaz
committed
May 16, 2023
1 parent
3b23a78
commit 632109a
Showing
5 changed files
with
61 additions
and
60 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,2 +1,2 @@ | ||
HDP_VERSION=3.3.0-criteo-20230320100819 | ||
HIVE_VERSION=1.2.1 | ||
HIVE_VERSION=1.2.1.spark2 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,7 +1,7 @@ | ||
wheel | ||
numpy | ||
pandas | ||
pypandoc | ||
pypandoc==1.5 | ||
py4j==0.10.7 | ||
pyarrow | ||
twine | ||
|