[pyspark] Fix xgboost spark estimator dataset repartition issues #8231
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
CC @trivialfis @wbo4958 Thanks! |
wbo4958
reviewed
Sep 8, 2022
WeichenXu123
force-pushed
the
fix-repartition
branch
2 times, most recently
from
0828de4
to
6701cd3
Compare
Signed-off-by: Weichen Xu <weichen.xu@databricks.com>
WeichenXu123
force-pushed
the
fix-repartition
branch
from
6701cd3
to
7bcdda3
Compare
Signed-off-by: Weichen Xu <weichen.xu@databricks.com>
WeichenXu123
force-pushed
the
fix-repartition
branch
from
002676f
to
8596b89
Compare
Signed-off-by: Weichen Xu <weichen.xu@databricks.com>
|
CC @wbo4958 |
wbo4958
reviewed
Sep 14, 2022
python-package/xgboost/spark/core.py
Outdated
| # Repartition on `rand` column to avoid repartition | ||
| # result unbalance. Directly using `.repartition(N)` might result in some | ||
| # empty partitions. | ||
| dataset = dataset.repartition(num_workers, rand(1)) |
There was a problem hiding this comment.
I still prefer to add a parameter to control if random shuffle. Since my PR #8245 can detect if the training data is empty DMatrix, so we can add some log printing about data skew and how to resolve it by enabling xxx parameter. Does that make sense?
|
Retriggering the CI, as we recently transitioned away from Jenkins |
WeichenXu123
force-pushed
the
fix-repartition
branch
from
19f7602
to
9410a89
Compare
|
@wbo4958 Is the PR good to merge ? (except the lint errors) |
|
Hi, could you please share the current status of this fix? Is it still necessary after @wbo4958 's PR? |
|
No, we can file the following PR for it. This PR looks good to me. |
wbo4958
approved these changes
Sep 17, 2022
|
@trivialfis @wbo4958 Ready for merging ! :) |
1 similar comment
|
@trivialfis @wbo4958 Ready for merging ! :) |
|
+1 ed |
wbo4958
approved these changes
Sep 22, 2022
trivialfis
approved these changes
Sep 22, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Signed-off-by: Weichen Xu weichen.xu@databricks.com
Fix xgboost spark estimator dataset repartition issues
rand(1)column.validationIndicatorColset, always trigger repartition.Closes #8221