Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speed up tests with federated learning enabled #8350

Merged
merged 2 commits into from Oct 17, 2022

Conversation

rongou
Copy link
Contributor

@rongou rongou commented Oct 17, 2022

Fixes #8334
Fixes #8338

Enabling federated learning slowed down some cpu-only tests, mainly because we increased the number of symbols drastically in libxgboost.so, and the XGBoostAPIGuard throwing an exception and ignoring it is very expensive with symbol lookups. Fixes:

Final result running python/test_updaters.py::TestTreeMethod::test_categorical_ohe:

  • Federated learning disabled: 14.33 seconds
  • Federated learning enabled: 13.99 seconds

Thanks to @eordentlich for finding the root cause.

@trivialfis @hcho3

@hcho3 hcho3 added the Blocking label Oct 17, 2022
Copy link
Collaborator

@hcho3 hcho3 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, assuming the CI passes. Thanks a lot for fixing it so quickly!

@trivialfis
Copy link
Member

Thank you for the fix! That is quick! Please help set the enable parameter to True:

def timeout(sec: int, *args: Any, enable: bool = False, **kwargs: Any) -> Any:
.

@hcho3
Copy link
Collaborator

hcho3 commented Oct 17, 2022

The CPU tests are now very fast:
image

@hcho3 hcho3 merged commit 8f3dee5 into dmlc:master Oct 17, 2022
@hcho3
Copy link
Collaborator

hcho3 commented Oct 17, 2022

@rongou @eordentlich The 1.7 release is unblocked now. Thanks!

@rongou rongou deleted the speed-up-fed-tests branch November 18, 2022 19:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Bunch of CPU tests are timing out
3 participants