Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consider adding POLARS_MAX_THREADS to joblib._parallel_backends.MAX_NUM_THREADS_VARS #1548

Open
ogrisel opened this issue Feb 19, 2024 · 1 comment

Comments

@ogrisel
Copy link
Contributor

ogrisel commented Feb 19, 2024

This would allow to protect against over-subscription when nesting polars multithreaded operations below joblib.Parallel calls when using the process-based loky backend.

See: pola-rs/polars#7922

Similarly to what we already do for OPENBLAS_NUM_THREADS, MKL_NUM_THREADS, OMP_NUM_THREADS and friends.

Note: in addition to adding the var to the list, the doc and tests need to be updated accordingly, see for instance:

https://github.com/search?q=repo%3Ajoblib%2Fjoblib+openblas_num_threads+openblas_num_threads&type=code

@lorentzenchr
Copy link

lorentzenchr commented Mar 25, 2024

Currently, one can only set the number of threads for polars to use once at the start of polars (before import polars), see pola-rs/polars#6157.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants