Numexpr on dask #8098
-
|
Numexpr uses multi-threading to achieve better performance. Will this resource contention on used with dask dataframe to parallelise workflows? |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 1 reply
-
|
Since pandas will sometimes use numexpr if it's available, Dask may compete with numexpr for thread usage with some dask.dataframe workloads. The dask.dataframe documentation suggests limiting numexpr to a single thread; see e.g. dask.dataframe.DataFrame.query |
Beta Was this translation helpful? Give feedback.
-
|
Thanks @douglasdavis . Concise and informative! |
Beta Was this translation helpful? Give feedback.
Since pandas will sometimes use numexpr if it's available, Dask may compete with numexpr for thread usage with some dask.dataframe workloads. The dask.dataframe documentation suggests limiting numexpr to a single thread; see e.g. dask.dataframe.DataFrame.query