Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add plots to illustrate what the optimizer does #11072

Merged
merged 6 commits into from
Apr 30, 2024
Merged

Add plots to illustrate what the optimizer does #11072

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
acc5289
Add plots to illustrate what the optimizer does
phofl Apr 25, 2024
b51f72f
Fixup
phofl Apr 25, 2024
cffe539
Update docs/source/dataframe-optimizer.rst
phofl Apr 30, 2024
6317e22
Update docs/source/dataframe-optimizer.rst
phofl Apr 30, 2024
10fc7e0
Update dataframe-optimizer.rst
phofl Apr 30, 2024
5f9a259
Fixup
phofl Apr 30, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
14 changes: 13 additions & 1 deletion docs/source/dataframe-optimizer.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ Optimizer
.. currentmodule:: dask.dataframe

.. note::
Dask DataFrame supports Query Planning since version 2023.03.0
Dask DataFrame supports Query Planning since version 2024.03.0

Optimization steps
------------------
Expand Down Expand Up @@ -38,13 +38,25 @@ The optimizations entail the following steps (this list is not complete):
``df.groupby(...).apply(...)`` after a merge operation will not shuffle the data
again if the groupby happens on the merge columns.

.. figure:: images/optimizer/avoiding-shuffles.svg
:align: center

Similarly, performing two subsequent Joins/Merges on the same join-column(s) will avoid shuffling the
data again. The optimizer identifies that the partitioning of the DataFrame is already as
expected and thus simplifies the operation to a single Shuffle and a trivial merge operation.
phofl marked this conversation as resolved.
Show resolved Hide resolved

- **Automatically resizing partitions:**

The IO layers automatically adjust the partition count based on the column subset
that is selected from the dataset. Very small partitions impact the scheduler and
expensive operations like shuffling negatively. This is addressed by adjusting
the partition count automatically.

.. figure:: images/optimizer/automatic-repartitioning.svg
:align: center

Selecting two columns that together have 40 MB per 200 MB file. The optimizer reduces the number of partitions by a factor of 5.

Exploring the optimized query
-----------------------------

Expand Down