Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve the speed of from_dataframe with a MultiIndex (by 40x!) #4184

Merged
merged 13 commits into from Jul 2, 2020
Merged

Improve the speed of from_dataframe with a MultiIndex (by 40x!) #4184

merged 13 commits into from Jul 2, 2020

Commits on Jun 26, 2020

  1. Add MultiIndexSeries.time_to_xarray() benchmark

    @shoyer
    shoyer committed Jun 26, 2020
    Configuration menu
    Copy the full SHA
    c340961 View commit details
    Browse the repository at this point in the history

  2. Improve the speed of from_dataframe with a MultiIndex 

    Fixes pydataGH-2459

Before:

    pandas.MultiIndexSeries.time_to_xarray
    ======= ========= ==========
    --             subset
    ------- --------------------
    dtype     True     False
    ======= ========= ==========
      int    505±0ms   37.1±0ms
     float   485±0ms   38.3±0ms
    ======= ========= ==========

After:

    pandas.MultiIndexSeries.time_to_xarray
    ======= ========= ==========
    --             subset
    ------- --------------------
    dtype     True     False
    ======= ========= ==========
      int    11.5±0ms   39.2±0ms
     float   12.5±0ms   26.6±0ms
    ======= ========= ==========

There are still some cases where we have to fall back to the existing
slow implementation, but hopefully they should now be relatively rare.
    @shoyer
    shoyer committed Jun 26, 2020
    Configuration menu
    Copy the full SHA
    aa5a7dd View commit details
    Browse the repository at this point in the history

  3. remove unused import

    @shoyer
    shoyer committed Jun 26, 2020
    Configuration menu
    Copy the full SHA
    cde5aff View commit details
    Browse the repository at this point in the history

Commits on Jun 27, 2020

  1. Simplify converting MultiIndex dataframes

    @shoyer
    shoyer committed Jun 27, 2020
    Configuration menu
    Copy the full SHA
    4136dc6 View commit details
    Browse the repository at this point in the history

  2. remove comments

    @shoyer
    shoyer committed Jun 27, 2020
    Configuration menu
    Copy the full SHA
    41556d2 View commit details
    Browse the repository at this point in the history

  3. remove types with NA

    @shoyer
    shoyer committed Jun 27, 2020
    Configuration menu
    Copy the full SHA
    f3e2d2d View commit details
    Browse the repository at this point in the history

  4. more multiindex dataframe tests

    @shoyer
    shoyer committed Jun 27, 2020
    Configuration menu
    Copy the full SHA
    e648448 View commit details
    Browse the repository at this point in the history

  5. add whats new note

    @shoyer
    shoyer committed Jun 27, 2020
    Configuration menu
    Copy the full SHA
    f50ba68 View commit details
    Browse the repository at this point in the history

Commits on Jun 30, 2020

  1. Preserve order of MultiIndex levels in from_dataframe

    @shoyer
    shoyer committed Jun 30, 2020
    Configuration menu
    Copy the full SHA
    0383aa1 View commit details
    Browse the repository at this point in the history

  2. Add todo note

    @shoyer
    shoyer committed Jun 30, 2020
    Configuration menu
    Copy the full SHA
    9ac4557 View commit details
    Browse the repository at this point in the history

  3. Rewrite from_dataframe to avoid passing around a dataframe

    @shoyer
    shoyer committed Jun 30, 2020
    Configuration menu
    Copy the full SHA
    96b544b View commit details
    Browse the repository at this point in the history

  4. Require that MultiIndexes are unique even with sparse=True

    @shoyer
    shoyer committed Jun 30, 2020
    Configuration menu
    Copy the full SHA
    8b51939 View commit details
    Browse the repository at this point in the history

Commits on Jul 2, 2020

  1. clarify comment

    @shoyer
    shoyer committed Jul 2, 2020
    Configuration menu
    Copy the full SHA
    ecaa332 View commit details
    Browse the repository at this point in the history