New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

`SimpleImputer.fit` strange behavior with `median` and `most_frequent` strategies #787

Open

abduhbm opened this issue Feb 4, 2021 · 0 comments · May be fixed by #788

Contributor

abduhbm commented Feb 4, 2021 •

edited

What happened:
SimpleImputer.fit with median and most_frequent strategies on frames compute different results comparing to scikit-learn.

What you expected to happen:
They should have consistent results with sklearn.impute.SimpleImputer.

Minimal Complete Verifiable Example:

df = pd.DataFrame({"A": [1, 1, np.nan, np.nan, 2, 2]})
# This should return the smallest value
b = dask_ml.impute.SimpleImputer(strategy="most_frequent", fill_value=None)
b.fit(df)
b.statistics_
>>> A    2.0
>>> dtype: float64

c = sklearn.impute.SimpleImputer(strategy="most_frequent", fill_value=None)
c.fit(df)
c.statistics_
>>> array([1.])

With median:

df = pd.DataFrame({"A": [1, 1, np.nan, np.nan, 2, 2]})
df = dd.from_pandas(df, 2)
b = dask_ml.impute.SimpleImputer(strategy="median", fill_value=None)
b.fit(df)
b.statistics_
>>> A    1.0
>>> dtype: float64

c = sklearn.impute.SimpleImputer(strategy="median", fill_value=None)
c.fit(df)
c.statistics_
>>> array([1.5])

Environment:

Dask version: 2021.01.1
Python version: 3.7.6
Operating System: MacOS
Install method (conda, pip, source): pip

The text was updated successfully, but these errors were encountered:

abduhbm linked a pull request

that will close this issue

Fix simple impute #788

Open

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment