Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

With multiple scorers, if a scorer fails, GridSearchCV will fail #830

Open
bolliger32 opened this issue May 3, 2021 · 0 comments · May be fixed by #831
Open

With multiple scorers, if a scorer fails, GridSearchCV will fail #830

bolliger32 opened this issue May 3, 2021 · 0 comments · May be fixed by #831

Comments

@bolliger32
Copy link

bolliger32 commented May 3, 2021

What happened:
I was running GridSearchCV with multiple scoring metrics. One of them ("neg_mean_poisson_deviance") was undefined for some folds b/c it is undefined when y_hat is 0. This was handled during scoring but when create_cv_results was called, this raised a TypeError: 'float' object is not subscriptable. This is b/c score would normally return a dictionary when mutliple scorers are requested but in this case it returned the value I had passed as error_score to GridSearchCV, which in this case was np.nan. The issue is between L274 and L297 in methods.py.

What you expected to happen:
I expected that score to be np.nan for the folds in which the scorer failed, but not to raise an error

Minimal Complete Verifiable Example:

from sklearn.linear_model import LinearRegression
from dask_ml.model_selection import GridSearchCV
from sklearn.model_selection import LeaveOneOut
import numpy as np

X = np.array([[1, 2],
              [2, 1],
              [0, 0]])

y = 3 * X[:, 0] + 4 * X[:, 1]
cv = LeaveOneOut()

ols = LinearRegression(fit_intercept=False)
regr = GridSearchCV(
    ols,
    {"normalize": [False, True]},
    scoring=["neg_mean_squared_error", "neg_mean_poisson_deviance"],
    refit=False,
    cv=cv,
    error_score=np.nan,
    n_jobs=1
)
regr.fit(X,y)

This gives the TypeError I mentioned

Anything else we need to know?:
I think this should be a fairly quick fix so I'm going to give it a try

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant