How to store a pipeline contains Gridsearchcv object?

[Erica-Ko]

I have the following model built using the Pipeline of scikit-learn pipeline:

def gen_binary_lightgbm(X_train, y_train, round_num, metric):
params = {
'learning_rate' : [0.001,0.01,0.05,0.08,0.1],
'reg_lambda' : [0,0.05,0.5,1]
}

gkf = KFold(n_splits=5, shuffle=True, random_state=42).split(X_train, y_train)
lgb_estimator = lgb.LGBMClassifier(objective='binary',num_boost_round=round_num)

gsearch = Pipeline([('scaler', StandardScaler()), ('model', GridSearchCV(
        estimator=lgb_estimator,
        param_grid=params,
        n_jobs=-1,
        scoring = metric,
        refit= metric,
        cv=gkf,
        verbose=-1,
        pre_dispatch=8,
        error_score=-999,
        return_train_score=True
        ))])

lgb_model = gsearch.fit(X_train, y_train)
return lgb_model

When I try to pickle the lgb_model which is pipeline:

with open(filename, 'wb') as file:
    joblib.dump(lightgbm_model, file)

It shows up:

TypeError: can't pickle generator objects

Anyone can help me with storing the best model that can be retrievedd and make predictions?

Thanks a lot.

[NicolasHug]

Can you please provide the entire code? Can you also try with a scikit-learn estimator instead of the LightGBM one?

Also, there's a data leak:

Pipeline([('scaler', StandardScaler()), ('model', GridSearchCV(...))

Should be

GridSearchCV(Pipeline([('scaler', StandardScaler()), ('model', lgbm_estimator)]))

In the first snippet, the CV done by grid-search takes as input the entire normalized data, and so the folds are not independent anymore. It seems that you're only doing model selection and not model evaluation so it might not be super important, but it's worth noting that this is in general incorrect.

[Erica-Ko]

Thank you for your reply. I have tried logistic regression with the following code:

def gen_logistic_regression(X_train, y_train, n_split=5):
    model = LogisticRegression(solver="liblinear")
    lr_model = Pipeline([('scaler', StandardScaler()), ('model', model)]) \n
    scores = cross_validate(lr_model, X_train, y_train, scoring=['accuracy', 'precision'], n_jobs=-1) \n
    lr_model.fit(X_train,y_train) \n
    return lr_model

I can pickle the resulted lr_model. It is the full code regarding the model. X_train is a df with 603 rows and 14 columns which are all numeric columns. y_train is the label that I would like to classify that contains 0 and 1. Metric is 'precision'. You are not clear with which part?

Thank you for pointing out the data leak problem. From my understanding, what you mean is the GridSearchCV inside the Pipeline already used up the whole dataset for input that's why the CV folds are not independent anymore. Am I correct?

[NicolasHug]

Can you please provide the entire code? Can you also try with a scikit-learn estimator instead of the LightGBM one?

I'm sorry but you haven't answered to these questions. We can't help you without the whole code as there might be issues in other places than what you're currently showing. In particular above lightgbm_model is undefined. Also, please confirm whether using a scikit-learn estimator instead of lgbm fails or not.

From my understanding, what you mean is the GridSearchCV inside the Pipeline already used up the whole dataset for input that's why the CV folds are not independent anymore. Am I correct?

I'm not entirely sure what you mean here but simply put, your code is equivalent to this:

X_train_scaled = StandardScaler().fit_transform(X_train)
GridSearchCV(lgbm_estimator, ...).fit(X_train_scaled, y_train)

The scaling part should be done within the cross-validation loop of GridSearchCV .

You may refer to https://scikit-learn.org/stable/common_pitfalls.html#data-leakage

[Erica-Ko]

Sorry for a bit late to come back. Thank you, Nicolas. After adopting the suggested way to prevent the data leakage problem. I can successfully pickle the GridSearchCV object. I know what you mean about data leakage. Thanks a lot for your help!