Different outputs from Optuna best trial value and classifier output trained with best hyperparameters

[mohammedsabiya]

Datasplit:

data = normalized_df_ros.to_numpy()
target = y_train_oversample

train_x, valid_x, train_y, valid_y = train_test_split(data, target, test_size=0.25, random_state = 52)

I have used the code below for Optuna hyperparameter tuning:

def objective_ros(trial):
  dtrain = xgb.DMatrix(train_x, label=train_y)
  dvalid = xgb.DMatrix(valid_x, label=valid_y)

  param = {
        "objective": "binary:logistic",
        "booster": trial.suggest_categorical("booster", ["gbtree", "gblinear", "dart"]),
        "lambda": trial.suggest_loguniform("lambda", 1e-8, 1.0),
        "learning_rate": trial.suggest_loguniform("alpha", 1e-8, 1.0),
        "subsample": trial.suggest_float("subsample", 0.05, 1.0),
  }

  if param["booster"] == "gbtree" or param["booster"] == "dart":
    param["max_depth"] = trial.suggest_int("max_depth", 1, 9)
    param["eta"] = trial.suggest_loguniform("eta", 1e-8, 1.0)
    param["gamma"] = trial.suggest_loguniform("gamma", 1e-8, 1.0)
    param["grow_policy"] = trial.suggest_categorical("grow_policy", ["depthwise", "lossguide"])
  if param["booster"] == "dart":
    param["sample_type"] = trial.suggest_categorical("sample_type", ["uniform", "weighted"])
    param["normalize_type"] = trial.suggest_categorical("normalize_type", ["tree", "forest"])
    param["rate_drop"] = trial.suggest_loguniform("rate_drop", 1e-8, 1.0)
    param["skip_drop"] = trial.suggest_loguniform("skip_drop", 1e-8, 1.0)

  bst = xgb.XGBClassifier(**param, random_state = 52)
  bst.fit(valid_x, valid_y)
  preds = bst.predict(valid_x)

  f1 =  f1_score(valid_y, preds, average='micro')
  return f1

if __name__ == "__main__":
    """
      The optuna creates the study to optimize the objective function with 100 trials and return the best trial
      that has the maximum F1-Score and the best hyperparameters of that trial

    """

    study = optuna.create_study(
        pruner=optuna.pruners.MedianPruner(n_warmup_steps=5), direction="maximize"
    )
    study.optimize(objective_ros, n_trials=3)
    print(study.best_trial)

    print("Number of finished trials: {}".format(len(study.trials)))

    print("Best trial:")
    trial = study.best_trial

    print("  F1 Score: {}".format(trial.value))

    print(" Best Hypeparameters: ")
    for key, value in trial.params.items():
        print("    {}: {}".format(key, value))

Output:

FrozenTrial(number=2, state=TrialState.COMPLETE, values=[0.9856801909307876], datetime_start=datetime.datetime(2024, 5, 8, 15, 35, 16, 222522), datetime_complete=datetime.datetime(2024, 5, 8, 15, 36, 2, 629269), params={'booster': 'dart', 'lambda': 0.002311280531979015, 'alpha': 0.0002916755277581915, 'subsample': 0.9746150609390157, 'max_depth': 6, 'eta': 1.2123850647079977e-05, 'gamma': 7.277285195329304e-05, 'grow_policy': 'lossguide', 'sample_type': 'weighted', 'normalize_type': 'tree', 'rate_drop': 0.0010080385982105485, 'skip_drop': 0.002532401998679777}, user_attrs={}, system_attrs={}, intermediate_values={}, distributions={'booster': CategoricalDistribution(choices=('gbtree', 'gblinear', 'dart')), 'lambda': FloatDistribution(high=1.0, log=True, low=1e-08, step=None), 'alpha': FloatDistribution(high=1.0, log=True, low=1e-08, step=None), 'subsample': FloatDistribution(high=1.0, log=False, low=0.05, step=None), 'max_depth': IntDistribution(high=9, log=False, low=1, step=1), 'eta': FloatDistribution(high=1.0, log=True, low=1e-08, step=None), 'gamma': FloatDistribution(high=1.0, log=True, low=1e-08, step=None), 'grow_policy': CategoricalDistribution(choices=('depthwise', 'lossguide')), 'sample_type': CategoricalDistribution(choices=('uniform', 'weighted')), 'normalize_type': CategoricalDistribution(choices=('tree', 'forest')), 'rate_drop': FloatDistribution(high=1.0, log=True, low=1e-08, step=None), 'skip_drop': FloatDistribution(high=1.0, log=True, low=1e-08, step=None)}, trial_id=2, value=None)
Number of finished trials: 3
Best trial:
  F1 Score: 0.9856801909307876
 Best Hypeparameters: 
    booster: dart
    lambda: 0.002311280531979015
    alpha: 0.0002916755277581915
    subsample: 0.9746150609390157
    max_depth: 6
    eta: 1.2123850647079977e-05
    gamma: 7.277285195329304e-05
    grow_policy: lossguide
    sample_type: weighted
    normalize_type: tree
    rate_drop: 0.0010080385982105485
    skip_drop: 0.002532401998679777

Using the best hyperparameters from Optuna, I re-trianed the 'XGBClassifier'

Code:

clf = xgb.XGBClassifier(**best_params, random_state = 52)
clf.fit(train_x, train_y)
valid_pred_ros = clf.predict(valid_x)

f1 =  f1_score(valid_y, valid_pred_ros, average='micro')
print("validation f1_score : ", f1)

Output:

validation f1_score : 0.937947494033413

Now, you can see the best value from Optuna is F1 Score: 0.9856801909307876, whereas the validation F1-score that is trained with the same hyperparameters and same XGBclassifier is validation f1_score : 0.937947494033413.

my question is why the values are different here though the parameters are the same?

[nzw0301]

Could you share the minimal reproducible code with us?

[mohammedsabiya]

Hi Kento, I have added the Python file and the required data to the following drive link. https://drive.google.com/drive/folders/1wfj2dYV2MWwdER1sCuuTAHQ39QeE3IhT?usp=sharing Thank you!

…

On Thu, 9 May 2024 at 04:47, Kento Nozawa ***@***.***> wrote: Could you share the minimal reproducible code with us? — Reply to this email directly, view it on GitHub <#5437 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEBGRQBBZ2TOOGBQVEAENFDZBLPUJAVCNFSM6AAAAABHNI2HO2VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TGNRSGUYTE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- Best Regards, Mohammed Sabiya Sujith Ahamed

[nzw0301]

Lovely. I'll check it in a few days.

[nzw0301]

Hi, I think this line should use the same name between dict's key and optuna's param name. This inconsistency looks the root cause of the issue.

        "learning_rate": trial.suggest_loguniform("alpha", 1e-8, 1.0),

In addition, objective and eval_metric are only specified in the objective and not in the call of the last fitting with best params.

[mohammedsabiya]

Yes, that makes sense. The problem is fixed now. Thanks, man!

…

On Thu, 9 May 2024 at 17:18, Kento Nozawa ***@***.***> wrote: Hi, I think this line should use the same name between dict's key and optuna's param name. This inconsistency looks the root caues. "learning_rate": trial.suggest_loguniform("alpha", 1e-8, 1.0), In addition, objective and eval_metric are only specified in the objective and not in the call of the last fitting with best params. — Reply to this email directly, view it on GitHub <#5437 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEBGRQAKJPKPU4U3MUB7JE3ZBOHTHAVCNFSM6AAAAABHNI2HO2VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TGNZQHEZDQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- Best Regards, Mohammed Sabiya Sujith Ahamed