Optimization history plot can't seem to converge or plateau

[adnan-umich]

Hello everyone, I have a pytorch segmentation model (Unet) hyper parameter search. The objective function has 3 outputs (the mean dice score, the average training loss, and the average validation loss). I'm running an Optuna based hyperparam optimization experiment aimed at maximizing mean DICE, while minimizing the other 2 parameters. Below is what my Optimization History plot looks like. Even after 200+ trials, it appears the optimizing is still having a hard time converging to a set of parameters ? I'm using the TPESampler sampler.

Please let me know If you need me to post any more information. Any advice is greatly appreciated. Thank you.

My hyper parameters are the following:

Optuna_settings.hyperparam.batch = discrete integer between [1, 5]

Optuna_settings.hyperparam.beta_1 = continuous float between [0.85, 0.999]

Optuna_settings.hyperparam.beta_2 = continuous float between[0.85, 0.999]

Optuna_settings.hyperparam.epoch = discrete integer between [200, 800]

Optuna_settings.hyperparam.learning_rate = continuous float between [0.00001, 0.0001]

Optuna_settings.hyperparam.loss = categorical variable ["DiceLoss", "DiceCELoss", "MaskedDiceLoss", "GeneralizedDiceLoss", "FocalLoss", "TverskyLoss"]

Optuna_settings.hyperparam.optimizer = categorical variable ["Adam", "AdamW"]

Optuna_settings.hyperparam.weight_decay = continuous float between [0, 0.000001]

[nzw0301]

The objective function has 3 outputs (the mean dice score, the average training loss, and the average validation loss).

If your objective function returns these three values, it is a multi-objective problem so the optimal solutions would have trade-off as described in https://en.wikipedia.org/wiki/Pareto_front; no single winner. I think you need to return the mean dice loss (on validation loss?) to make the optimisation converge instead of three values.

[nzw0301]

So you're saying I need to return a single metric instead of 3?

Yes.

But how will I minimize the two losses while also trying to maximize the mean dice score?

As explained by the Wikipedia page, it's often impossible. In addition, I think https://en.wikipedia.org/wiki/Overfitting will happen due to the existence of the average training loss and the average validation loss in the objectives, so I think it cannot achieve an optimal solution for all objectives with a single hyper-parameter config.

[adnan-umich]

I will try what you suggested. Thank you.
Why would Optuna allow an objective function to return more than one value, and why would it allow for multi-objective training?

[nzw0301]

Depends on the problem, multi objective is useful. For example, UNet's dice loss on a validation dataset and latency of UNet's feedforward computation and we tune the number of parameters, such as depth, conv's kernel size etc. In this case, using the minimal layer and small kernel size achieve the fastest latency, but it will be very high dice loss value because such small UNet has not enough expressiveness to minimise the loss function.

In your case, the training loss should be minimised in the training part of deep learning framework, not optuna's optimize.

[neel04]

@nzw0301 I'm doing pretty vanilla single objective optimization using TPE, with a pretty sizeable number of trials (~100) with pruning yet is still doesn't converge for me. My WandB plot feels like the sampling is happening randomly - the best runs are often just something randomly achieved and it never builds up on them.

[nzw0301]

Could you create a new post for your question with your objective?