How to manually adjust learning rate -- module.optimizers vs trainer.optimizers?

[daniel-layton-wright]

I'm having trouble manually adjusting the learning rate in my module when restoring from a checkpoint. The context is that my first epoch I use as sort of a 'dry run' test to make sure everything is working, e.g., that .backward works on the given hardware etc. But I don't want to actually update the model weights with this dummy data. I tried adding this in my on_fit_start

def on_fit_start(self):

    # Set learning rate to 0 temporarily
    for param_group in self.optimizers().param_groups:
        print('Setting lr to 0')
        self.lr_save_tmp.append(param_group['lr'])
        param_group['lr'] = 0.0

But it doesn't work. There's no error thrown or anything, but I can see that the loss is actually consistently improving.

But it works when starting fresh, not from a checkpoint.

I figured out that the trainer.optimizers and the module.optimizers do not match when loading a checkpoint. What is the reason for that?

However, even when I change the lr in trainer.optimizers, it still does not work.

Where is the learning rate that is actually used in optimization steps?

[daniel-layton-wright]

I figured out that the trainer optimizers are reloaded after the module's on_fit_start is called