About the Weight Initialization in PL

[magic282]

Hi,

I am tring to use BERT for a project. The pretrained BERT model is part of my model. I am wondering how will PL initialize the model weights. Will it overwrite the pretrained BERT weights?

Thanks.

[williamFalcon]

lightning doesn’t do any magic like this under the hood. you control all the weights and what gets initiated

[magic282]

I see. So where should I do the weight initialization step if I want to follow the PL design idea? In the __init__ of pl.LightningModule?

[williamFalcon]

This is up to you and you should follow standard PyTorch guidelines.
Normally it's done in __init__

[fingoldo]

Hi, I'm trying to add different weights initialization methods to the HPT procedure. Where is weights initialization currently happening in Lightning and how to override it? Or Lightning defers it to pytorch?