How to cast model layernorms to fp32 when using precision="bf16-true"? #19775
Unanswered
eric-tc-wong
asked this question in
Lightning Trainer API: Trainer, LightningModule, LightningDataModule
Replies: 1 comment
-
My current solution is to override the convert_module function in the HalfPrecision plugin. Although, I still see a large drop in the performance of the model compared to bf16-mixed. Please let me know if this is not the proper solution.
|
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
What is the proper way to cast certain layers in a model wrapped in LightningModule to float when using Trainer(precision='bf16-true')?
I am working with transformer models and the LayerNorms need to be in float. I thought this is common, but I find it hard to find documentation or examples.
I try casting them during setup, but they don't hold during fit_loop. I also tried casting in configure_optimizers, but then I get error on backward.:
Thanks
Beta Was this translation helpful? Give feedback.
All reactions