You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
It looks like GaLore is a drop-in replacement for the PyTorch optimizer, meaning that you can take any script that does pretraining/fine-tuning and replacement the part that defines optimizer.
It looks like GaLore is a drop-in replacement for the PyTorch optimizer, meaning that you can take any script that does pretraining/fine-tuning and replacement the part that defines optimizer.
Can u support fsdp + qlora soon , i stuck oom when max_seq_length large when fine-tune lora > 24gb with model 7b
Can repo support GaLore soon ?
https://github.com/jiaweizzhao/GaLore
The text was updated successfully, but these errors were encountered: