Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Trainer] Fix nan-loss condition #13911

Merged
merged 1 commit into from
Oct 6, 2021
Merged

[Trainer] Fix nan-loss condition #13911

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
11 changes: 7 additions & 4 deletions src/transformers/trainer.py
View file
Open in desktop
Expand Up @@ -1311,10 +1311,13 @@ def train(
else:
tr_loss_step = self.training_step(model, inputs)

if args.logging_nan_inf_filter and not is_torch_tpu_available():
if torch.isnan(tr_loss_step) or torch.isinf(tr_loss_step):
# if loss is nan or inf simply add the average of previous logged losses
tr_loss += tr_loss / (1 + self.state.global_step - self._globalstep_last_logged)
if (
args.logging_nan_inf_filter
and not is_torch_tpu_available()
and (torch.isnan(tr_loss_step) or torch.isinf(tr_loss_step))
):
# if loss is nan or inf simply add the average of previous logged losses
tr_loss += tr_loss / (1 + self.state.global_step - self._globalstep_last_logged)
else:
tr_loss += tr_loss_step

Expand Down