seconds/iteration is fast in first epoch, gets slower every subsequent epoch

[angadkalra]

I'm training a resnet101 3D model on grayscale images, 224x224x320, 16-bit precision, on VM with 4xV100 GPUs, using DDP and num_workers = 4. Batch size is 8 (2 per GPU). My first epoch goes very fast with 2.5s/it, but every epoch after that gets slower and slower and at epoch 6 I'm getting 6.5s/it. Any idea why this is happening or tips to speed up?

Thanks!

[ananthsub]

Do you have any data cached across epochs that you need to reset? Does the performance dramatically drop off at each epoch boundary, or does it gradually slow down during training execution?

[tchaton]

Dear @angadkalra,

Is your code open source and could be replicated, even with mocked data ?

Best,
T.C

[angadkalra]

No data that is cached across epochs. I'm using the metrics package to keep track of preds and targets for AUROC calculation but that gets reset at end of every epoch. Once the first epoch ends, the second one just starts very slow and gets to about 5s/it after 30-50 iterations, and then ends around the same value. Then every epoch after that is the same or slower.

@tchaton yes I can try and setup a colab for people to try...

I'm wondering in general if this is common or should the iteration speed be consistent across epochs?

[tchaton]

Yes, it should be consistent. A Colab would definitely be useful.

[complete-dope]

@tchaton Any updates on this ? what probably is causing this error ?
I am using pytorch lightning 1.9.1