Get the same result every evaluate

[Sakurakdx]

When I finish training for evaluation, the results obtained in each evaluation are the same, but the parameters in the training process are changed.
When I set the training and evaluation data to be the same and the sample size is 1, the result of every train step is different, but the evaluation result are the same.
My validation_step() is follow:

[tchaton]

Dear @Sakurakdx,

Would you mind reproducing these results with the BoringModel you can find under pl_examples?

Furthermore, you might have shuffle=True for training data, which might explain the result are different for training.

[tchaton]

It seems you load only the decoder right?

[rohitgr7]

are you sure it's roberta-large? standalone roberta-large is 1.8 gigs and it has more params

roberta-base has around 125Mil params though.

[Sakurakdx]

sorry, it's my fault. the model is roberta-base. And I thought roberta-base 480Mil params. Thanks for your correction.

But I want to kown why the model parameters are changed, but the result of each evaluation is still the same.
I check my model, and set the dev data as same as train data. The outputs of train loop will change, but the ouputs of the evaluate loop is stiil the same.

[Sakurakdx]

are you sure it's roberta-large? standalone roberta-large is 1.8 gigs and it has more params

roberta-base has around 125Mil params though.

Strangely, I didn’t change anything, and ran the code again and again, and the bug disappeared by itself.
Before that, I ran it many times, but the bug persisted. It shocked me a lot.
Anyway, thanks for your respons.

[rohitgr7]

glad its working for you now.. ping us if the issue comes up again :)