Test step with DDP

[david-waterworth]

Having refactored my code to avoid iterable datasets I've now got DDP training working (I also had to set ulimit to prevent another crash). However now it crashes at the test step.

The message implies DDP is not needed for testing - but I don't see any mention in the documentation of how to disable DDP once training has complete (plus I would assume that trainer.test() would do this if it were required).

Is there something I should be doing different - this is my train / test code - the test dateloader has batchsize=1

    trainer.fit(transformer, datamodule=dm)
    transformer.freeze()

    # run tests
    result = trainer.test(transformer, datamodule=dm)

-- Process 0 terminated with the following error:
Traceback (most recent call last):
File "/home/david/.pyenv/versions/transformer/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/home/david/.pyenv/versions/transformer/lib/python3.8/site-packages/pytorch_lightning/accelerators/ddp_spawn_backend.py", line 152, in ddp_train
model = model.configure_ddp(model, device_ids)
File "/home/david/.pyenv/versions/transformer/lib/python3.8/site-packages/pytorch_lightning/core/lightning.py", line 837, in configure_ddp
model = LightningDistributedDataParallel(model, device_ids=device_ids, find_unused_parameters=True)
File "/home/david/.pyenv/versions/transformer/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 269, in init
assert any((p.requires_grad for p in module.parameters())), (
AssertionError: DistributedDataParallel is not needed when a module doesn't have any parameter that requires a gradient.

[david-waterworth]

So looks like transformer.freeze() is the issue - commented that out and it's fine. Is that a bug?

[awaelchli]

Does not look like a but to me.

The error seems to come from the underlying PyTorch DistributedDataParallel, not Lightning.
It seems we can't really do anything. But freezing the model before test should not be needed, test does not alter the weights and puts model into eval mode anyway. Makes sense?

[david-waterworth]

Yes that makes sense.