Add a is_distrubted property to Trainer or LightningModule.

[richarddwang]

I am writing code that can run whether data parallelism is used,
without changing any single character of code.

So I have to write cross entropy loss logic under, instead of training_step, training_step_end,
whose input is just output of training_step,
or outputs from all devices when data parallelism is used.

Something like this will be very helpful then.

def xxx_step_end(self, all_device_outputs):
    if self.is_distributed # or self.trainer.is_distributed:
        result = concat_all(all_device_outputs)
    else:
        result = all_device_outputs # actually output of the only device, no need to concatencate

Note: We can't use isinstance(all_device_outputs, list) to check whether it is distributed. Considering output of xxx_step is just a list and dp is not used, it will be mistaken as dp used.

If it is ok, I would be happy to make a pr.

[SkafteNicki]

We already have such an attribute, however it is not that well exposed to the user:
trainer.accelerator_connector.is_distributed
Relevant code:
https://github.com/PyTorchLightning/pytorch-lightning/blob/832a03af7cae21bd4e6b7f752689a7ac6c2d4ce3/pytorch_lightning/trainer/connectors/accelerator_connector.py#L278-L287
Woudl that work for you @richarddwang ?

[richarddwang]

Hi @SkafteNicki
Actually I am using the mentioned attribute to do my work.
But an easy to find and short property would be good.

Also I have a question, will is_distributed be true when only model but not data parallelism used ?