Provide an option to reset metrics between multiple dataloaders?

[tshu-w]

currently metrics only reset between different stage: https://github.com/PyTorchLightning/pytorch-lightning/blob/dd8216a6b80fcabea0ee76f862f02f3d30bb3f53/pytorch_lightning/trainer/connectors/logger_connector/logger_connector.py#L263-L269 but I want to get seperate result for different dataloaders.

  "test_f1/dataloader_idx_1": 0.7213717103004456,
  "test_prc/dataloader_idx_1": 0.59375,
  "test_rec/dataloader_idx_1": 0.9188767671585083,
  "test_loss/dataloader_idx_1": 0.27367517352104187,
  "test_f1/dataloader_idx_0": 0.7213717103004456,
  "test_prc/dataloader_idx_0": 0.59375,
  "test_rec/dataloader_idx_0": 0.9188767671585083,
  "test_loss/dataloader_idx_0": 0.3688618838787079,

[tchaton]

Dear @tshu-w,

I would recommend you to use 1 set of metrics per dataloaders which is definitely safer.

Best,
T.C

[tshu-w]

Let me explain my scenario further, my model is not sure how many dataloaders datamodule has (one module for several datamodule), so I only init metric for the train, valid, and test. I have the following code and I want to get different dataloaders results seperately. What's the better way to do this? Why 1 set of metrics per dataloaders is safer.

        self.metrics(probs, labels)
        self.log_dict(self.metrics, prog_bar=True)

[rohitgr7]

I think what @tchaton meant here is to use a different set of metric instances only if you are using modular torchmetrics because they contain certain internal states for accumulation at the end. Mixing the states from different dataloader might mess with the internal states.

add_dataloader_idx is used to separate out the metrics at the end in case they are logged with the same key. If you have a different key for each dataloader, then you can set it to False.

[tshu-w]

@rohitgr7 Yes, I'm using modular torchmetrics but I'm not sure about the internal states, won't reset clear all of them?

[rohitgr7]

internal states are basically used to compute the metrics on the complete dataset at the end. For eg in the case of accuracy, it stores true-positive, true-negative..etc. required to calculate the accumulated accuracy at the end. And yes, it will reset, but reset is called automatically at the epoch end. But if you use the same instances for both the dataloaders, then the metric will update these internal states using the data from both the dataloaders at the step level because the current flow for the hook is

dataloader_idx0, step_hook, batch 0
dataloader_idx0, step_hook, batch 1
...
dataloader_idx1, step_hook, batch 0
dataloader_idx1, step_hook, batch 1
...

epoch end

[tshu-w]

@rohitgr7 Yes, What I hope is we can add a option to resets metrics between dataloader