The Trainer freeze when I use the logger

[xugaoqi1993]

Hello:
I am training my model with pytorch lightning. If I set the logger as False there are no issues. However, if i use the logger, the training processing will freeze and no error will be reported. Here are the screenprints when this problem occurs. The gpu memory will be occupied by about 1700mb, which is abount 23000mb in a normal training processing. It seems like the training process is stuck when the data is transported into gpu memory.
Is there any solutions to slove this problems?
Thank you

[CindyZJT]

I'm having the same issue too. Did you solve it ?

[CindyZJT]

I'm using TensorboardLogger.
I've followed step 1 to set the logger to None, and the issuse remains.

[jxtngx]

Okay. It is possibly something besides the logger then. Do you agree?

My next idea is to set callbacks and logger to None, so that we can start to isolate the problem without changing the distributed training flags.

[CindyZJT]

Thanks for the suggestion. This issuse still exists after setting callbacks and logger to None respectively.

[jxtngx]

can you please set devices to 1, strategy to "auto", and accelerator to "gpu".

[donglihe-hub]

Met the same issue here. The training hangs after setting logger=True. After about 12 mins the training process finally started.

One possible cause for the issue could be that the hyperparameters to be saved is too large. In my case, the hyperparameter file is 655 MB.

lightning ver.: 2.2.0.post0

    trainer = L.Trainer(
        logger=True,
        num_sanity_val_steps=0,
        accelerator="gpu",
        devices=1,
        enable_progress_bar=True,
        max_epochs=epochs,
        callbacks=callbacks,
        limit_train_batches=limit_train_batches,
        limit_val_batches=limit_val_batches,
        limit_test_batches=limit_test_batches,
        deterministic="warn",
        gradient_clip_val=0.5,
        gradient_clip_algorithm="value",
    )