Turn off hpc checkpoint saving in SLURM environment if trainer.fit(..., ckpt_path="last")

[rustamzh]

Description & Motivation

Hi all,
trainer.fit(ckpt_path="last") works perfectly now (in SLURM environment with autorequeue=True), especially if you set up end-of-epoch checkpoint callback. However, the trainer still saves middle-of-the-epoch checkpoints that remain unused because of trainer.fit(ckpt_path="last"). It would be great to not save hpc_ckpt if trainer.fit(ckpt_path="last").

This feature will close #13773

Pitch

No response

Alternatives

No response

Additional context

No response

cc @Borda

[murnanedaniel]

@rustamzh This is a very good idea, and should be implemented.
However, I want to understand exactly your setup, as I'm trying to implement this myself. You specify a ModelCheckpoint with dirpath equal to some artifact_directory, I assume? And then you also set the Trainer's default_root_dir to that same artifact_directory?

My issue here is that if I am training two models, using the same artifact_directory, won't the auto-resume get them mixed up when it uses ckpt_path="last"?

[rustamzh]

Hi. My dirpath is empty in the ModelCheckpoint callback, this way checkpoints are created in their own version-X (where X is SLURM jobID) subfolder and do not mix, since the autorequeued job retains its jobID.

[murnanedaniel]

Hi. My dirpath is empty in the ModelCheckpoint callback, this way checkpoints are created in their own version-X (where X is SLURM jobID) subfolder and do not mix, since the autorequeued job retains its jobID.

This makes a lot of sense, although I'm missing the way that your default root dir is configured. Would it be possible to share the code snippet that is most relevant to this configuration? It would be much appreciated!

[rustamzh]

default root dir is just non empty.