Control automatic resubmission on SLURM

[awaelchli]

What does this PR do?

Fixes #6389

Adds the ability to turn on or off the automatic resubmission that Lightning does when a job gets interrupted by the SLURM controller (via signal handling). Users who prefer to let libraries handle the resubmission (e.g. submitit) can now set

trainer = Trainer(plugins=SLURMEnvironment(auto_requeue=False))

Idea from @ananthsub.

Whether the the default for auto_requeue is True or False is currently being debated on the linked issue. For now, the PR implements the behavior we have already (auto_requeue=True by default).

Before submitting

• Was this discussed/approved via a GitHub issue? (not for typos and docs)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure your PR does only one thing, instead of bundling different changes together?
• Did you make sure to update the documentation with your changes? (if necessary)
• Did you write any new necessary tests? (not for typos and docs)
• Did you verify new and existing tests pass locally with your changes?
• Did you update the CHANGELOG? (not for typos, docs, test updates, or internal minor changes/refactorings)

PR review

Anyone in the community is free to review the PR once the tests have passed.
Before you start reviewing make sure you have read Review guidelines. In short, see the following bullet-list:

• Is this pull request ready for review? (if not, please submit in draft mode)
• Check that all items from Before submitting are resolved
• Make sure the title is self-explanatory and the description concisely explains the PR
• Add labels and milestones (and optionally projects) to the PR so it can be classified

Did you have fun?

I made sure I had fun coding 🙃

Part of #1 (it's a lie, this is just here to avoid noisy GitHub bot)

cc @Borda @awaelchli

[tchaton]

LGTM !

[ananthsub]

Thanks for adding this!

[marksibrahim]

Thank you for implementing this!

[marksibrahim]

@awaelchli Do you know which release this is expected to be a part of? I tried installing the latest 1.5.3 release (from 2 hours ago) and it doesn't seem to include the new SLURM auto_requeue argument.

[carmocca]

@marksibrahim Lightning 1.6, so a bit far a way. You will have to install directly from the GitHub repository in the meantime.