Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow users to pass extra options to the job submission command in the partition config #2970

Open
vkarak opened this issue Aug 23, 2023 · 4 comments · May be fixed by #3156
Open

Allow users to pass extra options to the job submission command in the partition config #2970

vkarak opened this issue Aug 23, 2023 · 4 comments · May be fixed by #3156

Comments

@vkarak
Copy link
Contributor

vkarak commented Aug 23, 2023

This is requested by EESSI/test-suite#68 as well as #2968. #2968 introduces a solution that is LSF-specific (in terms of naming), but the idea is to generalise it to the rest of job schedulers.

@boegel @casparvl Would that work in your case?

@vkarak vkarak added this to the ReFrame 4.4 milestone Aug 23, 2023
@vkarak vkarak modified the milestones: ReFrame 4.4, ReFrame 4.5 Oct 4, 2023
@vkarak vkarak removed this from the ReFrame 4.5 milestone Dec 12, 2023
@boegel
Copy link
Contributor

boegel commented Jan 18, 2024

@vkarak Sorry for not getting back to this earlier....

Yes, a way to pass additional options to the sbatch command should help (at least, as long as options like --partition override environment variables like $SBATCH_PARTITION, which I guess is the case, I didn't check)

@laraPPr
Copy link

laraPPr commented Feb 1, 2024

I was finally able to test it and passing --partition in the sbatch command works on hortense.

@laraPPr
Copy link

laraPPr commented Feb 1, 2024

To give a little more context I will clarify some parts of the setup of hortense and how I tested that passing --partition in the sbatch command would resolve our current issue.

The setup of Hortense:
To help our users we set the environments of the partitions in sticky modules. So that when users want to submit jobs to a specific partition they just have to swap to the right sticky module corresponding with the partition the user wants to submit to.

With ReFrame however we want to be able to submit to all partitions and not just the partition which is set in the environment.

We tried first to set the partition in the config file of hortense. However here the sbatch option is overruled by the environment that is set.

'access': hortense_access + ['--partition=cpu_rome'],

An other option was to see if passing the sbatch option in the command rather than the job script would overrule the environment. To test this I ran a jobscript that was made by a previous run of the ReFrame test suite. The goal was to submit the job to the cpu_milan partition while the cpu_rome environment is set.

The set environment during the test:

Currently Loaded Modules:
  1) env/vsc/dodrio/cpu_rome (S)   2) env/slurm/dodrio/cpu_rome (S)   3) env/software/dodrio/cpu_rome (S)   4) cluster/dodrio/cpu_rome (S)

the sbatch command:

sbatch --partition cpu_milan rfm_job.sh

With the following sbatch command the rfm_job.sh was correctly submitted to the cpu_milan partition.

@vkarak
Copy link
Contributor Author

vkarak commented Feb 7, 2024

Yes, indeed, Slurm precedence is command-line > environment variable > job script. Another possibility for reframe to support this would be to add a configuration parameter in sched_options to instruct reframe to pass the access options in command line instead of the script. This is probably a cleaner solution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment