[Bug] hydra.job.id cannot be used in hydra.sweep.dir

[YannDubs]

🐛 Bug

hydra.job.id cannot be used in hydra.sweep.dir but it can be used in hydra.sweep.subdir. The same holds for hydra.job.num.

To reproduce

config.yaml:

hydra:
  sweep:
    dir: ${hydra.runtime.cwd}/${hydra.job.id}
    subdir: ${hydra.job.num}

test.py:

import hydra
@hydra.main(config_name='config')
def main(cfg):
        pass
main()

Command python test.py -m

** Stack trace/error message **

omegaconf.errors.MissingMandatoryValue: Missing mandatory value : hydra.job.id
	full_key: hydra.sweep.dir
	reference_type=SweepDir
	object_type=SweepDir

System information

• Hydra Version : I tested both on 1.0.5 (stabel) and hydra-core-1.1.0.dev2 (pre)
• Python version : 3.8.5
• Operating system : Ubuntu

[YannDubs]

This is even worst when not sweeping actually:

config.yaml:

id : ${hydra:job.id}_${hydra:job.num}
hydra:
  run:
    dir: ${hydra.runtime.cwd}/out/${hydra.job.id}

test.py:

import hydra
import os
@hydra.main(config_name='config')
def main(cfg):
        print(cfg.id)
        print(os.getcwd())
main()

python test.py
# None_None
# /projects/test/out/???

As you see there are no error raised, but ${hydra.job.id} from hydra.run.dir is ??? while ${hydra:job.id} is None

[omry]

This is not a bug.
hydra.job.id only exists after the job is created. It's populated by the launcher itself.
The hydra.sweep.dir is created before the individual jobs.

In fact, it makes no sense at all if you think about it because sweeping on two jobs, with two different ids - you would end up with two different hydra.sweep.dir results even though it's created once when the sweep starts.

This is even worst when not sweeping actually

The Launcher and Sweeper are not used at all when you run without -m. Those variables are not populated outside of --multirun.

[YannDubs]

Thanks for both clarifications!

I knew that they were not populated without -m (or rather that they returned None), but I still think it's strange from a user perspective that it returns ??? or None depending on when they are called (although I now understand why this happens).

Also in the docs it might be worth mentioning that job.id is not used without -m, I'm not sure it's transparent from :

The job ID is populated by active Hydra launcher. For the basic launcher, the job ID is just a serial job number, but for other systems this could be the SLURM job ID or the AWS Instance ID.

For me a "serial job number" and "active hydra launcher" are not necessarily linked to a sweep.

Anyways I'm closing it as I see that job.id is just not meant to be used without sweeping.

[omry]

Yeah - I am considering changing the behavior to always use a the Launcher.
This is a serious change as it's user facing.

[YannDubs]

I think it would be more natural, although I understand that it directly impacts the user.

That being said, because it always returned Noneand couldn't be used in hydra.run.dir (as it returned ??? at that point) I'm not sure many people were using it without -m (so it won't negatively impact users). I was trying to, because I wanted to have a uniform code and directory structure regardless of -m.

[omry]

Two things here:

1. If Hydra is using your configured launcher, once you have an override for a launcher it would use it even if you don't use -m. this is a user facing behavior change.
2. Support disabling working directory management #910 will include some changes to the working directory management. One thing I want to do there is to have a single variable to hold the final output dir regardless of the execution mode.