New structure for clinicadl train

[14thibea]

As explained in #194 , we need to find a way to train a network from a JSON file (now we can only use TOML file).
First I proposed to change the option --config_file in train to accept both JSON and TOML files but it is not a good idea for two reasons:

1. JSON files already describe the mandatory arguments while TOML files don't... So they may not need the same command line
2. It may be confusing for the user to mix the two things.

As we want to separate these two commands, we need two different train commands. Moreover we other commands such as list_models #227 and resume could be also grouped and not be at the first level of the command line.
One first proposition would be use this structure:

• clinicadl
  • train
    • resume
    • list_models
    • from_json
    • from_toml / commandline?

The from_toml command is a bit misleading as the TOML file is not mandatory... Then any other solution is welcome!

[nburgos]

Could you describe in what situation(s) from_json would be used? Only when you do a random search or also for other situations?

[14thibea]

from_json would allow to reproduce the results obtained from a MAPS in a more concise way (no need of a TOML file with the options + the list of mandatory arguments). It is absolutely necessary for random search, but it may be useful in other ways.

[ravih18]

One other issue is that the clinicadl train is a command, and not a group of command. That means we cannot juste add from_json, list_models and resume as a clinicadl train subcommand without changing it.

An other solution would be to split the current train command with tasks, and see tasks as train sub commands instead of arguments. The clinicadl train command group would then have this structure:

• clinicadl
  • train
    • classification
    • regression
    • reconstruction
    • any new tasks such as segmentation...
    • resume
    • from_json
    • list_models
    • any new training feature

The advantages of this solution are:

• This doesn't change the current clinicadl train cli for the users, only the way it is implemented. clinicadl train classification CAPS_DIRECTORY TSV_DIRECTORY PREPROCESSING_JSON OUTPUT_MAPS_DIRECTORY will stay the same.
• All the commands related to training will be gathered in one group of commands.
• Splitting the different tasks allows us to separate the options that are specific to a task (they are not many but it is still better). For instance the options related to multi-class would be only available in classification sub command. It would also allow to give better insight of what are the default values (for instance directly show what is the default loss, default models...).

[nburgos]

Could a solution be something like

• clinicadl
  • train
    • classification
    • regression
    • reconstruction
    • any new tasks such as segmentation...
    • list_models
  • retrain
    • resume
    • from_json

?

The use-cases of the from_json command would need to be better specified I think, otherwise it's difficult to see what's the best solution to make it available. I understand for random searches, the "other ways" need to be made explicit.

[14thibea]

Or it could even be:

• clinicadl
  • train
    • classification
    • regression
    • reconstruction
    • any new tasks such as segmentation...
    • list_models
  • resume
  • retrain (from_json)

If we don't want to group resume and from_json

[ravih18]

@14thibea what do you mean by

the command line of train may become very long

Separate the train command line in two train and retrain may be a solution as well as keeping resume and retrain (from_json) separated from train.

However in my opinion it is better to keep all the commands related to a training job within the same group (a little bit like clinica run for all the pipelines).
For the resume command since it resumes a clinicadl train TASK command I think it is better to keep it in the same part of the tree.
Concerning the from_json, it does the same thing as clinicadl train TASK but with only one argument (instead of the list of arguments and options) so I would also keep it as a train sub command.

What do you think ?

Maybe we can also ask for @mdiazmel opinion ?

[omar-rifai]

My two cents: Having seperate sub-commands retrain (from json) and resume is much more readable to me.
IF we absolutely want to keep them at the same level as the classification regression etc.. commands. I think the name needs to be something completely different than just from_json as this just leads to confusion.

[mdiazmel]

Maybe we can also ask for @mdiazmel opinion ?

I like the idea of splitting clinicadl train TASK as subcommands instead of arguments. -config_file will remain an option of the clinicadl train command but -from_json will be and option for the subcommands inside the tasks. This solution seems clear for me.

I advocate for this solution because, in addition to Ravi's arguments, I prefer so keep the first level commands as simple as possible (even it that implies to complexify deeper-level command)

[14thibea]

It could also be interesting to change the names of the commands: for example from_json in reproduce or reuse_model.