Designing for layered configs

[epage]

This is inspired by #2759 in which there is a proposal for one approach for layering. Thought the actual layering best practices would best be split out in a discussion

[epage]

#2759's approach

Use case: I would like to merge ArgMatches into Config settings using the following rules:

1. ArgMatches have higher precedence over Config if they are explicitly set.
2. If config setting is present but ArgMatches has an argument with default value then config value is taken.
3. If config does not set the setting, then ArgMatches default value it taken.

[gibfahn]

From my experience with layered configs, you have some Args not in the Config and some Config not exposed as Args with each requiring defaulting of some sort.

You're probably right, but it feels like using serde to deserialise a config file into args, and then updating from std::env::args() would be quite neat. However it's worth noting this doesn't work with subcommands (at least with the derive API), as you end up with a SubCommand enum rather than a SubCommand struct, e.g. something like:

#[derive(Parser)]
struct Args {
    #[clap(subcommand)]
    cmd: Option<SubCommand>,
}

#[derive(Parser)]
pub enum SubCommand {
  // ...
}

but for a config file parse, you'd want something like:

#[derive(Deserialize)]
pub struct SubCommand {
  // ...
}

[epage]

Could you expand on why an enum vs a struct makes a difference?

[gibfahn]

Basically when you run a command like:

my-app subcommand-one --flag
my-app subcommand-two --other_flag

Each invocation only passes one subcommand, so you want enum SubCommand not struct SubCommand.

However in a config file, you would want to be able to set defaults for all subcommands, not just one, so the config file might look like this:

global:
  color: always

subcommand_one:
  flag: some-value

subcommand_two:
  other_flag: other-value

Which might deserialise to:

#[derive(Deserialize)]
struct Config {
  global: GlobalOpts,
  #[serde(flatten)]
  subcommand: SubCommand,
}

#[derive(Deserialize)]
struct SubCommand {
  // ...
}

So two bits that I ran into trying to make a config file work were:

1. The global options will contain a SubCommand enum (which can be hidden/ignored so not a big issue).
2. I had to manually create a subcommand struct from the subcommand enum I already used in clap.

Will see if I can push up a full worked example.

[epage]

Good point.

Seems like this could be handled by having all arguments for your subcommand in a separate struct. For the CLI, you then integrate them with an enum. For your config, you integrate them with a struct. This does require manually connecting the pieces though.

[gibfahn]

Yep, that's what I ended up doing, but duplicating the fields and docstrings seems painful, and like something a macro of some kind could do for you.

[epage]

At one point, I would use a

• trait ConfigSource with a method per field that returned an Option<RefType>.
• The Config and Args would both implement this.
• Config would have an update(other: &dyn ConfigSource) for merging
• Config would also have methods per field that return RefType, defaulting the value if it is None.

Example from cargo-release

[epage]

As I ended up with child-structs and things, the boiler plate for this became too much and I wasn't seeing value out of using the trait. I moved onto a concrete Config. See below.

[epage]

Recently, I've been using a struct Config

• Every field is Option<Owned>
• Config has a update(other: &Config)
• Config has a method per field that returns RefType, defaulting the value if it is None
• Args has a to_config

Example from cargo-release

[romange]

Agree with me that while this approach has most control over the layering logic, it's pretty verbose. Currently I also use clap-derived configs with Option<> only fields. The problem with this approach is that I have to fill in default values myself via 3rd layer (after arguments and config files), but then clap help message does not provide information about default values - which I think is suboptimal.

[epage]

Mentioned in another thread but for --help output with clap, #1695 could help.

[epage]

There is also the popular config crate which is designed around layered config

[epage]

There isn't clap integration atm

[epage]

The big reason why I've avoided this is it looks like it delays all error reporting until after merge, not letting you tell users which config source was the problem.

[romange]

Exactly, I used config crate for merging. I think using Option<> for everything is a hack that should be fixed.

struct ServerOpts {
    #[clap(short, long)]
    port: Option<u16>,

    #[clap(long)]
    bind: Option<IpAddr>,
    // Configuration toml file.
    #[clap(long, value_hint = ValueHint::FilePath)]
    config: Option<PathBuf>,
}

impl ServerOpts {
    fn default_config() -> Result<Config, ConfigError> {
        let mut settings = Config::new();

        settings
            .set_default("port", 8080)?
            .set_default("bind", "127.0.0.1")?;

        Ok(settings)
    }

    pub fn new() -> Result<Self, Report> {
        // Just to get a custom config path
        let app = Self::into_app();
        let matches = app.get_matches();

        let mut settings = Self::default_config()?;

        // Priority is: config -> environment -> runtime args.
        if let Ok(config) = matches.value_of_t::<PathBuf>("config") {
            settings.merge(File::from(config))?;
            debug!("file args {:?}", settings);
        }

        settings.merge(Environment::with_prefix("THANKFORCLAP"))?;

        // Merge runtime arguments into settings by converting them to toml first and then merging toml string.
        let runtime_args = <Self as FromArgMatches>::from_arg_matches(&matches)
            .expect("could not parse arguments");

        let toml = toml::to_string(&runtime_args).expect("could not convert to toml");
        settings.merge(File::from_str(&toml, FileFormat::Toml))?;

        let final_args = settings.try_into::<Self>()?;

        Ok(final_args)
    }
}

[epage]

Could you clarify what you find is hacky about Option, rather than me guessing (probably incorrectly) which part of that code sample you don't like?

[epage]

For --help output with clap, #1695 could help.

[pksunkara]

Maintainer's notes:

• Allow adding "external argv"s to be parsed alongside/before command line #748 alludes to a couple of different pieces:
  • A File parser driven by clap::App
  • Turn that config file into argv and merge it with existing argv
  • Or turn that config into an ArgMatches and allow merging ArgMatches

---

Note: There's a lot of previous discussion on #748. Let's continue this discussion there?

[epage]

I looked through that and it seemed to mostly be focused on one approach (external argv) while this is discussing the the broader problem.

[epage]

As with #2759, my concern with an config parser being driven by clap::App is there are three types of config

• Those sources from all sources
• Those sourced from CLI only
• Those sourced from command line only

Having an "easy route" that focuses on this will help encourage people to put everything in both the config and CLI when it shouldn't all be there. This also limits the type of config you can store.

I will say someone can implement this on top of clap and give it a try to see how well it works.

[epage]

The problem with merging argv is order can matter, whether with subcommands or with other less common cases like #2419. This becomes a solution that is severely limited for who it can help and could encourage people to not design their CLI like they should to get an easy path for config layering.

[epage]

clap3's Parser trait has something like ArgMatches merging. It only supports overwriting but not any kind of merging or conflict. It'd be interesting to see people give it a try to see how well it works.

From my own experience, whether the overwrite, merge, etc is usually more field dependency. Its hard to have a one-size-fits-all policy at the top level.

Also, until we have #2683, we'll be severely limited in what config can be stored inside of an ArgMatches.

[epage]

For completeness, I'm going to mention argfile support. The formats currently supported by the argfile crate do not support comments and you can only specify it on the command line, you can't load it yourself to extend argv. This does have the benefit of allowing the user to control where the args are inserted.

[epage]

Instead of a solution, this thread will be exploring requirements and their prioritization

Various requirements include:

• Merge from args, env, and undefined set of files
  • Some files will be on the command line, some will be implicitly discovered, sometimes a user will pass a --isolated flag to disable implicit discovery
  • The merge strategy is field specific (replace, append, conflict, etc)
  • Tell users what layer an error came from
  • Handling schema migration with layering is complicated
• Some config is CLI, env, or file specific while some is shared
• We want defaults to show up in --help
• Users tend to want a --dump-config for both debugging and as a jumping off point
• Config file format is application-specific
  • TOML is frequently used in Rust. Some will want YAML or a json variant. For git users, you will want to integrate in with .gitconfig and git's env variables.

What additional requirements are there? Is there a viable subset that covers enough people for a shared solution? What building blocks can be made to help with all of this?

[epage]

Hidden: yes, not present in help.

What is the use case for hidden? Are you thinking in terms of renaming flags and keeping compatibility?

[nyurik]

Hidden: yes, not present in help.

What is the use case for hidden? Are you thinking in terms of renaming flags and keeping compatibility?

@epage I would think a case where config file / env var has something not settable via CLI args would be hidden, but would still make it into the primary args struct.

[epage]

That sounds like a specific solution to this requirement:

Some config is CLI, env, or file specific while some is shared

[nyurik]

correct. Anything hidden is basically something unused/unsettable by the CLI. I guess there could be some undocumented/unpublished params too that should work but shouldn't show up in the help, but they wouldn't be hidden I think (i.e. for backwards compat support?). Still hacky though.

[nyurik]

Another aspect to consider is extendability/modularity: some applications may have git-like structure, where depending on the top-level command, there are different sub-options. I see two general use cases:

• global config files work the same regardless of the top level command, and the app chooses which part of the config to use
• top-level commands can have different config files, possibly required for the command to proceed, e.g. app copy <src.config> <dst.config>

[jaskij]

Problems with early error reporting:

1. If an option is required, the program often doesn't care where it comes from - so it is actually impossible to check for missing options until after the merge.
2. If the merge strategy is replace, does it matter that an invalid option was overwritten?

[epage]

If an option is required, the program often doesn't care where it comes from - so it is actually impossible to check for missing options until after the merge.

Yes, if you have required config, you can't check that until the end. However, in my layering so far, the only required data is taken on the command line and is not layered with anything else. It'd be interesting to explore examples otherwise.

If the merge strategy is replace, does it matter that an invalid option was overwritten?

It represents a latent bug, unless there is an intentional reason you are overlaying on top of that bug. I can invent situations for that but unsure how likely they are to occur.

[nyurik]

Apparently there are a few more options, but neither support Derive:

• viperus - a bit dated (last release 2years ago, last modified half a year ago). Supports Clap. Lots of file formats - JSON, TOML, YAML, dotenv file ,java properties config files. Supports auto-reloading (config watching) (2k downloads)
  • cannot use Clap3 until Iterating over ArgMatches #1206 is fixed.
• clap_conf - popular (16k downloads). Supports ENV, Clap, Toml.
• twelf supports layering and Clap Derive, a bit buggy, under active development.

[virtualritz]

Also the relatively recent twelf crate has layering & clap support (incl. Derive – but it needs work).

[nyurik]

thx @virtualritz, added to the list. BTW, I suspect the Args::command().ignore_errors(true).get_matches() trick is what is missing in the other ones to support Derive -- i'll hack on it a bit to see if i can make clap_conf or viperus work with it.

[epage]

Looking for feedback on the upcoming clap API design, in particular from a layered config perspective but also if people have thoughts in general

See #3792

[jaskij]

I basically don't use the builder API, so no help from me here, sadly.

One thing I've been looking forward to is validators which are run after parsing - so I can, for example, check if a number is in the valid range. Preferably without parsing the string twice. Right now I'm working around this by using a separate function which is called after clap.

[epage]

Depending on what you mean by "after parsing", the new API allows

    #[clap(value_parser = clap::value_parser!(u16).range(1..))]
    port: u16

• The string will only be parsed once
• For something outside of 1..=u16::MAX, the user will be given an error saying it needs to be within that range.

Granted, if the range is dependent on runtime factors, like a config file, this won't work. At least we will only parse the string once.

[jaskij]

By "after parsing" I mean that the validator gets passed a typed value, not a string.

Range on numbers is nice, works for most if not all of my needs along the other existing validators. But would be nice to be able to pass a function too.

[luketpeterson]

After a 4-day diversion into a config system that involved becoming very familiar with the code for the twelf crate (some might call it a yak-shave), I have arrived at a fairly clear vision for the design of a a config layering feature that I personally would like to use.

First a few observations:

• Errors from the parsing of the ArgMatches (get_matches) ideally wouldn’t be thrown all the way to the user unless those fields can’t be assigned values from one of the other layers.

Optional arguments aren’t enough to accomplish this however, because the user still needs to see the error if no layer was ultimately able to supply the argument value.

• I don’t recommend the way twelf has a dependency on a large number of format crates (e.g. toml, json, etc. etc.) because it leads to a cargo features mess, not to mention the lack of clean support for formats not included. I think a better option is to just allow a layer to invoke a generic closure that returns an object that implements Deserialize (or DeserializeOwned).

Most serde format crates have one or two line samples that provide an interface like this.

• I don’t know whether it’s better to force the user to implement Deserialize on the same struct they implement the Clap traits on, or whether it’s better to auto-generate a parallel struct that is the Deserialize cousin of the Clap struct from which the Command is created.

There are pros and cons to each. On the one hand, having to re-specify so many field attributes for Clap and Serde is a pain, and there is no struct if the Command builder API was used instead. But on the other hand, allowing the user to derive Deserialize independently gives a lot more control / flexibility. It sounds like pain either way.

A third solution may be to side-step this concern altogether by providing a way to Deserialize directly to ArgMatches.

So, for an API, I think an elegant design might be:

1. A way to deserialize directly to an ArgMatches.
2. A way to merge multiple ArgMatches together.
3. A way to parse the args (get_matches) without throwing an error to the user right away
4. A way to validate a final merged ArgMatches against a command, and throw errors at that point.

Finally, it could all be wrapped up in a convenience API that presents a nice “Layer” API.

I am offering to prototype this if it sounds like a good design / direction to everyone. Although It will probably take me a few weeks / months to get back to this as I already lost too much time to config management and I’m behind on other work.

[epage]

I've got some further questions

• How are arg-only or config-only fields handled?
• Shouldn't validation errors from a layer be reported more eagerly? It seems like required and conflicts would be the errors that should be delayed
• Would the ArgMatches deserializer just use the type hints for storing of types?
• When merging, how do we know when we should overwrite vs merge?
  • Note that ArgMatches doesn't really have knowledge of whats inside the AnyValue and so we can't do any processing on it once its stored (which also means we can't serialize)
• How would we handle defaults?

[luketpeterson]

How are arg-only or config-only fields handled?

I was imagining ArgMatches could be extended to include a source field for each arg, indicating which layer the value came from. So the merge and validation logic could ensure that only values from supported layers are merged, and appropriate errors are thrown if the values came from the wrong layers.

Shouldn't validation errors from a layer be reported more eagerly? It seems like required and conflicts would be the errors that should be delayed

Other errors certainly could be thrown more eagerly. As you say, the required and conflict errors are the main ones to postpone, but if we have a mechanism to postpone error throwing to a later call then we could move all validation there. It could be implemented either way.

Would the ArgMatches deserializer just use the type hints for storing of types?

Yes. That is what I was thinking. There may be some devils in the details that I'll discover when actually implementing it.

When merging, how do we know when we should overwrite vs merge?

I don't have a clear idea for a "deep-merge". For atomic (irreducible) args / configs, which I believe is most of them, it's just a matter of specifying which item takes precedence. I could see merging lists, etc. being a useful enhancement, but I need to understand the use case better to propose how to control the behavior. So IMO deep-merge could be punted, pending more user feedback.

How would we handle defaults?

I was imagining defaults would be treated like another layer by the logic. Maybe an implicit layer since Default would probably be the layer of last resort, so it doesn't dominate any other arg / cfg sources.

[gibfahn]

I think that if update_from_arg_matches() used .contains_id() && .value_source() != ValueSource::DefaultValue instead of just .contains_id() to check whether the value is present, you might be able to do something like this:

use clap::{CommandFactory, FromArgMatches, Parser};
use serde::Deserialize;

#[derive(Debug, Parser, Deserialize)]
struct Opts {
    #[clap(long)]
    a: Option<String>,

    #[serde(default)]
    #[clap(long, default_value = "def")]
    b: String,

    #[clap(long)]
    c: bool,
}

const CONFIG: &str = "
a: one
b: two
c: false
";

fn main() {
    let matches = <Opts as CommandFactory>::command().get_matches();
    let mut opts: Opts = serde_yaml::from_str(CONFIG).unwrap();
    opts.update_from_arg_matches(&matches).unwrap();
    dbg!(&opts);
}

Basically creating an ArgMatches from the command-line args, parsing the config file into the Opts struct, and then updating it with the values from the command line if set.

You'd probably have to keep the serde default value and the clap default value in sync though.

[gibfahn]

There is also the clap_serde_derive crate, which seems to offer a nice solution by creating a struct for your config that has everything wrapped in an Option<>.

I haven't used it or looked at the source due to the AGPL license, but if it handles subcommands properly it seems like a neat solution.