feat(builder): Custom parser for arg values

[epage]

Currently clap is focused on raw values and hard codes support for converting to Strings. Users could then set a validator on that but if they were parsing to validate, the result was thrown away.

This moves clap to an API that let's users specify the data type / parser and we'll store the parsed value and return it to the user.

This is best shown with an example:

let mut cmd = clap::Command::new("raw")
    .arg(
        clap::Arg::new("color")
            .long("color")
            .value_parser(["always", "auto", "never"])
            .default_value("auto")
    )
    .arg(
        clap::Arg::new("hostname")
            .long("hostname")
            .value_parser(clap::builder::NonEmptyStringValueParser::new())
            .takes_value(true)
            .required(true)
    )
    .arg(
        clap::Arg::new("port")
            .long("port")
            .value_parser(clap::value_parser!(u16).range(3000..))
            .takes_value(true)
            .required(true)
    );

let m = cmd.try_get_matches_from_mut(
    ["cmd", "--hostname", "rust-lang.org", "--port", "3001"]
).unwrap();

let color: &String = m.get_one("color")
    .expect("matches definition").expect("default");
assert_eq!(color, "auto");

let hostname: &String = m.get_one("hostname")
    .expect("matches definition").expect("required");
assert_eq!(hostname, "rust-lang.org");

let port: u16 = *m.get_one("port")
    .expect("matches definition").expect("required");
assert_eq!(port, 3001);

Arg::value_parser:

• Default is String
• Default will be overriden if allow_invalid_utf8(true) is called
• value_parser accepts any Into<ValueParser>, which includes
  • Numeric ranges
  • Arrays of possible values
  • Implementations of TypedValueParser including ones provided by clap
  • value_parser!(T) which will auto-select a parser from a type using deref specialization, inspired by feat: Auto-detect FromStr, TryFrom<&OsStr>, etc., in clap_derive #2298
    • While this will benefit all users, it was mostly written for clap_derive for a default value_parser when none is specified.

ArgMatches:

• get_one::<T>
• get_many::<T>
• remove_one::<T>
• remove_many::<T>
• get_raw for when generically walking over all of the data. This helped a lot with existing code (like validator_os), so I went ahead and did it. We might be able to get rid of this in the future but it would require calling get_one or get_many for every possible data type.

Overall, this helped more towards #3476 than I expected by allowing users to plugin their own logic. As an example, we were able to replace the following with value_parser:

• Arg::possible_values
• forbid_empty_value
• allow_invalid_utf8
• validator and validator_os

while all implemented in a way the compiler can more easily discard the parts of the API that are unused, helping us towards #1365

We also gained support for

• Parsing ArgEnums
• Parsing numeric ranges
• Finding the right parser for a given data type
• Completing bools as possible values
• Implicitly disallowing empty paths
• Automatically using a ValueHint::AnyPath when using a PathBuf parser

The builder API isn't as ergonomic as it was before but it is offering the user some more flexibility which is the main selling point of it over the builder API. For example, ArgMatches::get_one returns an Option for when not present, rather than an error to help with #2505. Rather than panicing when the get_one call is used with the wrong data type or the argument isn't defined, we now return an internal-facing enum to help with #3621.

ArgMatches::remove_* had to provide data in Arcs because

• We needed to keep ArgMatches impling Clone and didn't want to force the data to be Clone
• We wanted to support passing between threads
• The only helpful part in the Arc API I saw was try_unwrap but that leaves us with an Arc still if the ref count is more than 1.

One limitation of this API is it doesn't allow using the full power of ArgGroup. Currently, clap allows you accessing all of the values for a group. The new implementation limits that to groups where all present values are of the same type. For more background on why this was originally added to clap, see #283. We might want to re-evaluate this.

I had originally planned for the roll out of this to be staged and under unstable-* feature flags. I decided against this because of how close I feel clap4 is; we can always just change it then and this will get us feedback faster.

Related changes

We now set the source and occurrences for external subcommands.

A lot of refactors were made to the parsing logic to ensure we could pass all the Arg state we wanted down to MatchedArg, even if there is an occurrence without a value. Unfortunately, we do not carry this state around for when there are no occurrences.

Provide ArgMatches::remove_subcommand to allow for using remove_one / remove_many on subcommands, especially for clap_derive.

Deferred

Appearing in the order to be completed

Adjusting the derive API to this is proving to be difficult because of the current traits take &ArgMatches and because FromArgMatches::update_from_arg_matches derive for subcommands assumes that ArgMatches remains untouched. See #3734. Implementing the derive will automatically address

• derive: default to parse_os_str for OsString or PathBuf types #3496
• Error text for derived numeric ranges is unclear #3199
• try_from_str and try_from_os_str functions are called twice #3589

We have not yet deprecated the older builder methods as we probably want derive support implemented first so derive users do not get confused. See #3734.

• Deprecate possible_values in favor of value_parser
• Deprecate allow_invalid_utf8 in favor of ValueParser
• Deprecate arg value validators in favor of ValueParser
• Deprecate forbid_empty_values

We can't add a ValueParser to external subcommands because of the PartialEq requirement and we can't relax that until a breaking release. See #3733.

_one and _many functions do not error if the wrong one is used. We might want to consider that. See #3735.

Clarifying that Arg::default_value is for raw values

Providing an Arg::default_value_t

#2683 brought up the idea of defaulting value_name based on the ValueParser. This is still possible for us to look into, just not right now.

Currently, all values must be of the same type. This assumption has kept things simpler for now. I'm assuming this is ok because the main situation for having position-specific types is when using a separator and the user can provide their own implementation.

Compatibility

MSRV is now 1.56.0. This is needed for Bound::cloned and fits within official MSRV policy
(2 versions back) and unofficial (6 months, see #3267)

Because in clap 3.0.0, we tracked what is an OsString vs String, we were able to graft this into the existing API except for external subcommands.

We now assert on using the wrong method to look up defaults (value_of, value_of_os). This shouldn't be a breaking change as it'll assert when getting a real value.

Footer

Fixes #2683
Fixes #2505
Fixes #3621