api: Add params_show.

[daavoo]

Closes #6507

Uses repo.params.show with custom error_handler and postprocess the outputs for a more user-friendly structure.

Extend repo.params.show to accept stages argument to cover the "params of current stage" use case.

dvc.org P.R: iterative/dvc.org#3459

---

Examples

• No args.

Will use the current project as repo and retrieve all parameters,
for all stages, for the current revision.

>>> import json
>>> import dvc.api
>>> params = dvc.api.params_show()
>>> print(json.dumps(params, indent=4))
{
    "prepare": {
        "split": 0.2,
        "seed": 20170428
    },
    "featurize": {
        "max_features": 10000,
        "ngrams": 2
    },
    "train": {
        "seed": 20170428,
        "n_est": 50,
        "min_split": 0.01
    }
}

• Filter with stages.

>>> import json
>>> import dvc.api
>>> params = dvc.api.params_show(stages="prepare")
>>> print(json.dumps(params, indent=4))
{
    "prepare": {
        "split": 0.2,
        "seed": 20170428
    }
}

• Git URL as repo.

>>> import json
>>> import dvc.api
>>> params = dvc.api.params_show(
...     repo="https://github.com/iterative/demo-fashion-mnist")
{
    "train": {
        "batch_size": 128,
        "hidden_units": 64,
        "dropout": 0.4,
        "num_epochs": 10,
        "lr": 0.001,
        "conv_activation": "relu"
    }
}

• Using rev.

>>> import json
>>> import dvc.api
>>> params = dvc.api.get_params(
...     repo="https://github.com/iterative/demo-fashion-mnist",
...     rev="low-lr-experiment")
>>> print(json.dumps(params, indent=4))
{
      "train": {
          "batch_size": 128,
          "hidden_units": 64,
          "dropout": 0.4,
          "num_epochs": 10,
          "lr": 0.001,
          "conv_activation": "relu"
      }
}

[skshetry]

Wait, why? It’s a getter. Again, I am confused with porcelain/API discussion.

[efiop]

@skshetry The argument during the last retro discussion was that this is at least following the CLI conventions. Could consider adding get_params as an alias though.

[skshetry]

Right, but is dvc.api the place to have these CLI wrappers?

[daavoo]

CLI wrappers?

What do you call CLI wrappers? I think the idea is that we follow CLI conventions regarding names, but this doesn't mean that the API is a CLI wrapper.
This API uses the internal Python API, raises exceptions, and returns a structure meant to be used in Python (instead of the internal structure returned by CLI --json options, for example).

[dberenbaum]

Personally, my vote would be get_params because the most immediate use case here is to onboard new users who aren't already familiar with the CLI.

However, I would rather get this merged than continue to discuss naming.

[jorgeorpinel]

Filter with stages.

1 Could DVC detect a default stage based on the name of the file you're in? Then the no-args run would auto-filter that one stage's params specifically. Add params_show(all=True) to load all.

2 And maybe when loading a single stage's params, the JSON structure shouldn't have its name as key (so the code never needs to know it). E.g.

# prepare.py
import json
import dvc.api

params = dvc.api.params_show()
print(json.dumps(params))
{
  "split": 0.2,
  "seed": 20170428
}
# use params['split'] instead of params[list(params.keys())[0]]['split']

I guess the inconsistent structure for different calls may be a problem though.

[jorgeorpinel]

Is it very relevant to pick by param files? Maybe the natural targets should be param names (return simple dict) or even stage names and this could be a secondary optional arg.

[daavoo]

Is it very relevant to pick by param files?

In all dvc {x} show / dvc {x} diff commands, targets are files, i.e. https://dvc.org/doc/command-reference/metrics/show

Maybe the natural targets should be param names (return simple dict)

Not sure I follow what this would look like.

[dberenbaum]

I agree that stage names are likely most useful here, but we are also trying to mimic the CLI. It's a good idea @jorgeorpinel but there has already been a lot of related discussions and I would rather get something merged than try to optimize the order of args.

[jorgeorpinel]

Makes sense to follow the CLI.

[daavoo]

And maybe when loading a single stage's params, the JSON structure shouldn't have its name as key (so the code never needs to know it

@jorgeorpinel We don't include the stage name, this is just how the example-get-started params are set up where there are sections inside the params file matching the name of the stage:

https://github.com/iterative/example-get-started/blob/a470fae5c0ccf71641e10d99e69e97534c3d811e/params.yaml#L1-L4

[daavoo]

Could DVC detect a default stage based on the name of the file you're in? Then the no-args run would auto-filter that one stage's params specifically. Add params_show(all=True) to load all.

This would be quite a big assumption to make and not sure how common users have this setup.
stages argument was added to support the use case of the filter by current stage, I don't think the ambiguity of trying to figure it out the current stage would be worth comparing to requiring users to "hardcode" the stage name in the call

[efiop]

Hm, why do we need targets and stages? It it because targets should be paths? But what if you have stages with same names?

[efiop]

Does stages here accept stage at a custom path? It appears it doesn't, unless I'm missing something.

[daavoo]

why do we need targets and stages?

Because they are not mutually exclusive. They have to be decoupled to support several scenarios, for example:

• A stage (stage_A) uses params from different targets (target_A, target_B).

params_show(target_A, stage=stage_A)

• Different parts of a single target (target_A) can be used for multiple stages (stage_A, stage_B).

params_show(target_A, stage=stage_B)

But what if you have stages with same names?

I am not sure I follow. The targets and stages filters are applied separately, so it would be up to the user to provide unambiguous arguments.

Does stages here accept stage at a custom path?

What's is a stage at a custom path?

[efiop]

What's is a stage at a custom path?

Like path/to/dvc.yaml:mystage. Because mystage could be defined in multiple dvc.yaml at different levels.

[daavoo]

Like path/to/dvc.yaml:mystage. Because mystage could be defined in multiple dvc.yaml at different levels.

I updated to filter against stage.addressing instead of stage.name. The former includes the path/to/dvc.yaml: prefix when there are conflicts.

I tested with multiple dvc.yaml using same stage name and it works as expected when including the prefix in the argument

[efiop]

Thank you @daavoo ! Let's run with this and change on top if we'll need anything.