Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

api: Add params_show #3459

Merged
merged 38 commits into from
Jul 4, 2022
Merged
Show file tree
Hide file tree
Changes from 32 commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
83fd85d
api: Add params_show.
daavoo Jun 16, 2022
59e38bc
Updates
daavoo Jun 16, 2022
044cade
Apply suggestions from code review
jorgeorpinel Jun 16, 2022
6d16c25
sidebar: Use alphabetical order for api-reference
daavoo Jun 17, 2022
9bbbe25
Applied suggestions
daavoo Jun 17, 2022
3d33521
Motivation for remote `repo`
daavoo Jun 17, 2022
1f7ca1f
Merge branch 'master' into api-get-params
jorgeorpinel Jun 20, 2022
cc11224
Update content/docs/api-reference/params_show.md
jorgeorpinel Jun 20, 2022
6490cc9
Merge branch 'api-get-params' of github.com:iterative/dvc.org into ap…
jorgeorpinel Jun 20, 2022
c3b5e3b
ref: params_show *targets desc update
jorgeorpinel Jun 20, 2022
6b33374
ref: clarify params_show() descp
jorgeorpinel Jun 20, 2022
a631f32
ref: remove simple example from params_show() and
jorgeorpinel Jun 20, 2022
e19dda0
ref: simplify params)show() sample
jorgeorpinel Jun 20, 2022
c2d9ec8
ref: simplify params_show(rev)
jorgeorpinel Jun 20, 2022
c72d752
ref: improve params_show(deps)
jorgeorpinel Jun 20, 2022
cb77cb4
ref: improve params_show(stages)
jorgeorpinel Jun 20, 2022
92da7c5
ref: improve project version example of params_show()
jorgeorpinel Jun 20, 2022
23d7099
ref: improve params_show(targets) example and
jorgeorpinel Jun 20, 2022
090feaa
ref: reorg params_show() examples per
jorgeorpinel Jun 20, 2022
6e3b8a7
ref: shorten params_show(repo) example
jorgeorpinel Jun 20, 2022
834e47b
Merge branch 'master' into api-get-params
jorgeorpinel Jun 22, 2022
3060eef
api: json->py dict
jorgeorpinel Jun 22, 2022
7529503
Apply suggestions from code review
jorgeorpinel Jun 22, 2022
9572e72
api: move params_show(stages) example up, and
jorgeorpinel Jun 22, 2022
5032c5a
Merge branch 'api-get-params' of github.com:iterative/dvc.org into ap…
jorgeorpinel Jun 22, 2022
3ced721
Merge branch 'master' into api-get-params
jorgeorpinel Jun 25, 2022
1af16ab
ref: keep params_show examples hypothetical
jorgeorpinel Jun 25, 2022
2e02967
ref: params_show( stages before repo
jorgeorpinel Jun 25, 2022
2a70b7d
ref: more realistic params_show(stages) example
jorgeorpinel Jun 25, 2022
590dfdc
Merge branch 'main' into api-get-params
jorgeorpinel Jun 29, 2022
4f755c0
ref: clarify on params_show(*targets)
jorgeorpinel Jun 29, 2022
1c53cc6
Apply suggestions from code review
jorgeorpinel Jun 29, 2022
b8cbaf1
Apply suggestions from review
daavoo Jul 4, 2022
712464d
Remove reference to example repo
daavoo Jul 4, 2022
2525cf9
Update example
daavoo Jul 4, 2022
24f18aa
reorder
daavoo Jul 4, 2022
87ebba4
Wording
daavoo Jul 4, 2022
e8596d5
Remove example get started reference
daavoo Jul 4, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
198 changes: 198 additions & 0 deletions content/docs/api-reference/params_show.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,198 @@
# dvc.api.params_show()

Load <abbr>parameters</abbr> (name and values) tracked in a <abbr>DVC
project</abbr>.

```py
def params_show(
*targets: str, # Optional
stages: Optional[Union[str, Iterable[str]]] = None,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can this be moved above repo? stages seems more important, and it makes sense to me to put it right after targets and keep repo and rev together.

This comment was marked as resolved.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the signature from what I can tell. I don't think we should alter it for docs (other than that one comment I added).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That said if you plan to change the signature in the core repo then sure. Lmk

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that those are all keyword arguments so the order doesn't really affect the behavior. We could also update the order in core, if it makes more sense

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't mind updating in core, but to me it's more important to update here to start with more common use cases.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK. I moved stages one position up. Why not all the way up then? Since we deem the stage-filtering example most important... Which goes back to iterative/dvc#7613 (review)

I don't mind updating in core

I think that's ideal because at some point this may get copy-pasted again from there (on some update) and our custom docs order is gone.

repo: Optional[str] = None,
rev: Optional[str] = None,
deps: bool = False,
daavoo marked this conversation as resolved.
Show resolved Hide resolved
) -> Dict:
```

## Usage:

```py
import dvc.api

params = dvc.api.params_show()
```

## Description

Retrieves <abbr>parameter</abbr> keys and values from a <abbr>DVC project</abbr>
and returns a dictionary, such as:

```json
{
"split": 0.2,
"seed": 20170428
}
```

Without arguments, this function will retrieve all parameters from all tracked
param files (used in any `dvc.yaml` file). This applies to the current project
version when using Git (including any changes in the working tree).

The function parameters (below) let you restrict what's retrieved.

## Parameters

- `*targets` - one or more separate path(s) to valid parameter file(s) to
retrieve params from, for example `"params.py", "feat/params.toml"`. If no
`targets` are provided, all param files tracked in any `dvc.yaml` will be
targeted by default. Note that explicit targets don't have to be in a
`dvc.yaml` (unless `deps=True`).

- `repo` - specifies the location of the DVC project. It can be a URL or a file
system path. Both HTTP and SSH protocols are supported for online Git repos
(e.g. `[user@]server:project.git`). _Default_: The current project (found by
walking up from the current working directory tree).

- `stages` - one or more names of the stage(s) to retrieve params from.
_Default_: `None` (all parameters from all stages will be retrieved).

- `rev` - Git commit (any [revision](https://git-scm.com/docs/revisions) such as
a branch or tag name, a commit hash or an
[experiment](/doc/command-reference/exp) name). If `repo` is not a Git repo,
this option is ignored. _Default_: `None` (current working tree will be used)
jorgeorpinel marked this conversation as resolved.
Show resolved Hide resolved

- `deps` - whether to retrieve only params that are stage dependencies. Accepts
`True` or `False` (_default_).

## Example: Filter by stage name(s)

> Working on https://github.com/iterative/example-get-started, file
> `src/featurization.py`.
daavoo marked this conversation as resolved.
Show resolved Hide resolved

`stages` can be a single name (string):

```git
+import json
-import yaml

-params = yaml.safe_load(open("params.yaml"))["featurize"]
+import dvc.api
+
+params = dvc.api.params_show(stages="featurize")

...
-max_features = params["max_features"]
-ngrams = params["ngrams"]
+max_features = params["featurize"]["max_features"]
+ngrams = params["featurize"]["ngrams"]
```

Or an iterable of strings:

```py
import json
import dvc.api
params = dvc.api.params_show(stages=["featurize", "train"])
```

```json
{
"featurize": {
"max_features": 200,
"ngrams": 2
},
"train": {
"seed": 20170428,
"n_est": 50,
"min_split": 0.01
}
}
```

## Example: Load specific parameter file(s)

You can pass any valid param file path as target to load all of the parameters
defined in it:

```py
import json
import dvc.api
params = dvc.api.params_show("params.yaml")
```

```json
{
"run_mode": "prod",
"configs": {
"dev": "configs/params_dev.yaml",
...
```

Or multiple path targets:

```py
import json
import dvc.api
params = dvc.api.params_show(
"configs/params_dev.yaml", "configs/params_prod.yaml")
```

```json
{
"configs/params_prod.yaml:run_mode": "prod",
"configs/params_prod.yaml:config_file": "configs/params_prod.yaml",
...
"configs/params_dev.yaml:run_mode": "dev",
"configs/params_dev.yaml:config_file": "configs/params_dev.yaml",
...
```

## Example: Use a remote DVC repository

You can use the `repo` argument to retrieve parameters from any <abbr>DVC
repository</abbr> without having to clone it locally.

```py
import json
import dvc.api
params = dvc.api.get_params(
repo="https://github.com/iterative/demo-fashion-mnist")
daavoo marked this conversation as resolved.
Show resolved Hide resolved
daavoo marked this conversation as resolved.
Show resolved Hide resolved
```

```json
{
"train": {
"batch_size": 128,
"hidden_units": 64,
"dropout": 0.4,
...
```

## Example: Specify a project version

> Working on https://github.com/iterative/example-get-started

You can retrieve params from arbitrary Git commits, for example a branch name:

```py
import json
import dvc.api
params = dvc.api.params_show(rev="tune-hyperparams")
```

```json
{
"prepare": {
"split": 0.2,
"seed": 20170428
},
"featurize": {
"max_features": 200,
"ngrams": 2
},
"train": {
"seed": 20170428,
"n_est": 100,
"min_split": 8
}
}
```
8 changes: 3 additions & 5 deletions content/docs/command-reference/params/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,14 +112,12 @@ $ dvc stage add -n train -d train.py -d users.csv -o model.pkl \
> Python parameters files.

The `train.py` script will have some code to parse and load the needed
parameters. For example:
parameters. For example, you can use `dvc.api.params_show()`:

```py
from ruamel.yaml import YAML
import dvc.api

with open("params.yaml", 'r') as fd:
yaml = YAML()
params = yaml.load(fd)
params = dvc.api.params_show()

lr = params['lr']
epochs = params['train']['epochs']
Expand Down
8 changes: 3 additions & 5 deletions content/docs/command-reference/run.md
Original file line number Diff line number Diff line change
Expand Up @@ -424,14 +424,12 @@ $ dvc run -n train \
python train_model.py 20200105 model.p
```

`train_model.py` will include some code to open and parse the parameters:
`train_model.py` can use the `dvc.api.params_show()` to parse the parameters:

```py
from ruamel.yaml import YAML
import dvc.api

with open("params.yaml", 'r') as fd:
yaml = YAML()
params = yaml.load(fd)
params = dvc.api.params_show()

seed = params['seed']
lr = params['train']['lr']
Expand Down
8 changes: 3 additions & 5 deletions content/docs/command-reference/stage/add.md
Original file line number Diff line number Diff line change
Expand Up @@ -406,14 +406,12 @@ $ dvc stage add -n train \
python train_model.py 20200105 model.p
```

`train_model.py` will include some code to open and parse the parameters:
`train_model.py` can use the `dvc.api.params_show()` to parse the parameters:

```py
from ruamel.yaml import YAML
import dvc.api

with open("params.yaml", 'r') as fd:
yaml = YAML()
params = yaml.load(fd)
params = dvc.api.params_show()

seed = params['seed']
lr = params['train']['lr']
Expand Down
12 changes: 8 additions & 4 deletions content/docs/sidebar.json
Original file line number Diff line number Diff line change
Expand Up @@ -492,17 +492,21 @@
"slug": "get_url",
"label": "get_url()"
},
{
"slug": "make_checkpoint",
"label": "make_checkpoint()"
},
{
"slug": "open",
"label": "open()"
},
{
"slug": "read",
"label": "read()"
"slug": "params_show",
"label": "params_show()"
},
{
"slug": "make_checkpoint",
"label": "make_checkpoint()"
"slug": "read",
"label": "read()"
}
]
},
Expand Down