Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

api: Add params_show #3459

Merged
merged 38 commits into from
Jul 4, 2022
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
83fd85d
api: Add params_show.
daavoo Jun 16, 2022
59e38bc
Updates
daavoo Jun 16, 2022
044cade
Apply suggestions from code review
jorgeorpinel Jun 16, 2022
6d16c25
sidebar: Use alphabetical order for api-reference
daavoo Jun 17, 2022
9bbbe25
Applied suggestions
daavoo Jun 17, 2022
3d33521
Motivation for remote `repo`
daavoo Jun 17, 2022
1f7ca1f
Merge branch 'master' into api-get-params
jorgeorpinel Jun 20, 2022
cc11224
Update content/docs/api-reference/params_show.md
jorgeorpinel Jun 20, 2022
6490cc9
Merge branch 'api-get-params' of github.com:iterative/dvc.org into ap…
jorgeorpinel Jun 20, 2022
c3b5e3b
ref: params_show *targets desc update
jorgeorpinel Jun 20, 2022
6b33374
ref: clarify params_show() descp
jorgeorpinel Jun 20, 2022
a631f32
ref: remove simple example from params_show() and
jorgeorpinel Jun 20, 2022
e19dda0
ref: simplify params)show() sample
jorgeorpinel Jun 20, 2022
c2d9ec8
ref: simplify params_show(rev)
jorgeorpinel Jun 20, 2022
c72d752
ref: improve params_show(deps)
jorgeorpinel Jun 20, 2022
cb77cb4
ref: improve params_show(stages)
jorgeorpinel Jun 20, 2022
92da7c5
ref: improve project version example of params_show()
jorgeorpinel Jun 20, 2022
23d7099
ref: improve params_show(targets) example and
jorgeorpinel Jun 20, 2022
090feaa
ref: reorg params_show() examples per
jorgeorpinel Jun 20, 2022
6e3b8a7
ref: shorten params_show(repo) example
jorgeorpinel Jun 20, 2022
834e47b
Merge branch 'master' into api-get-params
jorgeorpinel Jun 22, 2022
3060eef
api: json->py dict
jorgeorpinel Jun 22, 2022
7529503
Apply suggestions from code review
jorgeorpinel Jun 22, 2022
9572e72
api: move params_show(stages) example up, and
jorgeorpinel Jun 22, 2022
5032c5a
Merge branch 'api-get-params' of github.com:iterative/dvc.org into ap…
jorgeorpinel Jun 22, 2022
3ced721
Merge branch 'master' into api-get-params
jorgeorpinel Jun 25, 2022
1af16ab
ref: keep params_show examples hypothetical
jorgeorpinel Jun 25, 2022
2e02967
ref: params_show( stages before repo
jorgeorpinel Jun 25, 2022
2a70b7d
ref: more realistic params_show(stages) example
jorgeorpinel Jun 25, 2022
590dfdc
Merge branch 'main' into api-get-params
jorgeorpinel Jun 29, 2022
4f755c0
ref: clarify on params_show(*targets)
jorgeorpinel Jun 29, 2022
1c53cc6
Apply suggestions from code review
jorgeorpinel Jun 29, 2022
b8cbaf1
Apply suggestions from review
daavoo Jul 4, 2022
712464d
Remove reference to example repo
daavoo Jul 4, 2022
2525cf9
Update example
daavoo Jul 4, 2022
24f18aa
reorder
daavoo Jul 4, 2022
87ebba4
Wording
daavoo Jul 4, 2022
e8596d5
Remove example get started reference
daavoo Jul 4, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
253 changes: 253 additions & 0 deletions content/docs/api-reference/params_show.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,253 @@
# dvc.api.params_show()

Returns all the <abbr>params</abbr> associated with the given <abbr>DVC
repository</abbr>.
jorgeorpinel marked this conversation as resolved.
Show resolved Hide resolved

```py
def params_show(
*targets: str,
repo: Optional[str] = None,
stages: Optional[Union[str, Iterable[str]]] = None,
daavoo marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can this be moved above repo? stages seems more important, and it makes sense to me to put it right after targets and keep repo and rev together.

This comment was marked as resolved.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the signature from what I can tell. I don't think we should alter it for docs (other than that one comment I added).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That said if you plan to change the signature in the core repo then sure. Lmk

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that those are all keyword arguments so the order doesn't really affect the behavior. We could also update the order in core, if it makes more sense

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't mind updating in core, but to me it's more important to update here to start with more common use cases.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK. I moved stages one position up. Why not all the way up then? Since we deem the stage-filtering example most important... Which goes back to iterative/dvc#7613 (review)

I don't mind updating in core

I think that's ideal because at some point this may get copy-pasted again from there (on some update) and our custom docs order is gone.

rev: Optional[str] = None,
deps: bool = False,
daavoo marked this conversation as resolved.
Show resolved Hide resolved
) -> Dict:
```

#### Usage:
jorgeorpinel marked this conversation as resolved.
Show resolved Hide resolved

```py
import dvc.api

params = dvc.api.params_show()
```

## Description

Retrieves the <abbr>params</abbr> tracked in a <abbr>DVC repository</abbr>.
jorgeorpinel marked this conversation as resolved.
Show resolved Hide resolved

Without arguments, this function will retrieve all <abbr>params</abbr> from all
parameter files, for the current revision.
daavoo marked this conversation as resolved.
Show resolved Hide resolved

See the options below to restrict the <abbr>params</abbr> retrieved.

## Parameters

- **`*targets`** (str, optional): Names of the parameter files to retrieve
jorgeorpinel marked this conversation as resolved.
Show resolved Hide resolved
<abbr>params</abbr> from. If no `targets` are provided, all parameter files
will be used. Note that targets don't necessarily have to be defined in
`dvc.yaml`

- **`repo`** (str, optional): location of the <abbr>DVC repository</abbr>.
Defaults to the current project (found by walking up from thecurrent working
directory tree). It can be a URL or a file system path. Both HTTP and SSH
protocols are supported for online Git repos (e.g. [user@]server:project.git).

- **`stages`** (Union[str, Iterable[str]], optional): Name(s) of the stages to
retrieve <abbr>params</abbr> from. Defaults to `None`. If no stages are
provided, all parameters from all stages will be retrieved.

- **`rev`**: (str, optional): Name of the Git
[revision](https://git-scm.com/docs/revisions) to retrieve <abbr>params</abbr>
from. Example of git revision can be a branch or tag name, a commit hash or a
daavoo marked this conversation as resolved.
Show resolved Hide resolved
dvc experiment name. Defaults to `HEAD`. If `repo` is not a Git repo, this
option is ignored.

- **`deps`**: (bool, optional): Whether to retrieve only <abbr>params</abbr>
that are <abbr>stage</abbr> dependencies or not. Defaults to `False`.

## Example: No arguments.
jorgeorpinel marked this conversation as resolved.
Show resolved Hide resolved

> Working on https://github.com/iterative/example-get-started

```py
import json
import dvc.api
params = dvc.api.get_params()
print(json.dumps(params, indent=4))
```

```json
{
"prepare": {
"split": 0.2,
"seed": 20170428
},
"featurize": {
"max_features": 200,
"ngrams": 2
},
"train": {
"seed": 20170428,
"n_est": 50,
"min_split": 0.01
}
}
```

## Example: Filtering with `stages`.
jorgeorpinel marked this conversation as resolved.
Show resolved Hide resolved

> Working on https://github.com/iterative/example-get-started

`stages` can a single string:

```py
import json
import dvc.api
params = dvc.api.get_params(stages="prepare")
print(json.dumps(params, indent=2))
```

```json
{
"prepare": {
"split": 0.2,
"seed": 20170428
}
}
```

Or an iterable of strings:

```py
import json
import dvc.api
params = dvc.api.get_params(stages=["prepare", "train"])
print(json.dumps(params, indent=2))
```

```json
{
"prepare": {
"split": 0.2,
"seed": 20170428
},
"train": {
"seed": 20170428,
"n_est": 50,
"min_split": 0.01
}
}
```

## Example: Using `rev`.
jorgeorpinel marked this conversation as resolved.
Show resolved Hide resolved

> Working on https://github.com/iterative/example-get-started

```py
import json
import dvc.api
params = dvc.api.get_params(rev="tune-hyperparams")
daavoo marked this conversation as resolved.
Show resolved Hide resolved
print(json.dumps(params, indent=2))
```

```json
{
"prepare": {
"split": 0.2,
"seed": 20170428
},
"featurize": {
"max_features": 200,
"ngrams": 2
},
"train": {
"seed": 20170428,
"n_est": 100,
"min_split": 8
}
}
```

## Example: Using `targets`.

> Working on `multi-params-files` folder of
> https://github.com/iterative/pipeline-conifguration

You can pass a single target:
jorgeorpinel marked this conversation as resolved.
Show resolved Hide resolved

```py
import json
import dvc.api
params = dvc.api.params_show("params.yaml")
print(json.dumps(params, indent=2))
```

```json
{
"run_mode": "prod",
"configs": {
"dev": "configs/params_dev.yaml",
"test": "configs/params_test.yaml",
"prod": "configs/params_prod.yaml"
},
"evaluate": {
"dataset": "micro",
"size": 5000,
"metrics": ["f1", "roc-auc"],
"metrics_file": "reports/metrics.json",
"plots_cm": "reports/plot_confusion_matrix.png"
}
}
```

Or multiple targets:

```py
import json
import dvc.api
params = dvc.api.params_show(
"configs/params_dev.yaml", "configs/params_prod.yaml")
print(json.dumps(params, indent=2))
```

```json
{
"configs/params_prod.yaml:run_mode": "prod",
"configs/params_prod.yaml:config_file": "configs/params_prod.yaml",
"configs/params_prod.yaml:data_load": {
"dataset": "large",
"sampling": {
"enable": true,
"size": 50000
}
},
"configs/params_prod.yaml:train": {
"epochs": 1000
},
"configs/params_dev.yaml:run_mode": "dev",
"configs/params_dev.yaml:config_file": "configs/params_dev.yaml",
"configs/params_dev.yaml:data_load": {
"dataset": "development",
"sampling": {
"enable": true,
"size": 1000
}
},
"configs/params_dev.yaml:train": {
"epochs": 10
}
}
```

## Example: Git URL as `repo`.
jorgeorpinel marked this conversation as resolved.
Show resolved Hide resolved

```py
import json
import dvc.api
params = dvc.api.get_params(
repo="https://github.com/iterative/demo-fashion-mnist")
daavoo marked this conversation as resolved.
Show resolved Hide resolved
daavoo marked this conversation as resolved.
Show resolved Hide resolved
print(json.dumps(params, indent=2))
```

```json
{
"train": {
"batch_size": 128,
"hidden_units": 64,
"dropout": 0.4,
"num_epochs": 10,
"lr": 0.001,
"conv_activation": "relu"
}
}
```
8 changes: 3 additions & 5 deletions content/docs/command-reference/params/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,14 +112,12 @@ $ dvc stage add -n train -d train.py -d users.csv -o model.pkl \
> Python parameters files.

The `train.py` script will have some code to parse and load the needed
parameters. For example:
parameters. For example, you can use `dvc.api.params_show()`:

```py
from ruamel.yaml import YAML
import dvc.api

with open("params.yaml", 'r') as fd:
yaml = YAML()
params = yaml.load(fd)
params = dvc.api.params_show()

lr = params['lr']
epochs = params['train']['epochs']
Expand Down
8 changes: 3 additions & 5 deletions content/docs/command-reference/run.md
Original file line number Diff line number Diff line change
Expand Up @@ -424,14 +424,12 @@ $ dvc run -n train \
python train_model.py 20200105 model.p
```

`train_model.py` will include some code to open and parse the parameters:
`train_model.py` can use the `dvc.api.params_show()` to parse the parameters:

```py
from ruamel.yaml import YAML
import dvc.api

with open("params.yaml", 'r') as fd:
yaml = YAML()
params = yaml.load(fd)
params = dvc.api.params_show()

seed = params['seed']
lr = params['train']['lr']
Expand Down
8 changes: 3 additions & 5 deletions content/docs/command-reference/stage/add.md
Original file line number Diff line number Diff line change
Expand Up @@ -406,14 +406,12 @@ $ dvc stage add -n train \
python train_model.py 20200105 model.p
```

`train_model.py` will include some code to open and parse the parameters:
`train_model.py` can use the `dvc.api.params_show()` to parse the parameters:

```py
from ruamel.yaml import YAML
import dvc.api

with open("params.yaml", 'r') as fd:
yaml = YAML()
params = yaml.load(fd)
params = dvc.api.params_show()

seed = params['seed']
lr = params['train']['lr']
Expand Down
4 changes: 4 additions & 0 deletions content/docs/sidebar.json
Original file line number Diff line number Diff line change
Expand Up @@ -488,6 +488,10 @@
"label": "Python API Reference",
"source": "api-reference/index.md",
"children": [
{
"slug": "params_show",
"label": "params_show()"
},
{
"slug": "get_url",
daavoo marked this conversation as resolved.
Show resolved Hide resolved
"label": "get_url()"
Expand Down