ref: clarify about types of metrics/plots

[jorgeorpinel]

Follow up to #2839 (review)

Both the metrics and plots cmd refs. explain 2 types of "metrics" (metrics and plots) -- done to to keep both references connected.

Calling plots a kind of metric is already confusing. Furthermore, in #2839 the concept of "2 types of plots files" was introduced (data series rendered with Vega vs. arbitrary image files) to the latter, which may be confusing.

• Also, the plots usage block needs wrapping to avoid scroll

  

UPDATE: Jump to #2956 (comment)

[jorgeorpinel]

Maybe we need to put the types of metrics in a proper metrics/plots guide and not explain the concepts in the ref...

@dberenbaum do you think we the core team should address this for now, or plan for a guide? Thanks

[dberenbaum]

Maybe we need to put the types of metrics in a proper metrics/plots guide and not explain the concepts in the ref...

👍

I do think the current plots command ref doesn't incorporate image support enough at the moment. It still refers mostly to "plots metrics," and I don't think the "Types of Metrics" section at the top makes much sense now.

Also, it doesn't look like it's addressed in any of show, diff, or modify, nor is it specified anywhere which options are supported for image files.

[pared]

When the time comes, maybe we need some visual aid to explain how we handle different metrics?

          +--------+                              
          | metric |                              
          +--------+                              
          *         *                             
        **           **                           
       *               *                          
+---------+         +-------+                     
| metrics |         | plots |                     
+---------+        *+-------+**                   
                 **            **                 
               **                **               
             **                    **             
    +-------------+         +-------------------+ 
    | image_plots |         | data_series_plots | 
    +-------------+         +-------------------+ 

[dberenbaum]

I like the tree, but do we consider image plots as metrics at all? They could be actual images, like computer vision examples. I'm not sure it's intuitive to think of an image file as a type of metric.

[daavoo]

I like the tree, but do we consider image plots as metrics at all? They could be actual images, like computer vision examples. I'm not sure it's intuitive to think of an image file as a type of metric.

Also... parallel coordinates plots 🤔 (would be part of data_series_plots?)

[jorgeorpinel]

do we consider image plots as metrics at all?

I think this is the key question. By "plots" we refer both to metrics that come in "data series" form (term "data" here is also not ideal) which can be visualized by plot classic (using Vega), and to custom images that can be anything really (not necessarily a plot per se right?).

Conceptually I'd organize it like:

metrics

• scalar (short text file)
• series (long text file)
• image (binary file)

Separately,

plots feature (may need renaming)

• generate from series metrics
• associate/render custom user images

But at that point this becomes a product question because command names and semantics or even file schemes could be changed a bit.

So should we transfer to the core repo? Try to patch the explanations in the ref. for now and/or plan for a guide? Thanks

[dberenbaum]

Since we are still clarifying the types of metrics and plots and we already have the discussion here, why don't we decide how to describe them here and then decide what we need to edit in help text, command reference, user guide, etc.?

I would only describe scalars as metrics because:

1. It's what I think of as metrics and what I generally see described as ML metrics: https://neptune.ai/blog/performance-metrics-in-machine-learning-complete-guide.
2. It's consistent with language in experiment tracking tools like mlflow and wandb. Mlflow specifically defines metrics as numeric key-value pairs. Wandb doesn't define metrics but only uses the term when describing scalars.
3. It fits with the dvc metrics terminology.

I don't think we should describe any kind of plots as metrics.

[jorgeorpinel]

why don't we decide how to describe them here
I don't think we should describe any kind of plots as metrics.

OK in this case we mainly have to edit the command refs. for both metrics and plots, and unlink them (don't call plots metrics).

Then we need to incorporate this approach to #2925 (Get Started) Cc @iesahin

Finally, to consider it as well for #2572 (User Guide)

[iesahin]

I have separated plots from metrics/params to describe them more independently in #3050

A visualization section can describe plots as CSV files or plots as images, as you see fit.

[jorgeorpinel]

we mainly have to edit the command refs. for both metrics and plots, and unlink them (don't call plots metrics)

Should we expect a PR from the core team on this? Or do we consider it docs work as the feature is already documented (in a way we don't love) @dberenbaum

[dberenbaum]

We need to actually update some of the help text in the core repo, and then to update the command ref. So at least some of it needs to happen from the core team, and then we can see whether it makes sense to have the docs team take over some of it. @pared Do you want me to create an issue and/or PR in the core repo, or do you want to do it yourself?

[dberenbaum]

Waiting on iterative/dvc#7086

[jorgeorpinel]

Migrating main points from #1442:

we could just skip this explanation altogether and only call metrics metrics, and plots plots.
Why even merge these 2 concepts into one? ... the format requirements for each kind of file are very different.

We need to come up with clear separation.

even when plots are metrics technically. We just probably don't need to stress that
The same overlap exists between other concepts technically, like "remote storage" and "cache" but we never call remote storage "remote cache"

[jorgeorpinel]

Waiting on iterative/dvc#7086

@dberenbaum @pared it's not obvious how that issue relates to the terminology discussion here. Could someone briefly explain the plan? Thanks

[jorgeorpinel]

Maybe we need to put the types of metrics in a proper metrics/plots guide

BTW the guide part should be covered in #2572 so we can limit this issue to the cmd ref. indeed + maybe mentions in the Get Started if needed.

[dberenbaum]

One idea: could all metrics and plots functions get mixed into dvc metrics show/plot/diff/config?

I thought the consensus from the discussion above was to cleanly separate these concepts. Mixing all metrics and plots functions into one command looks like the opposite direction to me.

Otherwise this is a core discussion and could be moved to the core repo as such.

As mentioned in #2956 (comment), there is some help text to update, but otherwise I'm not sure how it's a core discussion? It seems the main action point is to separate metrics and plots in the docs.

[jorgeorpinel]

Oh. No, I don't necessarily agree that the solution is to separate the concepts in docs because plots are conceptually types of metrics (general meaning of the word "metrics"). Again thinking of the example of an AUC-ROC curve: These are clearly modeling metrics (right?).

But if that's what you decide then sure, we can just call metrics metrics and plots plots, and never mix the terminology: we can use terms like "performance measurements" instead of "metrics" in the context of plots. Does that sound like a plan then?

[pared]

we can use terms like "performance measurements" instead of "metrics" in the context of plots. Does that sound like a plan then?

My thoughts were drifting in a similar direction: I was actually thinking about refactoring plots and metrics are types of metrics to we have two basic types of quality measurands: metrics and plots.

[dberenbaum]

Again thinking of the example of an AUC-ROC curve: These are clearly modeling metrics (right?).

Maybe I'm odd, but I wouldn't call an AUC-ROC curve or anything other than a scalar value a metric 🤷 .

Edit: I would call the AUC-ROC scalar value a metric, but I wouldn't describe the ROC curve as a metric.

But if that's what you decide then sure, we can just call metrics metrics and plots plots, and never mix the terminology: we can use terms like "performance measurements" instead of "metrics" in the context of plots. Does that sound like a plan then?

Sure. We can discuss further if needed once we have a PR to review.

[jorgeorpinel]

I wouldn't call an AUC-ROC curve or anything other than a scalar value a metric

Ah then indeed we should try a more general term like "performance measure" to separate the terminology 🙂

I updated the labels as this may involve more than just the cmd ref but no need to work on anything outside of small/strategic word changes in the cmd ref on your end @pared (we'll take it over from there). Thanks!

[shcheklein]

I don't see a big problem either, though I'm biased already. I remember initially I also had a lot of doubts.

I think partially this confusion comes from the fact that in loggers like W&B you would log metrics and then plot them online over time. Plots is something that is built on top of data that you log, including scalars, etc.

But they also have a way to log plots - https://docs.wandb.ai/guides/track/log/plots . Even though in the description they would say: "These simple charts make it easy to construct basic visualizations of metrics and results.". Not sure if that affects how this data is stored internally though or it's just a bunch of convenience functions.

And we mix a bit files with data (scalar or not) with plots (spec of a graphical representation). It means that we can't make a plot on top of metrics files? To show its trend for example? Or a plot that takes data from multiple outputs. Etc. It might make sense to detach plot specs from the outputs.

[pared]

It might make sense to detach plot specs from the outputs.

@shcheklein
This is a good point, especially that after iterative/dvc#7086 one will no longer have to specify something as a plot output to create a visualization out of that.

[jorgeorpinel]

loggers like W&B you would log metrics and then plot them

That's the connection between metrics and plots I had on the back of my mind as well, but not something DVC does at the moment.

[jorgeorpinel]

I think this will no longer be a concern after the changes in #3691 are merged.