Add GS:Visualization and Plots #3050

iesahin · 2021-11-30T15:39:43Z

This is mostly for discussion after the Notion experimentation went "we wanted the best, but it turned out like always" :)

I'll develop this as we discuss the scope of changes.

Closes #2925
Closes #3309

iesahin · 2021-12-01T10:03:14Z

The following are @shcheklein 's comments in Notion. Some of these are more related to #3051 than this,

It's not to my mind. The fact that we use DVCLive to produce metric files (and we should be doing this for live and for regular scalar metrics, confusion matrices, etc) doesn't mean we even have to mention it here.

The focus of this document (if it's a replacement of the exisiting get started) is to show the general mechanism of codifying metrics as files. Show regular commands - dvc plots show and only after that live metrics as an advanced example.

Or even if we start with live metrics (to emphasize the experiments tracking, we should still mention regular metrics).

I would also keep the existing metrics section in the regular NLP get started,. at least for now. I don't see any reasons to touch it.

If the project will use DVCLive, then there is no natural point that we'll be using ˋdvc metrics.ˋ

DVCLive should be used as an utility to dump regular metrics / plots files.

I think dvc metrics command is more related to pipelines

I don't any reason for this. The current get started metrics are about comparing experiments.

Also, it's not only about dvc metrics , it's about dvc plots . If we are removing the existing section we should have a hollistic approach to present those here.

I don't see any reasons to not include different types of metrics and plots in the project and briefly explain those here.

iesahin · 2021-12-01T10:03:39Z

I don't intend to cover metrics here. In my plan this page will be related to plots with experiments.

shcheklein · 2021-12-02T02:05:38Z

I don't intend to cover metrics here. In my plan this page will be related to plots with experiments.

sounds good, esp considering that metrics are covered by the first part in this case

content/docs/start/visualization.md

shcheklein · 2021-12-02T02:08:42Z

I assume that we keep the metrics, plots, etc section in the data trail as well, right?

content/docs/start/visualization.md

shcheklein · 2021-12-02T02:12:37Z

@iesahin thanks Emre, that's exactly right approach to the iteration. Makes it way easier to review and get on the same page before we jump into implementation.

iesahin · 2021-12-14T12:27:07Z

added some content to this for you to see the flow concretely @shcheklein @dberenbaum

iesahin · 2021-12-14T12:28:13Z

BTW, could we remove style checks in draft PRs? @shcheklein My setup in WSL seems not to run the commit hook, and I forget to run !yarn format % before commits.

shcheklein · 2021-12-14T22:05:39Z

BTW, could we remove style checks in draft PRs?

I think so, there is a ready_for_review event for this. I would confirm with team this though - some people might benefit from using CI even on draft.

iesahin · 2021-12-15T15:26:50Z

I think so, there is a ready_for_review event for this. I would confirm with team this though - some people might benefit from using CI even on draft.

On second thought, not a big deal. Thanks.

dberenbaum · 2021-12-15T19:31:06Z

Regarding how to introduce dvclive, I'm not sure whether it's better as a 3rd type of plot, or as a different way to generate the other types of plots. I think it's fine in this form and we can iterate later since there's no clear right way.

In case it seems more compelling, the alternative is to explain plots as data series or images, and mention dvclive briefly as the way that those plots files were generated here. This makes dvclive more of a helper utility and less of an entirely separate concept. For example:

Show the acc.tsv or similar from the dvclive output. Mention that it was generated by the dvclive callback, but this data can be generated manually as well.
Show misclassified examples or other images. Again mention that they were logged by dvclive but can be generated manually.

shcheklein · 2021-12-16T02:14:27Z

content/docs/start/visualization.md

@@ -0,0 +1,100 @@
+# Visualization with DVC Plots
+
+In parallel with versioning parameters and metrics by associating them with


Early to edit language, but this paragraph is too complicated for the intro.

"multivariate outputs"

DVC allows to generate plots - I think DVC also captures them, right? (keeps track)

In parallel with versioning parameters and metrics - this means that it should go right after experiments show/run?

shcheklein · 2021-12-16T02:19:14Z

Looks good overall for me, Emre. One question is the right place for this + properly integrate into the project that we already use (if it's not yet).

iesahin · 2021-12-28T13:49:46Z

Do you think is it better to cover parallel coordinates plot here? @shcheklein @dberenbaum

iesahin · 2021-12-29T08:26:34Z

Need to update the image in Update example-dvc-experiments with dvc exp init and confusion matrix example-repos-dev#97 it doesn't look good: https://dvc-org-iesahin-gs-visu-lamemw.herokuapp.com/doc/start/visualization

dberenbaum · 2021-12-29T18:55:27Z

Do you think is it better to cover parallel coordinates plot here? @shcheklein @dberenbaum

IMO no. We can't squeeze every visualization into GS, and this one is particular to comparing experiments, so it seems like an odd fit.

iesahin · 2022-01-04T16:56:56Z

The misclassification visualization is something like.

I think this one is better but requires a few more iterations.

iesahin · 2022-01-10T14:23:14Z

I've updated the project to use MNIST instead of Fashion-MNIST and the misclassification examples look like:

I think this one is easier to interpret.

iesahin · 2022-01-12T13:44:14Z

I'll wait for #3051 to be reviewed/merged in separate PR before going on to this. If you think we can proceed in parallel, please let me know. @shcheklein @dberenbaum @jorgeorpinel

jorgeorpinel

A few more suggestions. Only the last question about the dvclive image worries me a little, everything else is minor. Thanks again @iesahin -- we're almost there 👍🏼

content/docs/start/visualization.md

jorgeorpinel · 2022-04-14T14:51:34Z

content/docs/start/visualization.md

+model. You can see the report in `training_metrics/index.html` with your
+browser:
+
+![dvclive](/img/start_visualization_dvclive.png)


Suggested change

![dvclive](/img/start_visualization_dvclive.png)

![DVCLive plots](/img/start_visualization_dvclive.png)

The image is not readable though. Can the fonts be larger?

And maybe the top part with dvclive.json sample should be a regular code block (under the image) with a few rows of the data instead of just one.

This is the same in the documentation: https://dvc.org/doc/dvclive/dvclive-with-dvc#html-report

Zooming in to make the axis labels readable makes the whole chart so large. How could we solve this @daavoo ?

We can still change the label as suggested for now.

same in the documentation: https://dvc.org/doc/dvclive/dvclive-with-dvc#html-report

Yes we have that problem in a few places but at least those are GIFs and you can see changing plots so maybe it doesn't matter that much that the text is readable in some other cases.

I think it's better to create separate issue or deal with this in #3455

Will be tracked in #3470

Yes but still the image caption could be better. It's helpful to have descriptive image captions for SEO and even image searches.

jorgeorpinel

(Minor things I'll commit myself)

content/docs/start/visualization.md

jorgeorpinel · 2022-04-18T22:25:16Z

Ping 🙂 @iesahin ☝🏼 please see #3050 (review)

Co-authored-by: Jorge Orpinel <jorgeorpinel@users.noreply.github.com>

iesahin · 2022-04-25T20:05:42Z

Closing in favor of #3455

content/docs/start/visualization.md

jorgeorpinel · 2022-04-26T17:31:20Z

Closing in favor of #3455

Not a great strategy. Makes reviewing that other PR much more complicated. Removes my contributions from commit history... Etc.

Let's reopen and finish this PR, and then reorganize once all the material is in?

jorgeorpinel

Just 3 review comments left to commit or decide on:

Nothing major so approving

gatsby-cloud · 2022-04-26T17:41:18Z

Gatsby Cloud Build Report

dvc.org

🎉 Your build was successful! See the Deploy preview here.

Build Details

View the build logs here.

🕐 Build time: 3m

jorgeorpinel · 2022-04-26T18:38:25Z

OK @iesahin up to you if you want to resolve the above comments and

a) close this one but copy/paste the final result into https://github.com/iterative/dvc.org/pull/3455/files#diff-e852faa513d0975fe637b21cd31c7530b2ae1d16cfd6e9e6132128c3397c9e03

or

b) merge this, and then merge master into #3455 + move the file to it's new location.

Whatever is easier for you. Thanks again

content/docs/start/visualization.md

Co-authored-by: Jorge Orpinel <jorgeorpinel@users.noreply.github.com>

This reverts commit 5ff8661.

* reorg #3 for trails * shortened data/ urls * renamed access * renamed pipelines * redirects from older URLs to new doc/start/data/ URLs * added trail section titles and minor revision * deleting forgotten metrics file * update visualization from #3050 * Update content/docs/sidebar.json Co-authored-by: Jorge Orpinel <jorgeorpinel@users.noreply.github.com> * Update redirects-list.json Co-authored-by: Jorge Orpinel <jorgeorpinel@users.noreply.github.com> * renamed to experiments * moved visualization to experiments/ * Update content/docs/start/index.md Co-authored-by: Jorge Orpinel <jorgeorpinel@users.noreply.github.com> * reverting the sidebar titles * Restyled by prettier (#3471) Co-authored-by: Restyled.io <commits@restyled.io> * renamed the directory Co-authored-by: Jorge Orpinel <jorgeorpinel@users.noreply.github.com> Co-authored-by: restyled-io[bot] <32688539+restyled-io[bot]@users.noreply.github.com> Co-authored-by: Restyled.io <commits@restyled.io>

shcheklein temporarily deployed to dvc-org-iesahin-gs-visu-lamemw November 30, 2021 15:40 Inactive

jorgeorpinel assigned iesahin Dec 1, 2021

shcheklein reviewed Dec 2, 2021

View reviewed changes

content/docs/start/visualization.md Outdated Show resolved Hide resolved

shcheklein reviewed Dec 2, 2021

View reviewed changes

content/docs/start/visualization.md Outdated Show resolved Hide resolved

shcheklein reviewed Dec 2, 2021

View reviewed changes

content/docs/start/visualization.md Outdated Show resolved Hide resolved

shcheklein temporarily deployed to dvc-org-iesahin-gs-visu-lamemw December 14, 2021 12:14 Inactive

shcheklein temporarily deployed to dvc-org-iesahin-gs-visu-lamemw December 14, 2021 12:26 Inactive

restyled-io bot mentioned this pull request Dec 14, 2021

Restyle [WIP] Add GS:Visualization and Plots #3095

Closed

shcheklein temporarily deployed to dvc-org-iesahin-gs-visu-lamemw December 15, 2021 15:47 Inactive

shcheklein reviewed Dec 16, 2021

View reviewed changes

shcheklein temporarily deployed to dvc-org-iesahin-gs-visu-lamemw December 17, 2021 17:22 Inactive

shcheklein temporarily deployed to dvc-org-iesahin-gs-visu-lamemw December 28, 2021 16:49 Inactive

shcheklein temporarily deployed to dvc-org-iesahin-gs-visu-lamemw January 5, 2022 13:24 Inactive

shcheklein temporarily deployed to dvc-org-iesahin-gs-visu-60h6xb April 13, 2022 15:42 Inactive

jorgeorpinel reviewed Apr 14, 2022

View reviewed changes

content/docs/start/visualization.md Outdated Show resolved Hide resolved

content/docs/start/visualization.md Outdated Show resolved Hide resolved

content/docs/start/visualization.md Outdated Show resolved Hide resolved

Apply suggestions from code review

b4681fe

jorgeorpinel deployed to dvc-org-iesahin-gs-visu-60h6xb April 14, 2022 15:00 View deployment

jorgeorpinel mentioned this pull request Apr 19, 2022

start: Reorganize into Trails (Candidate #1, Take #2) #3444

Closed

iesahin and others added 3 commits April 20, 2022 17:50

renamed confusion.png to misclassified.png in the example

6f5264d

rename one more

f26f366

Update content/docs/start/visualization.md

c1f4961

Co-authored-by: Jorge Orpinel <jorgeorpinel@users.noreply.github.com>

iesahin mentioned this pull request Apr 20, 2022

start: bi-trail reorg #3455

Merged

2 tasks

iesahin added a commit that referenced this pull request Apr 25, 2022

update visualization from #3050

577a05f

iesahin closed this Apr 25, 2022

jorgeorpinel reviewed Apr 26, 2022

View reviewed changes

content/docs/start/visualization.md Outdated Show resolved Hide resolved

iterative deleted a comment from iesahin Apr 26, 2022

jorgeorpinel reopened this Apr 26, 2022

jorgeorpinel approved these changes Apr 26, 2022

View reviewed changes

jorgeorpinel reviewed Apr 26, 2022

View reviewed changes

content/docs/start/visualization.md Show resolved Hide resolved

Update content/docs/start/visualization.md

b0ed669

Co-authored-by: Jorge Orpinel <jorgeorpinel@users.noreply.github.com>

iesahin mentioned this pull request Apr 27, 2022

start: Update dvclive screenshots for more readability in Visualization #3470

Closed

iesahin and others added 3 commits April 27, 2022 17:05

Update content/docs/start/visualization.md

7e411e2

Co-authored-by: Jorge Orpinel <jorgeorpinel@users.noreply.github.com>

Revert "removed older metrics, plots, vis from the sidebar"

cf02e81

This reverts commit 5ff8661.

readd metric-params-plots

b2a832e

iesahin merged commit db3ef44 into master Apr 27, 2022

iesahin deleted the iesahin/gs-visualization branch April 27, 2022 14:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add GS:Visualization and Plots #3050

Add GS:Visualization and Plots #3050

iesahin commented Nov 30, 2021 •

edited by jorgeorpinel

iesahin commented Dec 1, 2021

iesahin commented Dec 1, 2021

shcheklein commented Dec 2, 2021

shcheklein commented Dec 2, 2021

shcheklein commented Dec 2, 2021

iesahin commented Dec 14, 2021

iesahin commented Dec 14, 2021

shcheklein commented Dec 14, 2021

iesahin commented Dec 15, 2021

dberenbaum commented Dec 15, 2021

shcheklein Dec 16, 2021

shcheklein commented Dec 16, 2021

iesahin commented Dec 28, 2021

iesahin commented Dec 29, 2021 •

edited

dberenbaum commented Dec 29, 2021

iesahin commented Jan 4, 2022

iesahin commented Jan 10, 2022

iesahin commented Jan 12, 2022

jorgeorpinel left a comment

jorgeorpinel Apr 14, 2022 •

edited

iesahin Apr 20, 2022

jorgeorpinel Apr 26, 2022

iesahin Apr 27, 2022

iesahin Apr 27, 2022

jorgeorpinel May 1, 2022

jorgeorpinel left a comment

jorgeorpinel commented Apr 18, 2022

iesahin commented Apr 25, 2022

jorgeorpinel commented Apr 26, 2022 •

edited

jorgeorpinel left a comment

gatsby-cloud bot commented Apr 26, 2022 •

edited

jorgeorpinel commented Apr 26, 2022 •

edited

		@@ -0,0 +1,100 @@
		# Visualization with DVC Plots

		In parallel with versioning parameters and metrics by associating them with

	![dvclive](/img/start_visualization_dvclive.png)
	![DVCLive plots](/img/start_visualization_dvclive.png)

Add GS:Visualization and Plots #3050

Add GS:Visualization and Plots #3050

Conversation

iesahin commented Nov 30, 2021 • edited by jorgeorpinel

iesahin commented Dec 1, 2021

iesahin commented Dec 1, 2021

shcheklein commented Dec 2, 2021

shcheklein commented Dec 2, 2021

shcheklein commented Dec 2, 2021

iesahin commented Dec 14, 2021

iesahin commented Dec 14, 2021

shcheklein commented Dec 14, 2021

iesahin commented Dec 15, 2021

dberenbaum commented Dec 15, 2021

shcheklein Dec 16, 2021

Choose a reason for hiding this comment

shcheklein commented Dec 16, 2021

iesahin commented Dec 28, 2021

iesahin commented Dec 29, 2021 • edited

dberenbaum commented Dec 29, 2021

iesahin commented Jan 4, 2022

iesahin commented Jan 10, 2022

iesahin commented Jan 12, 2022

jorgeorpinel left a comment

Choose a reason for hiding this comment

jorgeorpinel Apr 14, 2022 • edited

Choose a reason for hiding this comment

iesahin Apr 20, 2022

Choose a reason for hiding this comment

jorgeorpinel Apr 26, 2022

Choose a reason for hiding this comment

iesahin Apr 27, 2022

Choose a reason for hiding this comment

iesahin Apr 27, 2022

Choose a reason for hiding this comment

jorgeorpinel May 1, 2022

Choose a reason for hiding this comment

jorgeorpinel left a comment

Choose a reason for hiding this comment

jorgeorpinel commented Apr 18, 2022

iesahin commented Apr 25, 2022

jorgeorpinel commented Apr 26, 2022 • edited

jorgeorpinel left a comment

Choose a reason for hiding this comment

gatsby-cloud bot commented Apr 26, 2022 • edited

Gatsby Cloud Build Report

Build Details

jorgeorpinel commented Apr 26, 2022 • edited

iesahin commented Nov 30, 2021 •

edited by jorgeorpinel

iesahin commented Dec 29, 2021 •

edited

jorgeorpinel Apr 14, 2022 •

edited

jorgeorpinel commented Apr 26, 2022 •

edited

gatsby-cloud bot commented Apr 26, 2022 •

edited

jorgeorpinel commented Apr 26, 2022 •

edited