[QA] Visual Test Addon

[vanessayuenn]

Visual Test Addon

Feature owner: @ghengeveld

Please comment in this discussion thread for any bug report and feedback for this feature as you test out Storybook 8. Kindly adhere to the following format:

Project: Name of your project as entered in Notion
Version: The version of storybook you tested with
Bug/feedback: Details of the bugs/feedback you have

Feature owner should triage the comments here frequently and open issues as needed.

[JReinhold]

Project: None, fresh project
Version: 8.0.0-alpha.13, VTA 0.0.124
Bug/feedback:

I tested this in a freshly initialized project. The addon panel had the following message:

The link goes to https://www.chromatic.com/docs/visual-tests-addon/ which doesn't contain any information about requirements as far as I can tell.

I then initiated git (git init) and restarted SB, the message persisted. I also created an initial commit, the message persisted. I had to create a remote for the message to go away.

[thafryer]

This issue is now resolved in @chromatic-com/storybook@1.1.0.

[JReinhold]

Project: None, fresh project
Version: 8.0.0-alpha.13, VTA 0.0.124
Bug/feedback:

Icons are missing in the bottom right context menu, replaced with text.

[thafryer]

This issue is now resolved in @chromatic-com/storybook@1.1.0.

[JReinhold]

Project: None, fresh project
Version: 8.0.0-alpha.13, VTA 0.0.124
Bug/feedback:

The play button in the top left corner is missing. It was there a minute ago, but then after testing login and restarting SB a couple of times, it's gone.

[yannbf]

+1 to this, I couldn't find a way to run tests in MealDrop when installing the addon

[yannbf]

In MealDrop, I can't proceed testing because the "Run tests" button does not exist at all.

This is in Storybook 7.6.10

This is in Storybook 8

I tested versions of the addon ranging from 0.0.70 until 0.0.124, none of which had the button in the sidebar.

[shilman]

I've never seen the run tests sidebar button in SB8 in Storyblok. Furthermore, there's also a run test button in the addon panel but I can't figure out the heuristics for when it shows vs not.

[thafryer]

This issue is now resolved in @chromatic-com/storybook@1.1.0, through Norbert's PR to Storybook.

[JReinhold]

Project: None, fresh project
Version: 8.0.0-alpha.13, VTA 0.0.124
Bug/feedback:

There are changes, but there is no "X changes" button at the bottom of the sidebar.

[thafryer]

@JReinhold Can you share more detailed repro steps for how you encountered this? We haven't encountered this (on 0.0.124) but may have done some steps differently than you.

[shilman]

Project Storyblok
Version 8.0.0-alpha.15 / 0.0.124 addon

I don't see the buttons at the bottom. And I don't even see the orange dots to indicate changes. I think this might be because two of my tests failed (out of 200). I am not able to explore more because the play button has disappeared and I don't know how to get it back.

[thafryer]

I believe this issue is now resolved in @chromatic-com/storybook@1.1.0, through Norbert's PR to Storybook. @shilman and @JReinhold Can you confirm?

[shilman]

I did not test the most recent version, but it was fixed in an earlier version (on the storybook side, i believe) so I think we're good

[JReinhold]

Project: None, fresh project
Version: 8.0.0-alpha.13, VTA 0.0.124
Bug/feedback:

After running a succesful build I see this in the terminal:

⚠ Using outdated package
You are using an outdated version of the Chromatic CLI (v9.1.0).
A new major version is available (v10.4.0).
Run yarn upgrade chromatic --latest to upgrade.

1. I didn't add the Chromatic CLI manually at all, it was all automatic, so it seems strange to me that an outdated version got added
2. This project uses pnpm and not Yarn, so the upgrade command is wrong

[yannbf]

there's a chance this command uses npx internally and therefore ended up using a cached version from your machine

[shilman]

I saw the same issue @JReinhold even after I upgraded on my machine & restarted Storybook. So I think it's something about the addon's dependencies.

[thafryer]

@shilman @JReinhold, Can you confirm this is still an issue in @chromatic-com/storybook@1.1.0?

[JReinhold]

Project: None, fresh project
Version: 8.0.0-alpha.13, VTA 0.0.124
Bug/feedback:

After running a succesful build I see this in the terminal:

✔ Build 2 passed!
1 visual change were found in this build.
ℹ View build details at https://www.chromatic.com/build?appId=65b0247591dbe738282587b4&number=2

⚠ No 'chromatic' script found in your package.json
Would you like me to add it for you? [y/N]

I can't interact with the CLI so pressing Y or N doesn't have an effect. The prompt is useless.

[yannbf]

also, small typo in the CLI message: 1 visual change were found -> 1 visual change was found and potentially when there's more than one it will show 2 visual change were found -> 2 visual changes were found

[thafryer]

The issue of the CLI prompt has been resolved in @chromatic-com/storybook@1.1.0

[yannbf]

Project: None, fresh project
Version: 8.0.0-alpha.13, VTA 0.0.124
Bug/feedback:

There's a misalignment with the helper icon as the text in the second row is too long (and the "NEW" tag is too big maybe)

[yannbf]

Project: MealDrop
Version: 8.0.0-alpha.14
Bug/feedback:

When registering the addon, the browser is flooded with errors (same error, multiple times):

Warning: The final argument passed to useMemo changed size between renders. The order and size of this array must remain constant.

Previous: []
Incoming: [[object Object]]

Repro:

1. Go to https://github.com/yannbf/mealdrop/tree/bug/vta-addon
2. Run Storybook
3. Check the dev tools for errors
4. Remove the addon
5. See that the errors are gone

[vanessayuenn]

This might be a Storybook error #24265

[thafryer]

This no longer seems to be an issue in @chromatic-com/storybook@1.1.0. There are different console warnings now, though. I will make sure to note those for further resolution.

(cc: @vanessayuenn)

[yannbf]

General feedback:

Seems like the README is outdated and references configuration via main.js instead of chromatic.config.json

There are some missing configuration options too.
https://github.com/chromaui/addon-visual-tests#configuration

[yannbf]

I guess the README could just point to the docs?

[domyen]

I think this has been updated now (latest configuration docs)

[yannbf]

General feedback:

When using the addon, there are many logs which appear together with Storybook, but they don't make much sense in the context of using the addon. The addon should invoke Chromatic's CLI with some sort of logLevel flag (maybe even supress all logs), or specific flag that removes logs that won't make sense in the context of the addon. A few examples:

⚠ Using outdated package
You are using an outdated version of the Chromatic CLI (v9.1.0).
A new major version is available (v10.6.0).
Run yarn upgrade chromatic --latest to upgrade.

Your project is linked to GitHub so Chromatic will report results there.
This means you can pass the --exit-once-uploaded flag to skip waiting for build results.
Read more here: https://www.chromatic.com/docs/cli#chromatic-options

✖ Encountered 5 build errors: failing with exit code 2
Pass --allow-console-errors to succeed this command regardless of runtime build errors.
ℹ Review the errors at xyz

[shilman]

Project Storyblok
Version 8.0.0-alpha.15, 0.0.124 of addon

I hit the "Take snapshots" button, but the Storybook dev server was not running (I guess I'd shut it down for some reason). The UI reported "Waiting for build on develop" per the screenshot. However, it should have errored instead since it needs the dev server to be running.

[shilman]

Project Storyblok
Version 8.0.0-alpha.15 / 0.0.124 addon

When you run a build from the addon and you don't have a chromatic script in your package.json, it prompts you:

The build should run in the background and shouldn't prompt you.

[vanessayuenn]

same as this #25693 (comment)?

[thafryer]

@shilman This has been resolved in @chromatic-com/storybook@1.1.0

(cc: @vanessayuenn)

[shilman]

Project Storyblok
Version 8.0.0-alpha.15 / 0.0.124 addon

Addon brings in SB 7.6 dependencies which show up in storybook doctor. It should depend on 8.0.x versions:

[thafryer]

@shilman To verify whether this is still an issue, what do I need to do? I am testing using @chromatic-com/storybook@1.1.0

[thafryer]

Project: Fresh, via SB Init
Version: SB ^8.0.0-beta.0, VTA 1.1.0
Bug/feedback: Currently, you cannot close notifications by clicking on the "X" icon.

To Reproduce,

1. Run a build.
2. Click the the Close Icon.

[vanessayuenn]

tracked in #26036

[thafryer]

Project: Fresh, via SB Init
Version: SB ^8.0.0-beta.0, VTA 1.1.0
Bug/feedback: The sidebar bottom area does not match the desired height.

https://www.loom.com/share/183feeb50c164bb28d90ab55460aa154

[thafryer]

Note that we likely plan to make a change in VTA for this. However, Design folks may want to finalize the bottom height before release.

[domyen]

This bug is being tracked in VTA project

[valentinpalkovic]

Cross reference: #25695 (comment)

Indirectly relates to the VTA, but it is affected by the —test flag issue I have linked.