Galata failures on master, lack of artifacts, empty report

[krassowski]

Description

• After merging Add Galata in JupyterLab #10796 there are failures on master
• I tried to download ui-test-output artifacts as instructed in https://jupyterlab.readthedocs.io/en/latest/developer/contributing.html#visual-regression-and-ui-tests but this artifact is no longer there; the two artifacts that are downloadable contain:
  • galata-test-assets: it only contains .webm movies, and I cannot easily find the folder for the movie for the failing test case
  • galata-report: .webm movies and some .png files, but those are not helpful due to their names
    
• the report is empty for me, even though there are many files (as shown above):
  

[fcollonval]

After merging #10796 there are failures on master

I'm working on #10950 to make them more robust

I tried to download ui-test-output artifacts as instructed in https://jupyterlab.readthedocs.io/en/latest/developer/contributing.html#visual-regression-and-ui-tests but this artifact is no longer there. The two artifacts that are downloadable contain:
galata-test-assets: it only contains .webm movies, and I cannot easily find the folder for the movie for the failing test case
galata-report: .webm movies and some .png files, but those are not helpful due to their names

The two artifacts are now:

• galata-test-assets: Raw artifacts with one folder per tests - artifacts are videos and some images if an error occurred when comparing screenshots. Each folder is built using the same pattern <path-to-test-file>-<dashed-test-title>
  We can change the default settings to retain only the failing test as done previously by galata.
• galata-report: The report needs to be started with a server within the extracted folder. npx http-server -a localhost -o <path-to-report> - this is required as the report data is in JSON format loaded using fetch.

I opened an issue upstream to at least display an error in that case (microsoft/playwright#8549)

A possible workaround would be to build our own report template (I tried to avoid that to reduce our maintenance load).

[krassowski]

I'm working on #10950 to make them more robust

🎉

We can change the default settings to retain only the failing test as done previously by galata.

The previous instructions were very simple (download a folder, copy-paste). It was a good developer experience (running the docker locally was a bit tedious, so I came to prefer using CI).

galata-report: The report needs to be started with a server within the extracted folder. npx http-server -a localhost -o <path-to-report> - this is required as the report data is in JSON format loaded using fetch.

I would prefer to be able to just open it in browser, but I guess that for now adding a README.txt with this instruction would suffice.

[jtpio]

I would prefer to be able to just open it in browser, but I guess that for now adding a README.txt with this instruction would suffice.

Ideally we can find a way to make the report more easily accessible, for example by publishing the content as a PR comment: #10968

[jtpio]

@krassowski there have been a couple of iterations on the Galata setup since this issue was opened.

How do the report / process / artifacts look like to you? Do you think most of the items mentioned above have been addressed?

[krassowski]

Do we track making a bot command to update screenshots? I remember someone (you?) mentioned it in a comment on a PR, but cannot find it.

Let's close this one.