test(DIST-136): Increase resilience for visual tests in firefox #2

antoniofull · 2020-04-13T13:18:15Z

Increase resilience for visual tests

Refactor Embed Visual tests
Uses a non-animated form with an start button (the initial animation on the form causes the issue, since when the screenshot is taken the animation has started the elements are moving and so we have px changed each time)
Changes demo page title to be modified when the form is loaded (so we can test
Adds custom step that returns when the page was modified so we know the iframe is loaded
Disables tracking and animations on forms (although it does not seem to work)
Splits visual tests in mobile and desktop
Splits Travis job to run Firefox isolated in parallel
Makes all steps asynchronous
Removes Safari visual tests
Updates README
Explain the motivation for making this change.
Provide a test plan demonstrating that the code is solid.
Match the code formatting of the rest of the codebase.
Two other colleagues must approve this change.
QA have tested the solution provided.
Design and UX must validate this change.

antoniofull · 2020-04-13T13:18:49Z

.travis.yml

+        - yarn test:visual:mobile
+        - kill $EMBED_PID
+    - name: "Visual Tests Firefox (Internal)"
+      if: fork = false AND (branch = master OR branch = release)
+      script:
+        - yarn start --no-info & export EMBED_PID=$!
+        - yarn test:visual:install


Runs firefox visual tests in parallel on travis (speed up tests)

Yes, we can run tests in parallel for each browser 👍 I suggest you run yarn test:visual:mobile separately as well.

Sounds good I wasn't sure if creating many jobs in parallel would cause issues, thanks!

I dont think it would. Each job needs to clone the repo and install dependencies - if the task is short it usually should not have a separate job. If it takes longer than cloning and installing and has no dependencies it can be a separate job.

I think I would also split functional tests per browser to make it faster.

👍 give me 10 minutes and I'll push the changes

One build failed because both tests were running in parallel, I wonder if makes more sense to run one job with both tests in parallel (as @Gotusso set it up) or separated? I am not sure about it. Thoughts @mathio ? (right now I split them and removed the parallel flag)

Maybe try without the --parallel flag for cypress? But I would check with Franco first. So for now feel free to keep this as is.

Edit: I re-run the tests and they passed 🤷‍♂️ When run in separate Travis jobs it saves ~30 seconds of total run time.

README.md

antoniofull · 2020-04-13T13:20:10Z

demo/demo.js

+  // Changes the document title when the form is ready
+  if (event.data.type === 'form-ready') {
+    window.document.title = event.data.type
+  }


Once the form is loaded, the page will receive a postMessage with data form-ready we change the title of the document to match the event, like so we have a better idea that the form is ready without having to dig in to the form itself.

demo/demo.js

antoniofull · 2020-04-13T13:20:58Z

demo/full.html

@@ -27,7 +27,7 @@
      width="100%"
      height="100%"
      frameborder="0"
-      src="https://admin.typeform.com/to/pDgR2Z"
+      src="https://admin.typeform.com/to/mzy0hT?disable-tracking=true"


We use a static form without initial animation, since this seems to be the main issue

demo/popup.html

demo/widget.html

e2e/codeceptjs-utils.js

cypress · 2020-04-13T13:22:30Z

Test summary

11 • 0 • 0 • 0

Run details


Project	embed
Status	Passed
Commit	`6746522` ℹ️
Started	Apr 14, 2020 2:10 PM
Ended	Apr 14, 2020 2:11 PM
Duration	01:08 💡
OS	Linux Ubuntu Linux - 14.04
Browser	Firefox 67

View run in Cypress Dashboard ➡️

This comment has been generated by cypress-bot as a result of this project's GitHub integration settings. You can manage this integration in this project's settings in the Cypress Dashboard

antoniofull · 2020-04-13T13:23:21Z

e2e/custom-steps.js

+    },
+    async waitForIFrameMessage () {
+      this.executeAsyncScript(function (done) {
+        const interval = setInterval(function () {
+          if (window.document.title === 'form-ready') {
+            clearInterval(interval)
+            done()
+          }
+        }, 100)
+      })


We cannot use directly window object with codeceptjs and webdriver, so we use executeAsyncScript to run a function that checks at intervals when the title of the document is changed. see previous comments

antoniofull · 2020-04-13T13:24:04Z

e2e/cypress-utils.js

@@ -33,7 +33,7 @@ export const setupIframeTesting = (iframe) => {
  // We can only use it inside Chrome disabling web security --> see config
  // Won't work in popups
  if (Cypress.isBrowser('chrome')) {
-    getIframeBody(iframe).find('[data-qa="fixed-footer-progress"]').should('be.visible')
+    getIframeBody(iframe).find('[data-qa="start-button"]').should('be.visible')


NIT: Since we changed form, we still want to have some resilience to make sure the form is loaded internally, the new form has a start button so we check for it. This is for functional tests

e2e/visual-helper.js

e2e/visual/embed.js

package.json

antoniofull · 2020-04-13T13:26:54Z

package.json

+    "test:visual:chrome": "yarn codeceptjs --config codecept-visual.conf.js run --steps --grep @desktop",
+    "test:visual:firefox": "yarn codeceptjs --config codecept-visual.conf.js run --steps --profile firefox --grep @desktop",
+    "test:visual:mobile": "yarn codeceptjs --config codecept-visual.conf.js run --steps --grep @mobile",


Splits the visual tests to run separated for chrome, firefox and mobile

antoniofull · 2020-04-13T13:31:30Z

Test summary

8 • 3 • 0 • 0

Run details

Project embed
Status Failed
Commit 5cd5a79 ℹ️
Started Apr 13, 2020 1:21 PM
Ended Apr 13, 2020 1:22 PM
Duration 00:53 💡
OS Linux Ubuntu Linux - 14.04
Browser Firefox 75
View run in Cypress Dashboard ➡️

Failures

right-drawer.spec.js ⏯ 1
1 Right Drawer Embed Widget > Closes Embed Right Drawer using Keyboard
drawer.spec.js ⏯ 1
1 Drawer Embed Widget > Closes Embed Drawer using Keyboard
popup.spec.js ⏯ 1
1 Popup Embed Widget > Closes Embed Popup using Keyboard
This comment has been generated by cypress-bot as a result of this project's GitHub integration settings. You can manage this integration in this project's settings in the Cypress Dashboard

@Typeform/distribution there is a known issue with Firefox75 looking in to it

antoniofull · 2020-04-13T13:55:21Z

I have updated Travis to install Firefox v 67.0 which matches the one we use in Applitools

maxprilutskiy

Looks good to me, approved

.travis.yml

mathio · 2020-04-13T15:25:46Z

.travis.yml

+        - yarn test:visual:mobile
+        - kill $EMBED_PID
+    - name: "Visual Tests Firefox (Internal)"
+      if: fork = false AND (branch = master OR branch = release)
+      script:
+        - yarn start --no-info & export EMBED_PID=$!
+        - yarn test:visual:install


Maybe try without the --parallel flag for cypress? But I would check with Franco first. So for now feel free to keep this as is.

Edit: I re-run the tests and they passed 🤷‍♂️ When run in separate Travis jobs it saves ~30 seconds of total run time.

antoniofull · 2020-04-13T17:28:36Z

@mathio the tests pass, the question I think is more if we want to keep them in parallel (one travis job) or not ( separate travis jobs). While we might earn 30 seconds on the builds, keeping them parallels might be better solution for cypress logging/dashboard part.
I leave this pr open for now, let's check tomorrow with @Gotusso, is just a small detail I can change it quickly in both cases.

test(DIST-136): Fixes firefox75 bug test(DIST-136): Fixes firefox version test(DIST-136): Reverts functional tests test(DIST-136): Split tests in separate jobs test(DIST-136): Split tests in separate jobs Update .travis.yml Co-Authored-By: Matej Lednicky <matej.lednicky@typeform.com> test(DIST-136): Run functional tests in parallel Update .travis.yml Co-Authored-By: Matej Lednicky <matej.lednicky@typeform.com>

antoniofull requested a review from a team April 13, 2020 13:18