Convert E2E tests from Puppeteer to Playwright #2041

mikkamp · 2023-08-03T14:04:40Z

Changes proposed in this Pull Request:

This PR converts the E2E tests from Puppeteer to Playwright and removes any dependencies on the @woocommerce/e2e-* packages. This is a followup from #2030

Note
Until we resolve issue #2002 we are still using Node 14, which means the latest compatible version of Playwright was 1.33. Ideally this will be updated once we upgrade Node.

Overview of the changes to convert tests:

Conversion ID
The test Conversion ID used to be set through the Connection Test page. This has now been replaced with a custom plugin tests/e2e/bin/test-data.php which adds some API endpoints to set this data.
Site Settings
The payment gateway COD used to be enabled through an API call at the start of the test, this is now done in the wp-env start up script.
API requests
All API requests use the basic auth plugin to send requests to the API. We use this for both WordPress and WooCommerce endpoints, see tests/e2e/utils/api.js
Avoiding waitForSelector
In our previous tests we often used waitForSelector, while this is still compatible with Playwright, I've replaced it with other Locators. This was done to simplify the tests and reduce the need for utility functions.
Mocking API requests
Instead of using a utility for mocking and spying on API requests, it's been replaced with the route.fulfill. We use this to mock connections to Jetpack or Google.

Closes #1599

Detailed test instructions:

Checkout this PR and install packages: nvm use && npm ci
Start the wp-env environment: npm run wp-env:up
Install the Chromium browser for Playwright: npx playwright install chromium
Start the E2E tests: npm run test:e2e
Confirm that all the tests pass, and a test report is available in tests/e2e/test-result
Check that we can run the tests in debug mode and step through them in the browser: npm run test:e2e-dev

Additional details:

Currently the E2E workflow is set to run only on release branches or manually. I've triggered a manual run, and all the tests passed: https://github.com/woocommerce/google-listings-and-ads/actions/runs/5753482975/job/15596778163

Changelog entry

Dev - Convert E2E tests from Puppeteer to Playwright.

codecov · 2023-08-03T14:10:02Z

Codecov Report

Merging #2041 (e7891a3) into develop (3ec92b8) will increase coverage by 0.1%.
The diff coverage is n/a.

❗ Current head e7891a3 differs from pull request most recent head 3bb245b. Consider uploading reports for the commit 3bb245b to get more accurate results

Additional details and impacted files

@@             Coverage Diff             @@
##             develop   #2041     +/-   ##
===========================================
+ Coverage       58.2%   58.3%   +0.1%     
+ Complexity      4119    4114      -5     
===========================================
  Files            454     454             
  Lines          17701   17676     -25     
===========================================
  Hits           10297   10297             
+ Misses          7404    7379     -25

Flag	Coverage Δ
php-unit-tests	`58.3% <ø> (+0.1%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed	Coverage Δ
src/ConnectionTest.php	`0.0% <ø> (ø)`

puntope · 2023-08-07T17:15:01Z

tests/e2e/bin/test-data.php

@@ -0,0 +1,55 @@
+<?php


I'm aware this is a test file but would be nice to document a bit the functions.

puntope · 2023-08-07T17:17:24Z

tests/e2e/config/playwright.config.js

+module.exports = defineConfig( {
+	testDir: '../specs',
+
+	/* Maximum time one test can run for. */


💅

/* Maximum time in milliseconds one test can run for. */

puntope · 2023-08-07T17:19:43Z

tests/e2e/config/playwright.config.js

+
+	expect: {
+		/**
+		 * Maximum time expect() should wait for the condition to be met.


Maximum time, in milliseconds, expect() should wait for the condition to be met.

puntope · 2023-08-07T17:21:52Z

tests/e2e/config/playwright.config.js

+	/* Fail the build on CI if you accidentally left test.only in the source code. */
+	forbidOnly: !! process.env.CI,


💅 I don't understand well this comment.

I will suggest the one appearing below on the web:
https://playwright.dev/docs/test-configuration#:~:text=Whether%20to%20exit%20with%20an%20error%20if%20any%20tests%20are%20marked%20as%20test.only.%20Useful%20on%20CI.

Whether to exit with an error if any tests are marked as "test.only"

Thanks, this is mostly copying and pasting from other examples. I think we can just remove this setting all together.

puntope · 2023-08-07T17:22:40Z

tests/e2e/config/playwright.config.js

+
+	/* Shared settings for all the projects below. See https://playwright.dev/docs/api/class-testoptions. */
+	use: {
+		/* Maximum time each action such as `click()` can take. Defaults to 0 (no limit). */


/* Maximum time in milliseconds each action such as click() can take. Defaults to 0 (no limit). */

puntope · 2023-08-07T17:28:05Z

tests/e2e/global-setup.js

+			console.log( 'Admin state file does not exist.' );
+		} else {
+			console.log( 'Admin state file could not be deleted: ' + err );
+		}


Maybe console.error might be more accurate here.

puntope · 2023-08-07T17:33:32Z

tests/e2e/global-setup.js

+			console.log(
+				`Admin log-in failed, Retrying... ${ i }/${ adminRetries }`
+			);
+			console.log( e );


Maybe we can use console.warn. Also, I don't understand well why we need to retry 5 times?

These are the parts that we need to do the same as WC core, but it isn't reusable to the same extent it was when we had the e2e packages. So it's basically copied code from here with the comment further down:

This step was failing occasionally, and globalsetup doesn't retry, so make it retry

I've had it fail in one of the test runs I did in GitHub Actions, so it must be a bit flakey at times (which is resolved with the retry).

Maybe I should add some references where the code is copied from to help explain. I can change the to console.warn / error where appropriate though.

puntope · 2023-08-07T18:05:43Z

One test failed for me in local

✘ 7 [chromium] › gtag-events/gtag-events.test.js:74:2 › GTag events › Add to cart event is sent from a block shop page (2.0m)

puntope · 2023-08-07T18:10:41Z

Same with the rest of Add to cart events...

I will try to check tomorrow why.

mikkamp · 2023-08-08T08:53:31Z

One test failed for me in local

As mentioned I'd suggest to run the tests from PR #2043 as I did improve a few tests that were still a bit flakey. If they still fail there then I'd suggest to run them with npm run test:e2e-dev and then step through them to see where they fail. It should show the selectors it's matched while doing so.

mikkamp · 2023-08-08T12:03:53Z

@puntope I went ahead and fixed those suggestions you made. Regarding running the tests maybe you can try with the following commands to start with a clean slate (I just ran that locally and all the tests pass):

npm run wp-env destroy
npm run wp-env:up
npm run test:e2e

puntope · 2023-08-08T16:46:18Z

Thanks for the changes @mikkamp Unfortunately it is still stuck in the waitForRequest call. (Even if I merged #2043

Screen.Recording.2023-08-08.at.18.35.54.mov

puntope · 2023-08-09T10:58:01Z

Some conflicts to be resolved in tests/e2e/docker/initialize.sh

But ready to go. The issue was on my side likely.

mikkamp added 28 commits August 3, 2023 07:13

Install playwright 1.33

81da4a2

Add starting PlayWright configuration

50f8211

Add global PlayWright setup script

f50217a

Convert getting started test

199a5ff

Convert setup MC step 1

a24308c

Remove unused utilities

da443ff

Ignore test results in git

73eb58d

Exclude rule jest/no-done-callback in E2E js files

89b8bc9

Remove creation of ready page

8b0d7a0

Add axios as a dev dependency

742aedc

Use plugin for setting test conversion data

6b5929b

Setup payment gateway for purchase

c33748d

Add price to simple product

0bdbea3

Convert GTag tests to PlayWright

a261ef3

Remove unused util functions

4d07484

Turn off parallel processing

1b7b5b2

Replace e2e scripts

93c50bf

Replace waitForSelector

e8d151a

Remove woocommerce/e2e packages

ba89d8f

Remove puppeteer config files

8b5edc7

Update E2E testing instructions in documentation

0551035

Remove unused puppeteer packages

94ad50d

Remove unused docker init file

0687ea2

Fix GitHub workflow for E2E tests

602ab18

Install chromium browser for playwright

173066c

Wait for all resources to be loaded

165e370

Use waitForURL after redirect

f0ffddd

Switch toBeEnabled

b7a5447

mikkamp self-assigned this Aug 3, 2023

mikkamp marked this pull request as ready for review August 3, 2023 16:57

mikkamp requested a review from a team August 3, 2023 16:57

mikkamp mentioned this pull request Aug 4, 2023

Allow E2E testing with Release Candidates #2043

Merged

mikkamp changed the title ~~Convert E2E tests from Puppeteer to PlayWright~~ Convert E2E tests from Puppeteer to Playwright Aug 4, 2023

This was referenced Aug 4, 2023

Add WP CLI apache modules #2045

Closed

404 errors on the frontend when using wp-env #2044

Closed

puntope reviewed Aug 7, 2023

View reviewed changes

mikkamp added 3 commits August 8, 2023 12:50

Clarify comments in playwright config

2a773ad

Use console warn and error

3c51032

Add some function descriptions

e7891a3

mikkamp requested a review from puntope August 8, 2023 12:04

puntope approved these changes Aug 9, 2023

View reviewed changes

Merge develop into e2e/playwright-tests

3bb245b

mikkamp merged commit 92d15c7 into develop Aug 9, 2023
13 checks passed

mikkamp deleted the e2e/playwright-tests branch August 9, 2023 15:51

eason9487 added the changelog: dev Developer-facing only change. label Aug 15, 2023

eason9487 mentioned this pull request Aug 22, 2023

Release 2.5.3 #2067

Merged

17 tasks

puntope mentioned this pull request Aug 29, 2023

- #2079

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Convert E2E tests from Puppeteer to Playwright #2041

Convert E2E tests from Puppeteer to Playwright #2041

mikkamp commented Aug 3, 2023 •

edited

codecov bot commented Aug 3, 2023 •

edited

puntope Aug 7, 2023

puntope Aug 7, 2023

puntope Aug 7, 2023

puntope Aug 7, 2023

mikkamp Aug 8, 2023

puntope Aug 7, 2023

puntope Aug 7, 2023

puntope Aug 7, 2023

mikkamp Aug 8, 2023

puntope commented Aug 7, 2023

puntope commented Aug 7, 2023

mikkamp commented Aug 8, 2023

mikkamp commented Aug 8, 2023

puntope commented Aug 8, 2023

puntope commented Aug 9, 2023

		/* Fail the build on CI if you accidentally left test.only in the source code. */
		forbidOnly: !! process.env.CI,

Convert E2E tests from Puppeteer to Playwright #2041

Convert E2E tests from Puppeteer to Playwright #2041

Conversation

mikkamp commented Aug 3, 2023 • edited

Changes proposed in this Pull Request:

Detailed test instructions:

Additional details:

Changelog entry

codecov bot commented Aug 3, 2023 • edited

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

puntope commented Aug 7, 2023

puntope commented Aug 7, 2023

mikkamp commented Aug 8, 2023

mikkamp commented Aug 8, 2023

puntope commented Aug 8, 2023

puntope commented Aug 9, 2023

mikkamp commented Aug 3, 2023 •

edited

codecov bot commented Aug 3, 2023 •

edited