E2E Smoke test #4497

dac09 · 2022-02-16T10:09:57Z

Smoke tests for `rw dev` and `rw serve`

This PR does the following:

Modifies the test:project generator to copy files from __fixtures__/test-project. This can be disabled (i.e. you can rebuild, if you run it with --no-copy-from-fixtures
Adds playwright tests to do some simple checks on the test project
Updates cli checks workflow to become the smoke-test workflow

To run the smoke test locally, you need to do the following
a) Create a test project e.g. yarn build:test-project ~/Experiments/copy-guy-3 --ts --verbose --no-canary
b) and run PROJECT_PATH="~/Experiments/copy-guy-3" yarn smoke-test --headed.

Note: if you get a failure first time running it because of the playwright dependencies, please make a note of the command playwright tells you to run! I have install deps as part of the smoke-test script, but can't verify on my machine because I have the dependencies.

Other Stuff:
I also tried adding some visual snapshot tests, to make sure css/tailwind/etc are all working as expected - however I had to disable them as the snapshots I produced in gitpod seem to differ from the github runner ones, possibly due to fonts.

Mac and windows work though.

Maybe someone on linux could help with this? Not a requirement to merge the PR

Related to #4500

Closes #4259

Setup build:testproject script to copy from fixtures

Correct smoke-test path

Update timeout

dac09 · 2022-02-17T11:39:50Z

.github/actions/setup_test_project/setup_test_project.mjs

@@ -18,26 +17,5 @@ console.log({

 core.setOutput('test_project_path', test_project_path)

-// See https://github.com/actions/toolkit/tree/main/packages/io#cpmv.


This is now taken care of by yarn build:test-project

dac09 · 2022-02-17T11:40:07Z

.github/workflows/smoke-test.yaml

        run: |
          yarn install --immutable
-          yarn build:clean || echo "Project already clean"


This is also done as part of the test project generator

.github/workflows/smoke-test.yaml

dac09 · 2022-02-17T11:52:23Z

@jtoar, @thedavidprice ready for review.

We can build on top of this, as we see a need for more coverage!

Only problem I see is that the CLI checks take decades to complete on windows...

jtoar

@dac09 this is awesome! I really like how the fixtures let us couple all the server-starting logic with the test itself. 🚀

I'm still getting my head around what's going on, but I left a few initial comments. One more I had that wasn't specific to any particular line of code: since the processes are running in parallel and both are piping to stdout, do you think we can do something to make the output a little easier to follow, mostly for debugging purposes? I'll suggest an idea if I come up with one.

tasks/smoke-test/playwright.config.ts

tasks/smoke-test/playwright-fixtures/rwServe.fixture.ts

tasks/smoke-test/util.ts

tasks/test-project/test-project

jtoar · 2022-02-18T09:44:53Z

@dac09 mostly just throwing this out there, but what if we made the test project a playwright fixture too? That way playwright could handle everything.

tasks/smoke-test/playwright-fixtures/devServer.fixture.ts

thedavidprice · 2022-02-18T21:18:13Z

Nice @dac09 I'll take this for a review + spin on Monday!

…lready

…test * 'main' of github.com:redwoodjs/redwood: (23 commits) Netlify client getToken fix when GoTrue client refreshes JWT (redwoodjs#4539) Update dependency @supabase/supabase-js to v1.30.4 (redwoodjs#4536) Envelop: Don't use useImmediateIntrospection as it causes auth bug (redwoodjs#4538) Update dependency react-hook-form to v7.27.1 (redwoodjs#4521) try increasing timeout for flaky test (redwoodjs#4526) Update dependency stacktracey to v2.1.8 (redwoodjs#4519) Update dependency msw to v0.38.1 (redwoodjs#4525) Update dependency eslint-config-prettier to v8.4.0 (redwoodjs#4522) Provide a Revised Runtime Error Page (redwoodjs#4167) update yarn.lock v0.46.0 Update dependency esbuild to v0.14.23 (redwoodjs#4518) Fix Storybook build args (redwoodjs#4455) Update dependency react-helmet-async to v1.2.3 (redwoodjs#4502) Bump url-parse in /__fixtures__/example-todo-main-with-errors (redwoodjs#4511) Bump url-parse from 1.5.1 to 1.5.7 in /__fixtures__/example-todo-main (redwoodjs#4512) Update dependency fastify to v3.27.2 (redwoodjs#4516) Uncomment role checks (redwoodjs#4476) Update dependency zx to v5.1.0 (redwoodjs#4505) Update dependency firebase to v9.6.7 (redwoodjs#4514) ...

tasks/test-project/test-project

thedavidprice

Woah, DC, you really did a great job stitching together the elements here. Much much better than what I originally had in my mind.

I haven't run this locally yet — just stepped through code and looked through CI runs. Only question since I'm not familiar with Playwright --> have you tested a failure on rw dev or rw serve to make sure the Smoke Test fails the CI workflow?

Left some comments mostly about Telemetry related needs.

Thanks so much!

thedavidprice · 2022-02-22T07:09:07Z

tasks/test-project/test-project

+      {
+        title: 'Updating ports in redwood.toml...',
+        task: () => {
+          const REDWOOD_TOML_PATH = path.join(
+            OUTPUT_PROJECT_PATH,
+            'redwood.toml'
+          )
+          const redwoodToml = fs.readFileSync(REDWOOD_TOML_PATH).toString()
+          let newRedwoodToml = redwoodToml
+
+          newRedwoodToml = newRedwoodToml.replace(
+            /\port = 8910/,
+            'port = "${WEB_DEV_PORT:8910}"'
+          )
+
+          newRedwoodToml = newRedwoodToml.replace(
+            /\port = 8911/,
+            'port = "${API_DEV_PORT:8911}"'
+          )
+
+          fs.writeFileSync(REDWOOD_TOML_PATH, newRedwoodToml)
+        },
+      },


Could this go in the Playwright fixtures instead of here, where it will be included in every built Test-project (for both CI and local dev)?

Ummmm in theory... yes I could put it in playwright, but then I'd have to check if the toml is already modified.
However, this change, as you can see has the fallback to the usual 8910 and 8911. So normal test projects continue to work, even without the env vars defined

but then I'd have to check if the toml is already modified

Not sure what this means as the toml will be know 'cause it's coming from the fixture?

Not a dealbreaker. Just feels like a weird artifact for all the local dev usage. If you decide it's best to leave here, then maybe it deserves a comment to explain why it's there in the test project?

I've added a comment explaining why. I think given that this change has no impact unless you're actually setting the env var, I prefer to keep it in the test project generator, mainly so that the change is applied once.

Playwright fixtures can run multiple times, on different workers, which is why we'd have to add a check to see if the toml is already modified, which feels a little unnecessary to me.

.github/actions/setup_test_project/setup_test_project.mjs

.github/workflows/smoke-test.yaml

thedavidprice · 2022-02-23T00:50:25Z

@dac09 Just wanted to make sure you saw all my misc comments throughout from yesterday.

dac09 · 2022-02-23T12:18:20Z

@jtoar, @thedavidprice - all comments addressed from both of you - thank you for the in-depth reviews!

I tried enabling the visual test, but I'm afraid with the fonts varying ever so slightly, the snapshots don't match up, so best not to have a potentially flakey test in there

thedavidprice · 2022-02-23T18:05:31Z

Hellz. Yeah.

jtoar assigned dac09 Feb 16, 2022

dac09 added 2 commits February 16, 2022 23:31

Setup playwright smoke tests

130ed17

Setup build:testproject script to copy from fixtures

Try adding smoke test

f19dd75

dac09 force-pushed the chore/smoke-test branch from eb0758e to f19dd75 Compare February 17, 2022 07:53

dac09 added the release:chore This PR is a chore (means nothing for users) label Feb 17, 2022

dac09 added 10 commits February 17, 2022 15:03

No need to build fw first

f966b7e

Correct smoke-test path

Try playwright install

a1c8378

Update playwright install cmd

45d1bc3

Remove copy from setup_test_project

80869b9

Add serve tests

244ff66

Update timeout

Add visual check, Renable cli checks

a109349

Add linux snapshot

2f4c633

Add windows snapshot | try caching playwright binaries

e1d75c0

Move visual checks to rw serve, reduce threshold

521290d

Remove visual check for now

788c6eb

dac09 marked this pull request as ready for review February 17, 2022 11:39

dac09 requested review from thedavidprice and jtoar February 17, 2022 11:39

dac09 commented Feb 17, 2022

View reviewed changes

.github/workflows/smoke-test.yaml Show resolved Hide resolved

Merge branch 'main' into chore/smoke-test

04d0e97

jtoar reviewed Feb 18, 2022

View reviewed changes

jtoar mentioned this pull request Feb 18, 2022

[CI] Improve (and complete) new CLI check #4500

Closed

11 tasks

jtoar reviewed Feb 18, 2022

View reviewed changes

tasks/smoke-test/util.ts Show resolved Hide resolved

jtoar reviewed Feb 18, 2022

View reviewed changes

tasks/test-project/test-project Outdated Show resolved Hide resolved

jtoar reviewed Feb 18, 2022

View reviewed changes

tasks/smoke-test/playwright-fixtures/devServer.fixture.ts Outdated Show resolved Hide resolved

dac09 added 8 commits February 21, 2022 13:10

Add top level script to run smoke test

b8692eb

Cleanup

55a322f

Prepend subprocess outputs for tests | Build for serve if not built a…

5eed832

…lready

Specify import type

84f1c69

Refine test-project script

4f4bff0

Sort package.json

d16ff10

Merge branch 'main' into chore/smoke-test

86002ef

thedavidprice reviewed Feb 22, 2022

View reviewed changes

tasks/test-project/test-project Show resolved Hide resolved

thedavidprice reviewed Feb 22, 2022

View reviewed changes

dac09 added 9 commits February 23, 2022 17:08

Disable telemetry on setting up test project

7a2b610

Address more comments from DP

64aef7f

Merge branch 'main' into chore/smoke-test

9497ba9

Fix lockfile

1158df4

Match up fs-extra versions

48189d7

Disable telemetry for building test project a different way

34f512f

Remove yarn install in test-project generator

38f0746

Update linux snapshot using codespaces

c865f9e

Remove snapshot comparison

a41f702

dac09 and others added 2 commits February 23, 2022 19:19

Remove unused snapshots

e98f4bf

Merge branch 'main' into chore/smoke-test

88fef62

thedavidprice merged commit 93c6ba9 into redwoodjs:main Feb 23, 2022

jtoar added this to the next-release milestone Feb 23, 2022

dac09 deleted the chore/smoke-test branch February 23, 2022 18:37

thedavidprice modified the milestones: next-release, v0.47.0 Feb 24, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

E2E Smoke test #4497

E2E Smoke test #4497

dac09 commented Feb 16, 2022 •

edited

dac09 Feb 17, 2022

dac09 Feb 17, 2022

dac09 commented Feb 17, 2022

jtoar left a comment

jtoar commented Feb 18, 2022

thedavidprice commented Feb 18, 2022

thedavidprice left a comment

thedavidprice Feb 22, 2022

dac09 Feb 22, 2022

thedavidprice Feb 23, 2022

dac09 Feb 23, 2022

thedavidprice commented Feb 23, 2022

dac09 commented Feb 23, 2022 •

edited

thedavidprice commented Feb 23, 2022

		@@ -18,26 +17,5 @@ console.log({

		core.setOutput('test_project_path', test_project_path)

		// See https://github.com/actions/toolkit/tree/main/packages/io#cpmv.

E2E Smoke test #4497

E2E Smoke test #4497

Conversation

dac09 commented Feb 16, 2022 • edited

Smoke tests for rw dev and rw serve

dac09 Feb 17, 2022

Choose a reason for hiding this comment

dac09 Feb 17, 2022

Choose a reason for hiding this comment

dac09 commented Feb 17, 2022

jtoar left a comment

Choose a reason for hiding this comment

jtoar commented Feb 18, 2022

thedavidprice commented Feb 18, 2022

thedavidprice left a comment

Choose a reason for hiding this comment

thedavidprice Feb 22, 2022

Choose a reason for hiding this comment

dac09 Feb 22, 2022

Choose a reason for hiding this comment

thedavidprice Feb 23, 2022

Choose a reason for hiding this comment

dac09 Feb 23, 2022

Choose a reason for hiding this comment

thedavidprice commented Feb 23, 2022

dac09 commented Feb 23, 2022 • edited

thedavidprice commented Feb 23, 2022

dac09 commented Feb 16, 2022 •

edited

Smoke tests for `rw dev` and `rw serve`

dac09 commented Feb 23, 2022 •

edited