devops: refactor GitHub Actions workflows #30524

mxschmitt · 2024-04-24T21:47:15Z

https://github.com/microsoft/playwright-internal/issues/203

.github/workflows/tests_primary.yml

dgozman · 2024-04-24T23:54:14Z

.github/workflows/tests_primary.yml

+        is-linux: true
+        validate-coverage: ${{ matrix.browser }}
+        browser-to-install: ${{ matrix.browser }}
+        command: npm run test -- --project=${{ matrix.browser }}-*


We'll also need env for trace/video/etc bots.

I'd also make "PWTEST_BOT_NAME" an argument.

I think passing the env is simpler than plumbing it via an input option and setting it as an env var again and easier to understand?

.github/actions/run-test/action.yml

yury-s · 2024-04-25T20:26:04Z

.github/actions/run-test/action.yml

+  bot-name:
+    description: 'Bot name'
+    required: true
+  flakiness-client-id:


can it be declared as a secret: and propagated automatically from the calling workflow without explicitly mentioning these 3 all the time?

Unfortunately not. Composite Actions don't have access to secrets/vars as per here.

github-actions · 2024-05-01T18:15:31Z

Test results for "tests 1"

2 flaky

⚠️ [chromium-library] › library/trace-viewer.spec.ts:240:1 › should have network requests
⚠️ [firefox-page] › page/frame-goto.spec.ts:46:3 › should continue after client redirect

27277 passed, 671 skipped
✔️✔️✔️

Merge workflow run.

github-actions · 2024-05-01T19:28:34Z

Test results for "tests 2"

4 fatal errors, not part of any test
7 failed
❌ [firefox-library] › library/inspector/cli-codegen-2.spec.ts:340:7 › cli codegen › should record open in a new tab with url
❌ [firefox-library] › library/video.spec.ts:189:5 › screencast › should capture static page
❌ [firefox-library] › library/inspector/cli-codegen-2.spec.ts:340:7 › cli codegen › should record open in a new tab with url
❌ [webkit-library] › library/capabilities.spec.ts:67:3 › should play video @smoke
❌ [webkit-library] › library/capabilities.spec.ts:99:3 › should play audio @smoke
❌ [webkit-library] › library/signals.spec.ts:62:7 › signals › should report browser close signal 2
❌ [webkit-library] › library/tracing.spec.ts:412:14 › should produce screencast frames crop

54 flaky

⚠️ [chromium-library] › library/emulation-focus.spec.ts:202:12 › should trigger hover state concurrently
⚠️ [chromium-page] › page/locator-misc-1.spec.ts:28:3 › should hover when Node is removed
⚠️ [chromium-page] › page/page-add-locator-handler.spec.ts:90:5 › should work with locator.hover()
⚠️ [chromium-page] › page/page-drag.spec.ts:289:5 › Drag and drop › should work with the helper method
⚠️ [chromium-library] › library/inspector/cli-codegen-1.spec.ts:76:7 › cli codegen › should click after same-document navigation
⚠️ [chromium-page] › page/locator-misc-1.spec.ts:28:3 › should hover when Node is removed
⚠️ [chromium-page] › page/page-add-locator-handler.spec.ts:90:5 › should work with locator.hover()
⚠️ [chromium-page] › page/page-drag.spec.ts:53:5 › Drag and drop › should not send dragover on the first mousemove
⚠️ [chromium-page] › page/page-event-request.spec.ts:110:3 › should report navigation requests and responses handled by service worker
⚠️ [firefox-page] › page/page-goto.spec.ts:81:3 › should work with Cross-Origin-Opener-Policy
⚠️ [firefox-library] › library/tracing.spec.ts:412:14 › should produce screencast frames scale
⚠️ [firefox-library] › library/video.spec.ts:189:5 › screencast › should capture static page
⚠️ [firefox-page] › page/page-mouse.spec.ts:166:3 › should trigger hover state
⚠️ [firefox-page] › page/page-add-locator-handler.spec.ts:210:5 › should work when owner frame detaches
⚠️ [firefox-library] › library/browsercontext-add-cookies.spec.ts:141:3 › should isolate persistent cookies
⚠️ [firefox-library] › library/browsercontext-pages.spec.ts:128:3 › should not leak listeners during navigation of 20 pages
⚠️ [firefox-library] › library/browsertype-connect.spec.ts:932:3 › launchServer only › should be able to reconnect to a browser 12 times without warnings
⚠️ [firefox-library] › library/capabilities.spec.ts:22:3 › SharedArrayBuffer should work @smoke
⚠️ [firefox-library] › library/inspector/cli-codegen-1.spec.ts:56:7 › cli codegen › should ignore programmatic events
⚠️ [firefox-library] › library/inspector/cli-codegen-1.spec.ts:76:7 › cli codegen › should click after same-document navigation
⚠️ [firefox-library] › library/inspector/cli-codegen-2.spec.ts:138:7 › cli codegen › should upload multiple files
⚠️ [firefox-library] › library/inspector/cli-codegen-2.spec.ts:493:7 › cli codegen › should --save-trace
⚠️ [firefox-library] › library/selector-generator.spec.ts:161:5 › selector generator › should handle second non-unique data-testid
⚠️ [firefox-library] › library/tracing.spec.ts:292:5 › should overwrite existing file
⚠️ [firefox-page] › page/page-click.spec.ts:99:3 › should click the 1x1 div
⚠️ [firefox-page] › page/page-goto.spec.ts:81:3 › should work with Cross-Origin-Opener-Policy
⚠️ [firefox-page] › page/page-click.spec.ts:850:3 › should not hang when frame is detached
⚠️ [firefox-page] › page/page-event-request.spec.ts:171:3 › should return response body when Cross-Origin-Opener-Policy is set
⚠️ [chromium-page] › page/locator-misc-1.spec.ts:21:3 › should hover @smoke
⚠️ [chromium-page] › page/page-add-locator-handler.spec.ts:90:5 › should work with locator.hover()
⚠️ [chromium-page] › page/page-drag.spec.ts:25:5 › Drag and drop › should work @smoke
⚠️ [chromium-page] › page/page-mouse.spec.ts:166:3 › should trigger hover state
⚠️ [chromium-page] › page/page-leaks.spec.ts:123:5 › expect should not leak
⚠️ [chromium-library] › library/inspector/cli-codegen-1.spec.ts:23:7 › cli codegen › should click
⚠️ [chromium-page] › page/page-event-request.spec.ts:110:3 › should report navigation requests and responses handled by service worker
⚠️ [webkit-library] › library/inspector/cli-codegen-3.spec.ts:22:7 › cli codegen › should click locator.first
⚠️ [webkit-page] › page/page-drag.spec.ts💯5 › Drag and drop › should cancel on escape
⚠️ [webkit-page] › page/page-drag.spec.ts:246:5 › Drag and drop › should work if the drag event is captured but not canceled
⚠️ [webkit-library] › library/inspector/cli-codegen-1.spec.ts:197:7 › cli codegen › should not target selector preview by text regexp
⚠️ [webkit-page] › page/page-add-locator-handler.spec.ts:90:5 › should work with locator.hover()
⚠️ [webkit-page] › page/page-drag.spec.ts:70:5 › Drag and drop › should work inside iframe
⚠️ [webkit-page] › page/page-drag.spec.ts💯5 › Drag and drop › should cancel on escape
⚠️ [webkit-page] › page/page-drag.spec.ts:330:5 › Drag and drop › should work with locators
⚠️ [webkit-library] › library/browsercontext-viewport.spec.ts:143:12 › should drag with high dpi
⚠️ [webkit-library] › library/inspector/cli-codegen-1.spec.ts:23:7 › cli codegen › should click
⚠️ [webkit-library] › library/inspector/cli-codegen-1.spec.ts:159:7 › cli codegen › should work with TrustedTypes
⚠️ [webkit-library] › library/inspector/cli-codegen-1.spec.ts:324:7 › cli codegen › should fill [contentEditable]
⚠️ [webkit-library] › library/inspector/cli-codegen-2.spec.ts:340:7 › cli codegen › should record open in a new tab with url
⚠️ [webkit-library] › library/inspector/cli-codegen-3.spec.ts:415:7 › cli codegen › should generate getByPlaceholder
⚠️ [webkit-library] › library/selector-generator.spec.ts:108:5 › selector generator › should not use text for select
⚠️ [webkit-library] › library/browsertype-connect.spec.ts:528:5 › run-server › should error when saving download after deletion
⚠️ [webkit-library] › library/trace-viewer.spec.ts:893:1 › should display waitForLoadState even if did not wait for it
⚠️ [webkit-library] › library/tracing.spec.ts:412:14 › should produce screencast frames fit
⚠️ [webkit-library] › library/tracing.spec.ts:412:14 › should produce screencast frames scale

169038 passed, 7580 skipped, 2617 did not run
✔️✔️✔️

Merge workflow run.

dgozman reviewed Apr 24, 2024

View reviewed changes