Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

E2E Tests: Try running in Firefox (in addition to Chrome) #28498

Closed
wants to merge 5 commits into from

Conversation

ockham
Copy link
Contributor

@ockham ockham commented Jan 26, 2021

Description

First stab at running e2e tests in Firefox, when run in CI.

Sometimes we see errors that only manifest in Firefox, such as the issue that was fixed by #28212. In order to catch those earlier, it makes sense to add Firefox coverage to our automated tests.

Inspired by Automattic/jetpack#17893 (which I haven't gotten to work yet 馃槵)

How has this been tested?

See if the e2e tests GH Actions pass 馃

To run locally (with tests visible):

npm run test-e2e -- --puppeteer-interactive --puppeteer-product=firefox

@ockham ockham added the [Type] Automated Testing Testing infrastructure changes impacting the execution of end-to-end (E2E) and/or unit tests. label Jan 26, 2021
@ockham ockham self-assigned this Jan 26, 2021
@github-actions
Copy link

github-actions bot commented Jan 26, 2021

Size Change: +80 B (0%)

Total Size: 1.37 MB

Filename Size Change
build/annotations/index.js 3.77 kB -1 B (0%)
build/autop/index.js 2.83 kB -1 B (0%)
build/block-directory/index.js 9.08 kB -1 B (0%)
build/block-editor/index.js 123 kB +75 B (0%)
build/components/index.js 275 kB +11 B (0%)
build/compose/index.js 11.2 kB -1 B (0%)
build/core-data/index.js 15.2 kB -1 B (0%)
build/data-controls/index.js 829 B -1 B (0%)
build/data/index.js 8.97 kB +2 B (0%)
build/dom/index.js 4.93 kB -1 B (0%)
build/edit-navigation/index.js 11.1 kB +2 B (0%)
build/element/index.js 4.61 kB -1 B (0%)
build/format-library/index.js 6.77 kB -2 B (0%)
build/html-entities/index.js 622 B -1 B (0%)
build/is-shallow-equal/index.js 698 B -1 B (0%)
build/media-utils/index.js 5.32 kB +1 B (0%)
build/nux/index.js 3.41 kB +1 B (0%)
build/plugins/index.js 2.54 kB +1 B (0%)
build/priority-queue/index.js 791 B +2 B (0%)
build/redux-routine/index.js 2.84 kB -3 B (0%)
build/shortcode/index.js 1.7 kB +1 B (0%)
build/token-list/index.js 1.27 kB -1 B (0%)
鈩癸笍 View Unchanged
Filename Size Change
build/a11y/index.js 1.14 kB 0 B
build/api-fetch/index.js 3.4 kB 0 B
build/blob/index.js 665 B 0 B
build/block-directory/style-rtl.css 1.01 kB 0 B
build/block-directory/style.css 1.01 kB 0 B
build/block-editor/style-rtl.css 11.9 kB 0 B
build/block-editor/style.css 11.9 kB 0 B
build/block-library/blocks/archives/editor-rtl.css 61 B 0 B
build/block-library/blocks/archives/editor.css 60 B 0 B
build/block-library/blocks/audio/editor-rtl.css 58 B 0 B
build/block-library/blocks/audio/editor.css 58 B 0 B
build/block-library/blocks/audio/style-rtl.css 103 B 0 B
build/block-library/blocks/audio/style.css 103 B 0 B
build/block-library/blocks/block/editor-rtl.css 161 B 0 B
build/block-library/blocks/block/editor.css 161 B 0 B
build/block-library/blocks/button/editor-rtl.css 475 B 0 B
build/block-library/blocks/button/editor.css 474 B 0 B
build/block-library/blocks/button/style-rtl.css 453 B 0 B
build/block-library/blocks/button/style.css 451 B 0 B
build/block-library/blocks/buttons/editor-rtl.css 227 B 0 B
build/block-library/blocks/buttons/editor.css 227 B 0 B
build/block-library/blocks/buttons/style-rtl.css 297 B 0 B
build/block-library/blocks/buttons/style.css 297 B 0 B
build/block-library/blocks/calendar/style-rtl.css 208 B 0 B
build/block-library/blocks/calendar/style.css 208 B 0 B
build/block-library/blocks/categories/editor-rtl.css 84 B 0 B
build/block-library/blocks/categories/editor.css 83 B 0 B
build/block-library/blocks/categories/style-rtl.css 79 B 0 B
build/block-library/blocks/categories/style.css 79 B 0 B
build/block-library/blocks/code/style-rtl.css 90 B 0 B
build/block-library/blocks/code/style.css 90 B 0 B
build/block-library/blocks/columns/editor-rtl.css 190 B 0 B
build/block-library/blocks/columns/editor.css 190 B 0 B
build/block-library/blocks/columns/style-rtl.css 421 B 0 B
build/block-library/blocks/columns/style.css 421 B 0 B
build/block-library/blocks/cover/editor-rtl.css 392 B 0 B
build/block-library/blocks/cover/editor.css 393 B 0 B
build/block-library/blocks/cover/style-rtl.css 1.25 kB 0 B
build/block-library/blocks/cover/style.css 1.25 kB 0 B
build/block-library/blocks/embed/editor-rtl.css 486 B 0 B
build/block-library/blocks/embed/editor.css 486 B 0 B
build/block-library/blocks/embed/style-rtl.css 375 B 0 B
build/block-library/blocks/embed/style.css 375 B 0 B
build/block-library/blocks/file/editor-rtl.css 199 B 0 B
build/block-library/blocks/file/editor.css 198 B 0 B
build/block-library/blocks/file/style-rtl.css 248 B 0 B
build/block-library/blocks/file/style.css 248 B 0 B
build/block-library/blocks/freeform/editor-rtl.css 2.45 kB 0 B
build/block-library/blocks/freeform/editor.css 2.45 kB 0 B
build/block-library/blocks/gallery/editor-rtl.css 679 B 0 B
build/block-library/blocks/gallery/editor.css 679 B 0 B
build/block-library/blocks/gallery/style-rtl.css 1.07 kB 0 B
build/block-library/blocks/gallery/style.css 1.06 kB 0 B
build/block-library/blocks/group/editor-rtl.css 318 B 0 B
build/block-library/blocks/group/editor.css 317 B 0 B
build/block-library/blocks/group/style-rtl.css 57 B 0 B
build/block-library/blocks/group/style.css 57 B 0 B
build/block-library/blocks/heading/editor-rtl.css 129 B 0 B
build/block-library/blocks/heading/editor.css 129 B 0 B
build/block-library/blocks/heading/style-rtl.css 76 B 0 B
build/block-library/blocks/heading/style.css 76 B 0 B
build/block-library/blocks/html/editor-rtl.css 281 B 0 B
build/block-library/blocks/html/editor.css 281 B 0 B
build/block-library/blocks/image/editor-rtl.css 717 B 0 B
build/block-library/blocks/image/editor.css 716 B 0 B
build/block-library/blocks/image/style-rtl.css 477 B 0 B
build/block-library/blocks/image/style.css 478 B 0 B
build/block-library/blocks/latest-comments/editor-rtl.css 159 B 0 B
build/block-library/blocks/latest-comments/editor.css 158 B 0 B
build/block-library/blocks/latest-comments/style-rtl.css 269 B 0 B
build/block-library/blocks/latest-comments/style.css 269 B 0 B
build/block-library/blocks/latest-posts/editor-rtl.css 137 B 0 B
build/block-library/blocks/latest-posts/editor.css 137 B 0 B
build/block-library/blocks/latest-posts/style-rtl.css 523 B 0 B
build/block-library/blocks/latest-posts/style.css 522 B 0 B
build/block-library/blocks/list/editor-rtl.css 65 B 0 B
build/block-library/blocks/list/editor.css 65 B 0 B
build/block-library/blocks/list/style-rtl.css 63 B 0 B
build/block-library/blocks/list/style.css 63 B 0 B
build/block-library/blocks/media-text/editor-rtl.css 191 B 0 B
build/block-library/blocks/media-text/editor.css 191 B 0 B
build/block-library/blocks/media-text/style-rtl.css 535 B 0 B
build/block-library/blocks/media-text/style.css 532 B 0 B
build/block-library/blocks/more/editor-rtl.css 434 B 0 B
build/block-library/blocks/more/editor.css 434 B 0 B
build/block-library/blocks/navigation-link/editor-rtl.css 392 B 0 B
build/block-library/blocks/navigation-link/editor.css 394 B 0 B
build/block-library/blocks/navigation-link/style-rtl.css 704 B 0 B
build/block-library/blocks/navigation-link/style.css 702 B 0 B
build/block-library/blocks/navigation/editor-rtl.css 1.38 kB 0 B
build/block-library/blocks/navigation/editor.css 1.37 kB 0 B
build/block-library/blocks/navigation/style-rtl.css 171 B 0 B
build/block-library/blocks/navigation/style.css 171 B 0 B
build/block-library/blocks/nextpage/editor-rtl.css 395 B 0 B
build/block-library/blocks/nextpage/editor.css 395 B 0 B
build/block-library/blocks/paragraph/editor-rtl.css 109 B 0 B
build/block-library/blocks/paragraph/editor.css 109 B 0 B
build/block-library/blocks/paragraph/style-rtl.css 273 B 0 B
build/block-library/blocks/paragraph/style.css 273 B 0 B
build/block-library/blocks/post-author/editor-rtl.css 209 B 0 B
build/block-library/blocks/post-author/editor.css 209 B 0 B
build/block-library/blocks/post-author/style-rtl.css 183 B 0 B
build/block-library/blocks/post-author/style.css 184 B 0 B
build/block-library/blocks/post-comments-form/style-rtl.css 249 B 0 B
build/block-library/blocks/post-comments-form/style.css 249 B 0 B
build/block-library/blocks/post-content/editor-rtl.css 139 B 0 B
build/block-library/blocks/post-content/editor.css 139 B 0 B
build/block-library/blocks/post-excerpt/editor-rtl.css 73 B 0 B
build/block-library/blocks/post-excerpt/editor.css 73 B 0 B
build/block-library/blocks/post-featured-image/editor-rtl.css 338 B 0 B
build/block-library/blocks/post-featured-image/editor.css 338 B 0 B
build/block-library/blocks/post-featured-image/style-rtl.css 100 B 0 B
build/block-library/blocks/post-featured-image/style.css 100 B 0 B
build/block-library/blocks/preformatted/style-rtl.css 63 B 0 B
build/block-library/blocks/preformatted/style.css 63 B 0 B
build/block-library/blocks/pullquote/editor-rtl.css 183 B 0 B
build/block-library/blocks/pullquote/editor.css 183 B 0 B
build/block-library/blocks/pullquote/style-rtl.css 316 B 0 B
build/block-library/blocks/pullquote/style.css 316 B 0 B
build/block-library/blocks/query-loop/editor-rtl.css 90 B 0 B
build/block-library/blocks/query-loop/editor.css 89 B 0 B
build/block-library/blocks/query-loop/style-rtl.css 315 B 0 B
build/block-library/blocks/query-loop/style.css 317 B 0 B
build/block-library/blocks/query-pagination-numbers/editor-rtl.css 122 B 0 B
build/block-library/blocks/query-pagination-numbers/editor.css 121 B 0 B
build/block-library/blocks/query-pagination/editor-rtl.css 270 B 0 B
build/block-library/blocks/query-pagination/editor.css 262 B 0 B
build/block-library/blocks/query-pagination/style-rtl.css 168 B 0 B
build/block-library/blocks/query-pagination/style.css 168 B 0 B
build/block-library/blocks/query/editor-rtl.css 159 B 0 B
build/block-library/blocks/query/editor.css 160 B 0 B
build/block-library/blocks/quote/editor-rtl.css 61 B 0 B
build/block-library/blocks/quote/editor.css 61 B 0 B
build/block-library/blocks/quote/style-rtl.css 169 B 0 B
build/block-library/blocks/quote/style.css 169 B 0 B
build/block-library/blocks/rss/editor-rtl.css 201 B 0 B
build/block-library/blocks/rss/editor.css 202 B 0 B
build/block-library/blocks/rss/style-rtl.css 290 B 0 B
build/block-library/blocks/rss/style.css 290 B 0 B
build/block-library/blocks/search/editor-rtl.css 165 B 0 B
build/block-library/blocks/search/editor.css 165 B 0 B
build/block-library/blocks/search/style-rtl.css 342 B 0 B
build/block-library/blocks/search/style.css 344 B 0 B
build/block-library/blocks/separator/editor-rtl.css 99 B 0 B
build/block-library/blocks/separator/editor.css 99 B 0 B
build/block-library/blocks/separator/style-rtl.css 236 B 0 B
build/block-library/blocks/separator/style.css 236 B 0 B
build/block-library/blocks/shortcode/editor-rtl.css 504 B 0 B
build/block-library/blocks/shortcode/editor.css 504 B 0 B
build/block-library/blocks/site-logo/editor-rtl.css 201 B 0 B
build/block-library/blocks/site-logo/editor.css 201 B 0 B
build/block-library/blocks/site-logo/style-rtl.css 117 B 0 B
build/block-library/blocks/site-logo/style.css 117 B 0 B
build/block-library/blocks/social-link/editor-rtl.css 164 B 0 B
build/block-library/blocks/social-link/editor.css 165 B 0 B
build/block-library/blocks/social-links/editor-rtl.css 711 B 0 B
build/block-library/blocks/social-links/editor.css 712 B 0 B
build/block-library/blocks/social-links/style-rtl.css 1.37 kB 0 B
build/block-library/blocks/social-links/style.css 1.37 kB 0 B
build/block-library/blocks/spacer/editor-rtl.css 302 B 0 B
build/block-library/blocks/spacer/editor.css 302 B 0 B
build/block-library/blocks/spacer/style-rtl.css 48 B 0 B
build/block-library/blocks/spacer/style.css 48 B 0 B
build/block-library/blocks/subhead/editor-rtl.css 99 B 0 B
build/block-library/blocks/subhead/editor.css 99 B 0 B
build/block-library/blocks/subhead/style-rtl.css 80 B 0 B
build/block-library/blocks/subhead/style.css 80 B 0 B
build/block-library/blocks/table/editor-rtl.css 489 B 0 B
build/block-library/blocks/table/editor.css 489 B 0 B
build/block-library/blocks/table/style-rtl.css 386 B 0 B
build/block-library/blocks/table/style.css 386 B 0 B
build/block-library/blocks/tag-cloud/editor-rtl.css 118 B 0 B
build/block-library/blocks/tag-cloud/editor.css 118 B 0 B
build/block-library/blocks/tag-cloud/style-rtl.css 94 B 0 B
build/block-library/blocks/tag-cloud/style.css 94 B 0 B
build/block-library/blocks/template-part/editor-rtl.css 680 B 0 B
build/block-library/blocks/template-part/editor.css 679 B 0 B
build/block-library/blocks/text-columns/editor-rtl.css 95 B 0 B
build/block-library/blocks/text-columns/editor.css 95 B 0 B
build/block-library/blocks/text-columns/style-rtl.css 166 B 0 B
build/block-library/blocks/text-columns/style.css 166 B 0 B
build/block-library/blocks/verse/editor-rtl.css 62 B 0 B
build/block-library/blocks/verse/editor.css 62 B 0 B
build/block-library/blocks/verse/style-rtl.css 87 B 0 B
build/block-library/blocks/verse/style.css 87 B 0 B
build/block-library/blocks/video/editor-rtl.css 504 B 0 B
build/block-library/blocks/video/editor.css 503 B 0 B
build/block-library/blocks/video/style-rtl.css 193 B 0 B
build/block-library/blocks/video/style.css 193 B 0 B
build/block-library/common-rtl.css 1.01 kB 0 B
build/block-library/common.css 1.01 kB 0 B
build/block-library/editor-rtl.css 9.06 kB 0 B
build/block-library/editor.css 9.05 kB 0 B
build/block-library/index.js 143 kB 0 B
build/block-library/style-rtl.css 8.61 kB 0 B
build/block-library/style.css 8.61 kB 0 B
build/block-library/theme-rtl.css 748 B 0 B
build/block-library/theme.css 748 B 0 B
build/block-serialization-default-parser/index.js 1.88 kB 0 B
build/block-serialization-spec-parser/index.js 3.06 kB 0 B
build/blocks/index.js 48.3 kB 0 B
build/components/style-rtl.css 15.5 kB 0 B
build/components/style.css 15.5 kB 0 B
build/date/index.js 31.8 kB 0 B
build/deprecated/index.js 769 B 0 B
build/dom-ready/index.js 571 B 0 B
build/edit-navigation/style-rtl.css 938 B 0 B
build/edit-navigation/style.css 944 B 0 B
build/edit-post/index.js 306 kB 0 B
build/edit-post/style-rtl.css 6.51 kB 0 B
build/edit-post/style.css 6.5 kB 0 B
build/edit-site/index.js 24 kB 0 B
build/edit-site/style-rtl.css 4.01 kB 0 B
build/edit-site/style.css 4.01 kB 0 B
build/edit-widgets/index.js 23.6 kB 0 B
build/edit-widgets/style-rtl.css 3.17 kB 0 B
build/edit-widgets/style.css 3.18 kB 0 B
build/editor/editor-styles-rtl.css 658 B 0 B
build/editor/editor-styles.css 660 B 0 B
build/editor/index.js 41.8 kB 0 B
build/editor/style-rtl.css 3.89 kB 0 B
build/editor/style.css 3.89 kB 0 B
build/escape-html/index.js 735 B 0 B
build/format-library/style-rtl.css 637 B 0 B
build/format-library/style.css 639 B 0 B
build/hooks/index.js 2.27 kB 0 B
build/i18n/index.js 3.56 kB 0 B
build/keyboard-shortcuts/index.js 2.53 kB 0 B
build/keycodes/index.js 1.93 kB 0 B
build/list-reusable-blocks/index.js 3.14 kB 0 B
build/list-reusable-blocks/style-rtl.css 629 B 0 B
build/list-reusable-blocks/style.css 628 B 0 B
build/notices/index.js 1.85 kB 0 B
build/nux/style-rtl.css 731 B 0 B
build/nux/style.css 727 B 0 B
build/primitives/index.js 1.42 kB 0 B
build/reusable-blocks/index.js 2.92 kB 0 B
build/rich-text/index.js 13.4 kB 0 B
build/server-side-render/index.js 2.76 kB 0 B
build/url/index.js 3.02 kB 0 B
build/viewport/index.js 1.86 kB 0 B
build/warning/index.js 1.14 kB 0 B
build/wordcount/index.js 1.22 kB 0 B

compressed-size-action

@ockham
Copy link
Contributor Author

ockham commented Jan 26, 2021

Aah, we're using puppeteer-core (which doesn't download any browsers) rather than puppeteer itself...

@gziolo
Copy link
Member

gziolo commented Jan 27, 2021

Aah, we're using puppeteer-core (which doesn't download any browsers) rather than puppeteer itself...

Yes, we don't want everyone to download 150-ish MB hoping they run e2e tests locally for every PR 馃槄

Base automatically changed from master to trunk March 1, 2021 15:45
@Mamaduka
Copy link
Member

I think this would be easier after the Playwright migration is finished. See #38851.

@kevin940726
Copy link
Member

This is now possible after #44631 in Playwright. Should we close this now?

@Mamaduka Mamaduka closed this Mar 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
[Type] Automated Testing Testing infrastructure changes impacting the execution of end-to-end (E2E) and/or unit tests.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants