feat: Re-introduce Run All specs for End to End under experimentalRunAllSpecs flag #24745

lmiller1990 · 2022-11-21T00:21:29Z

Closes Run All Specs for E2E #24168

User facing changelog

Adds a new experimental flag, experimentalRunAllSpecs, for End to End Testing. Enable it in cypress.config:

import { defineConfig } from 'cypress'

export default defineConfig({
  e2e: {
    experimentalRunAllSpecs: true
  }
})

Additional details

This PR adds back in support for running multiple tests sequentially in Open Mde for End to End testing. It can be accessed from both the Specs page and the Runner page in the Cypress App:

Steps to test

Either check out this branch locally, or install the pre-release binary. Those are here.
Enable the experiment:

import { defineConfig } from 'cypress'

export default defineConfig({
  e2e: {
    experimentalRunAllSpecs: true
  }
})

Open Cypress in Open Mode. The Specs page and Runner page should be have "Run N Specs" labels that are revealed when you hover a directory. Clicking the button will re-open the browser and execute all the specs sequentially.

You can also use the text search to only execute a sub-set of specs. Saving any of the specs will re-execute all the specs that matched your criteria (search, directory, or both).

Note: It's not fully a11y yet. You cannot do Run All Specs with a keyboard only yet. I filed an issue for a fast follow: #24748. If there's a trivial fix, that'd be great.

How has the user experience changed?

See above.

PR Tasks

Have tests been added/updated?
Has the original issue (or this PR, if no issue exists) been tagged with a release in ZenHub? (user-facing changes only)
[x Has a PR for user-facing changes been opened in cypress-documentation? https://github.com/chore: add experimentalRunAllSpecs experiment docs cypress-documentation#4860
Have API changes been updated in the type definitions?

cypress-bot · 2022-11-21T00:21:32Z

Thanks for taking the time to open a PR!

Create a Draft Pull Request if your PR is not ready for review. Mark the PR as Ready for Review when you're ready for a Cypress team member to review the PR.
Become familiar with the Code Review Checklist for guidelines on coding standards and what needs to be done before a PR can be merged.

lmiller1990 · 2022-11-21T00:24:39Z

Doing some testing, collecting my feedback here.

Edit: I've fixed these in #24746. Please review.

Feedback

## Button position on inline spec list is inconsistent

The icons appear in different positions. Should it align right?

simplescreenrecorder-2022-11-21_10.29.50.mp4

Another example:

css.mp4

I think it should always be a fixed width from the right. That's what the designs hint at.

Experimental flag types

Current I can set experimentalRunAllSpecs at the top level. This throws an error, telling me it's E2E only. How about we make the types reflect this, too?

export default = {
  experimentalRunAllSpecs: truer
}

IDE says it's valid, but Cypress throws an error.

* wip run-all-specs-ui * added component tests for specs-list, inline-specs-list, and run-all-specs * updated tests for specslist, inlinespeclist, and runAllSpecs * simplify test * make prop with default value optional * enable run all specs * use named slot for clarify * use Record type * remove un-necessary dynamic component * use group and hover to inline css * fix tests * fix tests and use snapshot for style testing Co-authored-by: Lachlan Miller <lachlan.miller.1990@outlook.com>

cypress · 2022-11-21T02:12:28Z

Test summary

511 • 16 • 44 • 95 • 10

Run details


Project	cypress
Status	Failed
Commit	`e182240`
Started	Nov 22, 2022 6:24 AM
Ended	Nov 22, 2022 6:42 AM
Duration	17:08 💡
OS	Windows 10.0.17763
Browser	Chrome 107

View run in Cypress Dashboard ➡️

Failures

Run group: app-e2e (Windows, Chrome )
	cypress\e2e\runner\sessions.ui.cy.ts	1
1	... > setup has failing Cypress command
	cypress\e2e\runner\reporter.hooks.cy.ts	1
1	hooks > displays commands under correct hook
	cypress\e2e\run-all-specs.cy.ts	1
1	run-all-specs > can run all specs with filter and live-reloading
	cypress\e2e\runner\reporter-ct-vite.errors.cy.ts	1
1	Vite - errors ui > assertion failures
	cypress\e2e\settings.cy.ts	1
1	App: Settings > visits settings page
	cypress\e2e\subscriptions\specChange-subscription.cy.ts	1
1	specChange subscription > specs list > responds to specChange event for an added file
	cypress\e2e\runner\reporter.errors.cy.ts	1
1	errors ui > exception failures
	cypress\e2e\runner\reporter-ct-webpack.uncaught-errors.cy.ts	1
1	uncaught errors
This comment includes only the first 10 test failures. See all 13 failures in the Cypress Dashboard.
Run group: launchpad-e2e (Windows, Chrome )
	cypress\e2e\top-nav-launchpad.cy.ts	2
1	... > when there is a project id > if the project has no runs, shows "record your first run" prompt after choosing testing type
2	... > global mode > shows "continue" button after login if project selected

Flakiness

	cypress\e2e\migration.cy.ts	1
1	Full migration flow for each project > completes journey for migration-component-testing
	cypress\e2e\top-nav-launchpad.cy.ts	3
1	... > logs out user if cloud request returns unauthorized
2	... > global mode > shows "continue" button after login if project selected has project id
3	... > global mode > shows "connect project" button after login if project selected has no project id
	cypress\e2e\scaffold-component-testing.cy.ts	1
1	scaffolding component testing > vue3-vite-ts-unconfigured > scaffolds component testing for Vue 3 and Vite
This comment includes only the first 5 flaky tests. See all 10 flaky tests in the Cypress Dashboard.

This comment has been generated by cypress-bot as a result of this project's GitHub integration settings. You can manage this integration in this project's settings in the Cypress Dashboard

* update ui [skip ci] * change types to reflect run all specs is e2e only

Closes undefined

marktnoonan · 2022-11-21T21:16:10Z

packages/app/src/specs/InlineRunAllSpecs.vue

+        hocus-fill-color="indigo-100"
+        class="inline-flex align-text-bottom"
+        data-cy="play-button"
+        @click.stop="emits('runAllSpecs')"


I think this click handler should be on the button, not the icon. When I use the keyboard on the button it doesn't rerun specs for me, it just toggles the folder open and closed.

Hmm, moving the handler didn't help, I think because the parent is using @click.capture to manage the toggle. That code would need some extra logic to not toggle if the click event came from the button.

I moved the listener anyway, it should be on the button. There is a follow up for better a11y.

marktnoonan · 2022-11-21T21:17:14Z

packages/app/src/specs/InlineRunAllSpecs.vue

+        class="font-normal text-sm inline-flex"
+        data-cy="tooltip-content"
+      >
+        Run {{ specNumber }} specs


We need to handle pluralization here, we could use the i18n for this.

Oh it's already in the main Spec Explorer, we can do that here.

Suggested change

Run {{ specNumber }} specs

{{ t('specPage.runAllSpecs', specNumber) }}

marktnoonan · 2022-11-21T22:26:40Z

packages/app/src/specs/SpecsRunAllSpecs.cy.tsx

+      cy.findByTestId('play-button').should('be.visible')
+    })
+
+    it('Renders styles correctly', () => {


In the future let's use a Percy snapshot for this kind of thing and not assert all these CSS values. It's not important whether or not these exact values are used for these elements, what matters is that visual appearance is as expected.

lmiller1990 · 2022-11-22T07:03:51Z

Some known windows linting fails, fixed by #24758

All other pipelines ✔️, merging.

cypress-bot · 2022-11-22T19:32:22Z

Released in 11.2.0.

This comment thread has been locked. If you are still experiencing this issue after upgrading to
Cypress v11.2.0, please open a new issue.

ZachJW34 and others added 2 commits November 17, 2022 08:58

feat: re-enable Run All Specs (#24683)

093310f

fix: disable query in run-mode

6f52bea

lmiller1990 mentioned this pull request Nov 21, 2022

feat: Run-all-specs-ui #24738

Merged

1 task

amehta265 and others added 3 commits November 21, 2022 11:42

Merge remote-tracking branch 'origin/develop' into feature/run-all-specs

b544e88

fix typescript

536f5e0

lmiller1990 added 2 commits November 21, 2022 12:49

fix test

3941d90

build binary for run all specs

5e080d7

lmiller1990 mentioned this pull request Nov 21, 2022

Run All Specs for E2E #24168

Closed

lmiller1990 added 3 commits November 21, 2022 14:08

fix: tweaks for Run All Specs (#24746)

3727fef

* update ui [skip ci] * change types to reflect run all specs is e2e only

chore: separate run all specs components

47142a4

fix typescript error

30163cb

lmiller1990 marked this pull request as ready for review November 21, 2022 07:22

lmiller1990 requested a review from a team November 21, 2022 07:27

ZachJW34 and others added 3 commits November 21, 2022 13:55

fix: run-all UI tweaks and keyboard nav (#24757)

3328bf4

Closes undefined

fix UI test

87b2af5

Merge remote-tracking branch 'origin/develop' into feature/run-all-specs

2fb4c09

ZachJW34 approved these changes Nov 21, 2022

View reviewed changes

marktnoonan reviewed Nov 21, 2022

View reviewed changes

astone123 approved these changes Nov 21, 2022

View reviewed changes

lmiller1990 and others added 5 commits November 22, 2022 09:05

move listener and use i18n [skip ci] (#24761)

033ab79

fix: run-all-specs duplicate file-matching (#24763)

f66a953

chore: limit linting [skip ci]

d445524

do not get separator until app is loaded

0565428

fix test

e182240

lmiller1990 merged commit 4bbd78e into develop Nov 22, 2022

lmiller1990 deleted the feature/run-all-specs branch November 22, 2022 07:28

cypress-bot bot locked as resolved and limited conversation to collaborators Nov 22, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Re-introduce Run All specs for End to End under experimentalRunAllSpecs flag #24745

feat: Re-introduce Run All specs for End to End under experimentalRunAllSpecs flag #24745

lmiller1990 commented Nov 21, 2022 •

edited

cypress-bot bot commented Nov 21, 2022

lmiller1990 commented Nov 21, 2022 •

edited

Experimental flag types

cypress bot commented Nov 21, 2022 •

edited

marktnoonan Nov 21, 2022

marktnoonan Nov 21, 2022

lmiller1990 Nov 21, 2022

marktnoonan Nov 21, 2022

marktnoonan Nov 21, 2022

lmiller1990 Nov 21, 2022

marktnoonan Nov 21, 2022

lmiller1990 commented Nov 22, 2022

cypress-bot bot commented Nov 22, 2022

	Run {{ specNumber }} specs
	{{ t('specPage.runAllSpecs', specNumber) }}

feat: Re-introduce Run All specs for End to End under experimentalRunAllSpecs flag #24745

feat: Re-introduce Run All specs for End to End under experimentalRunAllSpecs flag #24745

Conversation

lmiller1990 commented Nov 21, 2022 • edited

User facing changelog

Additional details

Steps to test

How has the user experience changed?

PR Tasks

cypress-bot bot commented Nov 21, 2022

lmiller1990 commented Nov 21, 2022 • edited

Experimental flag types

cypress bot commented Nov 21, 2022 • edited

Test summary

Run details

Failures

Flakiness

marktnoonan Nov 21, 2022

Choose a reason for hiding this comment

marktnoonan Nov 21, 2022

Choose a reason for hiding this comment

lmiller1990 Nov 21, 2022

Choose a reason for hiding this comment

marktnoonan Nov 21, 2022

Choose a reason for hiding this comment

marktnoonan Nov 21, 2022

Choose a reason for hiding this comment

lmiller1990 Nov 21, 2022

Choose a reason for hiding this comment

marktnoonan Nov 21, 2022

Choose a reason for hiding this comment

lmiller1990 commented Nov 22, 2022

cypress-bot bot commented Nov 22, 2022

lmiller1990 commented Nov 21, 2022 •

edited

lmiller1990 commented Nov 21, 2022 •

edited

cypress bot commented Nov 21, 2022 •

edited