Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: junit reporters crashing on failed asserts #24677

Merged
merged 7 commits into from Nov 14, 2022
Merged

fix: junit reporters crashing on failed asserts #24677

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
9d54bf8
fix: junit reporters crashing on failures
ryanthemanuel Nov 14, 2022
426a99f
handle other platforms
ryanthemanuel Nov 14, 2022
552b855
Fix system tests
ryanthemanuel Nov 14, 2022
5b9281a
fix snapshot
ryanthemanuel Nov 14, 2022
f92004f
fix snapshot
ryanthemanuel Nov 14, 2022
099522d
fix snapshot
ryanthemanuel Nov 14, 2022
a9fd2f3
Merge branch 'develop' into ryanm/fix/reporters
ryanthemanuel Nov 14, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
63 changes: 50 additions & 13 deletions system-tests/__snapshots__/reporters_spec.js
View file
Open in desktop
Expand Up @@ -9,6 +9,8 @@ We searched for the reporter in these paths:
Learn more at https://on.cypress.io/reporters

Error: Cannot find module '/foo/bar/.projects/e2e/node_modules/module-does-not-exist'
Require stack:
- lib/reporter.js
[stack trace lines]
`

Expand All @@ -21,14 +23,14 @@ exports['e2e reporters supports junit reporter and reporter options 1'] = `
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Cypress: 1.2.3 │
│ Browser: FooBrowser 88 │
│ Specs: 1 found (simple_passing.cy.js) │
│ Searched: cypress/e2e/simple_passing.cy.js │
│ Specs: 2 found (simple_passing.cy.js, simple_failing.cy.js) │
│ Searched: cypress/e2e/simple_passing.cy.js, cypress/e2e/simple_failing.cy.js │
└────────────────────────────────────────────────────────────────────────────────────────────────┘


────────────────────────────────────────────────────────────────────────────────────────────────────

Running: simple_passing.cy.js (1 of 1)
Running: simple_passing.cy.js (1 of 2)

(Results)

Expand All @@ -51,6 +53,39 @@ exports['e2e reporters supports junit reporter and reporter options 1'] = `
- Finished processing: /XXX/XXX/XXX/cypress/videos/simple_passing.cy.js.mp4 (X second)


────────────────────────────────────────────────────────────────────────────────────────────────────

Running: simple_failing.cy.js (2 of 2)

(Results)

┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Tests: 2 │
│ Passing: 0 │
│ Failing: 2 │
│ Pending: 0 │
│ Skipped: 0 │
│ Screenshots: 2 │
│ Video: true │
│ Duration: X seconds │
│ Spec Ran: simple_failing.cy.js │
└────────────────────────────────────────────────────────────────────────────────────────────────┘


(Screenshots)

- /XXX/XXX/XXX/cypress/screenshots/simple_failing.cy.js/simple failing spec -- fai (2560x1440)
ls1 (failed).png
- /XXX/XXX/XXX/cypress/screenshots/simple_failing.cy.js/simple failing spec -- fai (2560x1440)
ls2 (failed).png


(Video)

- Started processing: Compressing to 32 CRF
- Finished processing: /XXX/XXX/XXX/cypress/videos/simple_failing.cy.js.mp4 (X second)


====================================================================================================

(Run Finished)
Expand All @@ -59,8 +94,10 @@ exports['e2e reporters supports junit reporter and reporter options 1'] = `
Spec Tests Passing Failing Pending Skipped
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ ✔ simple_passing.cy.js XX:XX 1 1 - - - │
├────────────────────────────────────────────────────────────────────────────────────────────────┤
│ ✖ simple_failing.cy.js XX:XX 2 - 2 - - │
└────────────────────────────────────────────────────────────────────────────────────────────────┘
✔ All specs passed! XX:XX 1 1 - - -
✖ 1 of 2 failed (50%) XX:XX 3 1 2 - -


`
Expand Down Expand Up @@ -272,11 +309,11 @@ Because this error occurred during a \`after all\` hook we are skipping the rema

(Screenshots)

- /XXX/XXX/XXX/cypress/screenshots/simple_failing_hook.cy.js/simple failing hook s (1280x720)
- /XXX/XXX/XXX/cypress/screenshots/simple_failing_hook.cy.js/simple failing hook s (2560x1440)
pec -- beforeEach hooks -- never gets here -- before each hook (failed).png
- /XXX/XXX/XXX/cypress/screenshots/simple_failing_hook.cy.js/simple failing hook s (1280x720)
- /XXX/XXX/XXX/cypress/screenshots/simple_failing_hook.cy.js/simple failing hook s (2560x1440)
pec -- afterEach hooks -- runs this -- after each hook (failed).png
- /XXX/XXX/XXX/cypress/screenshots/simple_failing_hook.cy.js/simple failing hook s (1280x720)
- /XXX/XXX/XXX/cypress/screenshots/simple_failing_hook.cy.js/simple failing hook s (2560x1440)
pec -- after hooks -- fails on this -- after all hook (failed).png


Expand Down Expand Up @@ -448,11 +485,11 @@ Because this error occurred during a \`after all\` hook we are skipping the rema

(Screenshots)

- /XXX/XXX/XXX/cypress/screenshots/simple_failing_hook.cy.js/simple failing hook s (1280x720)
- /XXX/XXX/XXX/cypress/screenshots/simple_failing_hook.cy.js/simple failing hook s (2560x1440)
pec -- beforeEach hooks -- never gets here -- before each hook (failed).png
- /XXX/XXX/XXX/cypress/screenshots/simple_failing_hook.cy.js/simple failing hook s (1280x720)
- /XXX/XXX/XXX/cypress/screenshots/simple_failing_hook.cy.js/simple failing hook s (2560x1440)
pec -- afterEach hooks -- runs this -- after each hook (failed).png
- /XXX/XXX/XXX/cypress/screenshots/simple_failing_hook.cy.js/simple failing hook s (1280x720)
- /XXX/XXX/XXX/cypress/screenshots/simple_failing_hook.cy.js/simple failing hook s (2560x1440)
pec -- after hooks -- fails on this -- after all hook (failed).png


Expand Down Expand Up @@ -624,11 +661,11 @@ Because this error occurred during a \`after all\` hook we are skipping the rema

(Screenshots)

- /XXX/XXX/XXX/cypress/screenshots/simple_failing_hook.cy.js/simple failing hook s (1280x720)
- /XXX/XXX/XXX/cypress/screenshots/simple_failing_hook.cy.js/simple failing hook s (2560x1440)
pec -- beforeEach hooks -- never gets here -- before each hook (failed).png
- /XXX/XXX/XXX/cypress/screenshots/simple_failing_hook.cy.js/simple failing hook s (1280x720)
- /XXX/XXX/XXX/cypress/screenshots/simple_failing_hook.cy.js/simple failing hook s (2560x1440)
pec -- afterEach hooks -- runs this -- after each hook (failed).png
- /XXX/XXX/XXX/cypress/screenshots/simple_failing_hook.cy.js/simple failing hook s (1280x720)
- /XXX/XXX/XXX/cypress/screenshots/simple_failing_hook.cy.js/simple failing hook s (2560x1440)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why would this change increase the snapshot size? This seems quite large for a default snapshot size.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that's the size of the launched browser

pec -- after hooks -- fails on this -- after all hook (failed).png


Expand Down
18 changes: 13 additions & 5 deletions system-tests/test/reporters_spec.js
View file
Open in desktop
Expand Up @@ -36,22 +36,30 @@ describe('e2e reporters', () => {

it('supports junit reporter and reporter options', function () {
return systemTests.exec(this, {
spec: 'simple_passing.cy.js',
spec: 'simple_passing.cy.js,simple_failing.cy.js',
snapshot: true,
reporter: 'junit',
reporterOptions: 'mochaFile=junit-output/result.[hash].xml,testCaseSwitchClassnameAndName=true',
expectedExitCode: 2,
})
.then(() => {
return glob(path.join(e2ePath, 'junit-output', 'result.*.xml'))
.then((paths) => {
expect(paths.length).to.eq(1)
expect(paths.length).to.eq(2)

return Promise.all([fs.readFileAsync(paths[0], 'utf8'), fs.readFileAsync(paths[1], 'utf8')])
.then((results) => {
const str = results.join('')

return fs.readFileAsync(paths[0], 'utf8')
.then((str) => {
expect(str).to.include('<testsuite name="simple passing spec"')
expect(str).to.include('<testcase name="passes"')

expect(str).to.include('classname="simple passing spec passes"')

expect(str).to.include('<testsuite name="simple failing spec"')
expect(str).to.include('<testcase name="fails1"')
expect(str).to.include('<testcase name="fails2"')
expect(str).to.include('classname="simple failing spec fails1"')
expect(str).to.include('classname="simple failing spec fails2"')
})
})
})
Expand Down
9 changes: 5 additions & 4 deletions tooling/v8-snapshot/cache/prod-darwin/snapshot-meta.cache.json
View file
Open in desktop
Expand Up @@ -45,12 +45,16 @@
"./packages/https-proxy/lib/ca.js",
"./packages/net-stubbing/node_modules/debug/src/node.js",
"./packages/network/node_modules/minimatch/minimatch.js",
"./packages/server/lib/browsers/utils.ts",
"./packages/server/lib/capture.js",
"./packages/server/lib/cloud/exception.ts",
"./packages/server/lib/errors.ts",
"./packages/server/lib/modes/record.js",
"./packages/server/lib/modes/run.ts",
"./packages/server/lib/open_project.ts",
"./packages/server/lib/project-base.ts",
"./packages/server/lib/socket-ct.ts",
"./packages/server/lib/util/process_profiler.ts",
"./packages/server/lib/util/suppress_warnings.js",
"./packages/server/node_modules/@benmalka/foxdriver/node_modules/graceful-fs/polyfills.js",
"./packages/server/node_modules/glob/node_modules/minimatch/minimatch.js",
Expand Down Expand Up @@ -825,20 +829,17 @@
"./packages/server/lib/browsers/firefox-util.ts",
"./packages/server/lib/browsers/firefox.ts",
"./packages/server/lib/browsers/index.ts",
"./packages/server/lib/browsers/utils.ts",
"./packages/server/lib/browsers/webkit-automation.ts",
"./packages/server/lib/browsers/webkit.ts",
"./packages/server/lib/cache.js",
"./packages/server/lib/cloud/api.ts",
"./packages/server/lib/cloud/auth.ts",
"./packages/server/lib/cloud/exception.ts",
"./packages/server/lib/cloud/routes.ts",
"./packages/server/lib/cloud/user.ts",
"./packages/server/lib/config.ts",
"./packages/server/lib/controllers/iframes.ts",
"./packages/server/lib/cypress.js",
"./packages/server/lib/environment.js",
"./packages/server/lib/errors.ts",
"./packages/server/lib/gui/windows.ts",
"./packages/server/lib/log.js",
"./packages/server/lib/makeDataContext.ts",
Expand Down Expand Up @@ -869,7 +870,6 @@
"./packages/server/lib/util/file-opener.ts",
"./packages/server/lib/util/fs.ts",
"./packages/server/lib/util/glob.js",
"./packages/server/lib/util/process_profiler.ts",
"./packages/server/lib/util/proxy.ts",
"./packages/server/lib/util/resourceTypeAndCredentialManager.ts",
"./packages/server/lib/util/socket_allowed.ts",
Expand Down Expand Up @@ -3644,6 +3644,7 @@
"./packages/net-stubbing/node_modules/mime-types/index.js",
"./packages/network/lib/allow-destroy.ts",
"./packages/network/lib/blocked.ts",
"./packages/network/lib/ca.ts",
"./packages/network/lib/concat-stream.ts",
"./packages/network/lib/http-utils.ts",
"./packages/network/lib/index.ts",
Expand Down
8 changes: 4 additions & 4 deletions tooling/v8-snapshot/cache/prod-linux/snapshot-meta.cache.json
View file
Open in desktop
Expand Up @@ -45,10 +45,14 @@
"./packages/https-proxy/lib/ca.js",
"./packages/net-stubbing/node_modules/debug/src/node.js",
"./packages/network/node_modules/minimatch/minimatch.js",
"./packages/server/lib/browsers/utils.ts",
"./packages/server/lib/capture.js",
"./packages/server/lib/cloud/exception.ts",
"./packages/server/lib/errors.ts",
"./packages/server/lib/modes/record.js",
"./packages/server/lib/modes/run.ts",
"./packages/server/lib/open_project.ts",
"./packages/server/lib/util/process_profiler.ts",
"./packages/server/lib/project-base.ts",
"./packages/server/lib/socket-ct.ts",
"./packages/server/lib/util/suppress_warnings.js",
Expand Down Expand Up @@ -824,20 +828,17 @@
"./packages/server/lib/browsers/firefox-util.ts",
"./packages/server/lib/browsers/firefox.ts",
"./packages/server/lib/browsers/index.ts",
"./packages/server/lib/browsers/utils.ts",
"./packages/server/lib/browsers/webkit-automation.ts",
"./packages/server/lib/browsers/webkit.ts",
"./packages/server/lib/cache.js",
"./packages/server/lib/cloud/api.ts",
"./packages/server/lib/cloud/auth.ts",
"./packages/server/lib/cloud/exception.ts",
"./packages/server/lib/cloud/routes.ts",
"./packages/server/lib/cloud/user.ts",
"./packages/server/lib/config.ts",
"./packages/server/lib/controllers/iframes.ts",
"./packages/server/lib/cypress.js",
"./packages/server/lib/environment.js",
"./packages/server/lib/errors.ts",
"./packages/server/lib/gui/windows.ts",
"./packages/server/lib/log.js",
"./packages/server/lib/makeDataContext.ts",
Expand Down Expand Up @@ -868,7 +869,6 @@
"./packages/server/lib/util/file-opener.ts",
"./packages/server/lib/util/fs.ts",
"./packages/server/lib/util/glob.js",
"./packages/server/lib/util/process_profiler.ts",
"./packages/server/lib/util/proxy.ts",
"./packages/server/lib/util/resourceTypeAndCredentialManager.ts",
"./packages/server/lib/util/socket_allowed.ts",
Expand Down
8 changes: 4 additions & 4 deletions tooling/v8-snapshot/cache/prod-win32/snapshot-meta.cache.json
View file
Open in desktop
Expand Up @@ -45,10 +45,14 @@
"./packages/https-proxy/lib/ca.js",
"./packages/net-stubbing/node_modules/debug/src/node.js",
"./packages/network/node_modules/minimatch/minimatch.js",
"./packages/server/lib/browsers/utils.ts",
"./packages/server/lib/capture.js",
"./packages/server/lib/cloud/exception.ts",
"./packages/server/lib/errors.ts",
"./packages/server/lib/modes/record.js",
"./packages/server/lib/modes/run.ts",
"./packages/server/lib/open_project.ts",
"./packages/server/lib/util/process_profiler.ts",
"./packages/server/lib/project-base.ts",
"./packages/server/lib/socket-ct.ts",
"./packages/server/lib/util/suppress_warnings.js",
Expand Down Expand Up @@ -827,20 +831,17 @@
"./packages/server/lib/browsers/firefox-util.ts",
"./packages/server/lib/browsers/firefox.ts",
"./packages/server/lib/browsers/index.ts",
"./packages/server/lib/browsers/utils.ts",
"./packages/server/lib/browsers/webkit-automation.ts",
"./packages/server/lib/browsers/webkit.ts",
"./packages/server/lib/cache.js",
"./packages/server/lib/cloud/api.ts",
"./packages/server/lib/cloud/auth.ts",
"./packages/server/lib/cloud/exception.ts",
"./packages/server/lib/cloud/routes.ts",
"./packages/server/lib/cloud/user.ts",
"./packages/server/lib/config.ts",
"./packages/server/lib/controllers/iframes.ts",
"./packages/server/lib/cypress.js",
"./packages/server/lib/environment.js",
"./packages/server/lib/errors.ts",
"./packages/server/lib/gui/windows.ts",
"./packages/server/lib/log.js",
"./packages/server/lib/makeDataContext.ts",
Expand Down Expand Up @@ -871,7 +872,6 @@
"./packages/server/lib/util/file-opener.ts",
"./packages/server/lib/util/fs.ts",
"./packages/server/lib/util/glob.js",
"./packages/server/lib/util/process_profiler.ts",
"./packages/server/lib/util/proxy.ts",
"./packages/server/lib/util/resourceTypeAndCredentialManager.ts",
"./packages/server/lib/util/socket_allowed.ts",
Expand Down
4 changes: 4 additions & 0 deletions tooling/v8-snapshot/src/setup/force-no-rewrite.ts
View file
Open in desktop
Expand Up @@ -58,4 +58,8 @@ export default [
'packages/server/lib/open_project.ts',
'packages/server/lib/project-base.ts',
'packages/server/lib/socket-ct.ts',
'packages/server/lib/browsers/utils.ts',
'packages/server/lib/cloud/exception.ts',
'packages/server/lib/errors.ts',
'packages/server/lib/util/process_profiler.ts',
]