Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Should the spec reporter report retries? #5014

Open
4 tasks done
lilnasy opened this issue Oct 9, 2023 · 1 comment · May be fixed by #5099
Open
4 tasks done

Should the spec reporter report retries? #5014

lilnasy opened this issue Oct 9, 2023 · 1 comment · May be fixed by #5099
Labels
area: reporters involving a specific reporter semver-major implementation requires increase of "major" version number; "breaking changes" status: accepting prs Mocha can use your help with this one! type: feature enhancement proposal

Comments

@lilnasy
Copy link

lilnasy commented Oct 9, 2023

Prerequisites

  • Checked that your issue hasn't already been filed by cross-referencing issues with the faq label
  • Checked next-gen ES issues and syntax problems by using the same environment and/or transpiler configuration without Mocha to ensure it isn't just a feature that actually isn't supported in the environment in question or a bug in your code.
  • 'Smoke tested' the code to be tested by running it outside the real test suite to get a better sense of whether the problem is in the code under test, your usage of Mocha, or Mocha itself
  • Ensured that there is no discrepancy between the locally and globally installed versions of Mocha. You can find them with: node_modules/.bin/mocha --version(Local) and mocha --version(Global). We recommend that you not install Mocha globally.

Description

We have a few flaky tests and we would like to be aware of them. Currently, the spec reporter does not make a distinction between a test that succeeded first try and one that needed to be retried. We are probably going to create a custom reporter that extends Mocha.reporters.Spec to report the flaky tests at the end, but if it can be upstreamed, that would be ideal.

If the idea has good reception with mocha maintainers, I could try making a PR.
@lilnasy lilnasy mentioned this issue Oct 10, 2023
Closed
@JoshuaKGoldberg
Copy link
Member

I think this is a good idea. Vitest already does this with a yellow (retry x${task.result.retryCount}) (vitest-dev/vitest#1929). Unless anybody has an alternate suggestion, I'd say we can go with that for ecosystem consistency.

The spec reporter is meant for human consumption, so adding extra info like a retry count shouldn't hurt... but just to be safe, I think we should consider this a breaking change. You never know when someone's parsing spec reporter output instead of the JSON reporter for some bizarre reason. [Relevant xkcd "workflow" comic]

@JoshuaKGoldberg JoshuaKGoldberg added type: feature enhancement proposal area: reporters involving a specific reporter semver-major implementation requires increase of "major" version number; "breaking changes" and removed unconfirmed-bug labels Jan 21, 2024
@JoshuaKGoldberg JoshuaKGoldberg added the status: accepting prs Mocha can use your help with this one! label Feb 7, 2024
@CheadleCheadle CheadleCheadle linked a pull request Feb 22, 2024 that will close this issue
Open
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area: reporters involving a specific reporter semver-major implementation requires increase of "major" version number; "breaking changes" status: accepting prs Mocha can use your help with this one! type: feature enhancement proposal
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

spec-reporter prints retries CheadleCheadle/mocha
2 participants
@JoshuaKGoldberg @lilnasy