feat: add `test.failing` method #12610

michalwarda · 2022-03-27T13:18:14Z

Summary

This fixes #4627. In certain scenarios ie. when writing code in BDD way developers would like to write down a more broad test that fails purposefully, and later as soon as it passes it should inform that it can now be marked as not failing anymore.
Feature itself is heavily inspired by RSpec pending tests.

Test plan

Added both unit and e2e tests for the feature. Everything should be covered :).

facebook-github-bot · 2022-03-27T13:18:18Z

Hi @michalwarda!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at cla@fb.com. Thanks!

SimenB

This all looks great to me, thanks! We're missing type tests, a changelog entry and signing the CLA, hopefully those are quick 🙂

However, as you mention in #4627 (comment), this doesn't use TestMode. I don't really understand why it's not possible to do so now? I don't think we need only and skip modifiers on a .failing test - people should just change the modifier in that case.

mrazauskas · 2022-03-29T08:27:41Z

@michalwarda The type tests live here. You can run them with yarn test-types (just remember to build before). Please, do ping me if something is puzzling.

michalwarda · 2022-04-06T06:35:37Z

Sorry had a busy week! I'll try to finish the PR during the weekend or at the start of next week!

zpao · 2022-04-18T23:50:13Z

Poking to trigger CLA status check

facebook-github-bot · 2022-04-19T00:56:45Z

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks!

facebook-github-bot · 2022-04-19T00:56:48Z

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks!

SimenB · 2022-04-21T11:36:14Z

This isn't breaking, so removing it from the milestone

michalwarda · 2022-04-26T17:28:15Z

Ok! I've finished adding type specs!
Similarly I've added the docs to changelog. Wish I was faster to land in 28 but I'll do my best to at least land in the next one!

Answering the feedback below:

This all looks great to me, thanks! We're missing type tests, a changelog entry and signing the CLA, hopefully those are quick slightly_smiling_face

However, as you mention in #4627 (comment), this doesn't use TestMode. I don't really understand why it's not possible to do so now? I don't think we need only and skip modifiers on a .failing test - people should just change the modifier in that case.

I think it's important that a user can skip a failing test.
If ie. someone is trying to debug the failing test because it's passing for some reason then adding a only to it rather than replacing failing with only will keep the behavior as it was. Otherwise it might be a bit confusing.
Similarly because the code in a failing test is "running" then a user might want to add a skip to a failing test because he is debugging another one.

And 1 more thing I noticed is that a new concurrent modifier is being added to the Global.It. I think it is in conflict with my changes. I did not try to make it work with that because I'm not sure how it's supposed to work yet :D

packages/jest-types/__typetests__/globals.test.ts

docs/GlobalAPI.md

michalwarda · 2022-04-26T17:53:22Z

And 1 more thing. I'm NGL I have no idea how to implement it in jest-jasmine2. I think I would need an intro to do that

SimenB · 2022-04-26T17:54:37Z

Don't bother with jasmine, it's fine to just handle circus for now 👍

docs/GlobalAPI.md

michalwarda · 2022-04-29T17:02:23Z

Hey! Not sure about what should I do now, if we're waiting for a good moment for merge then cool! If you need me to do anything else about the pr hit me up with the request :) and I'll clean it up more :). I have my weekend empty and I can probably manage to fit in some time for changes :)

Also, if you're open to it I was thinking about a potential restructuring to the whole binding of flags like failing or concurrent to tests that might clean up the code because it's quickly getting out of hand how it's being used. I was thinking about using Proxy objects for that. I can prepare an example of that if you're interested in it at all.

SimenB · 2022-04-29T17:21:50Z

Sorry, missed your last push! Triggered CI now, will review tomorrow (CET) 🙂

SimenB · 2022-04-30T07:17:45Z

I can prepare an example of that if you're interested in it at all.

Definitely! A separate PR, but it sounds interesting 🙂

SimenB

looking really good!

docs/GlobalAPI.md

e2e/__tests__/__snapshots__/testFailing.test.ts.snap

e2e/__tests__/testFailing.test.ts

docs/GlobalAPI.md

SimenB · 2022-04-30T07:26:36Z

docs/GlobalAPI.md

@@ -731,6 +731,60 @@ test.each`
 });


oh, and test.concurrent.failing should work

Ok. I think I've managed to cover this! Though I'm not sure how to document it because it will be at least another few cases in README. test.concurrent.failing, test.concurrent.skip.failing, test.concurrent.only.failing.

Should I add all of those? If so do we want it to say like Works like test.failing but concurrently? Or a whole copy of description?

I agree we should refactor these docs to not mention every variant, but rather which modifiers are compatible with each other. Not sure how to best do that though, maybe we can leave it for a follow-up?

SimenB · 2022-05-02T06:35:12Z

@michalwarda ping 🙂 I really wanna get this over the line and release it 😀

e2e/test-failing/__tests__/worksWithConcurrentMode.test.js

e2e/test-failing/__tests__/worksWithConcurrentOnlyMode.test.js

michalwarda · 2022-05-05T11:13:40Z

docs/GlobalAPI.md

@@ -731,6 +731,60 @@ test.each`
 });


Ok. I think I've managed to cover this! Though I'm not sure how to document it because it will be at least another few cases in README. test.concurrent.failing, test.concurrent.skip.failing, test.concurrent.only.failing.

docs/GlobalAPI.md

packages/jest-types/__typetests__/globals.test.ts

michalwarda · 2022-05-05T14:35:33Z

@SimenB

@michalwarda ping slightly_smiling_face I really wanna get this over the line and release it grinning

I also wish to do that :D. If I'm not mistaken the last thing left to do is potentially add description about failing for concurrent and decide if we want to go with failing each.

Though for the failing.each part I would wait both from technical perspective, (as I mentioned, I have an idea on how to make it much cleaner to define those "modes" so that it will be more readable), and from documentation perspective I think we would need to decide on how to document those nested modes in a way that failing will not have to be mentioned for each variant.

packages/jest-jasmine2/src/jasmineAsyncInstall.ts

SimenB

thank you! I think the issue this fixed is almost the first one I submitted 😅 Very happy to see it land!

I think we can keep docs tweak for later

packages/jest-jasmine2/src/jasmineAsyncInstall.ts

Co-authored-by: Tom Mrazauskas <tom@mrazauskas.de>

github-actions · 2022-06-06T00:07:41Z

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Please note this issue tracker is not a help forum. We recommend using StackOverflow or our discord channel for questions.

michalwarda added 3 commits March 27, 2022 13:59

Add purposefully failing test case.

b8a31ce

Handle failing with only and skipped

b48dc98

Add documentation about failing tests

e5516b3

michalwarda mentioned this pull request Mar 27, 2022

Feature: Purposefully failing test (test.failing) #4627

Closed

SimenB requested changes Mar 29, 2022

View reviewed changes

SimenB added this to the Jest 28 milestone Apr 6, 2022

facebook-github-bot added the cla signed label Apr 19, 2022

SimenB modified the milestones: Jest 28, High priority future Apr 21, 2022

michalwarda and others added 3 commits April 26, 2022 18:28

Add failing to types

7547ebe

Add info about failing to changelog

159c4f8

Merge branch 'main' into main

839964c

michalwarda force-pushed the main branch from 0ac1eee to 839964c Compare April 26, 2022 16:53

michalwarda requested a review from SimenB April 26, 2022 17:19

mrazauskas reviewed Apr 26, 2022

View reviewed changes

packages/jest-types/__typetests__/globals.test.ts Show resolved Hide resolved

michalwarda force-pushed the main branch from ee1db56 to 774358d Compare April 26, 2022 17:31

mrazauskas reviewed Apr 26, 2022

View reviewed changes

docs/GlobalAPI.md Outdated Show resolved Hide resolved

docs/GlobalAPI.md Outdated Show resolved Hide resolved

mrazauskas reviewed Apr 26, 2022

View reviewed changes

docs/GlobalAPI.md Outdated Show resolved Hide resolved

michalwarda force-pushed the main branch from 774358d to c29e59f Compare April 26, 2022 19:18

Add missing specs and docs for skipping with different aliases

913623a

SimenB added 3 commits April 30, 2022 09:15

Merge branch 'main' into michalwarda_main

c52e21d

snapshot

d9a5af8

move changelog entry

a0df0bf

correct changelog link

fde999d

SimenB requested changes Apr 30, 2022

View reviewed changes

michalwarda and others added 5 commits May 5, 2022 10:35

Add another point to README for failing test.

bf007fa

Add skipping failing functionality with jasmine

874d454

Add skipping failing functionality to concurrent

4ed39f6

Merge branch 'facebook:main' into main

73e8484

Add type tests for concurrent functions

60ffb4c

mrazauskas reviewed May 5, 2022

View reviewed changes

e2e/test-failing/__tests__/worksWithConcurrentMode.test.js Outdated Show resolved Hide resolved

e2e/test-failing/__tests__/worksWithConcurrentOnlyMode.test.js Outdated Show resolved Hide resolved

michalwarda commented May 5, 2022

View reviewed changes

Fix typos in tests

3e70058

SimenB reviewed May 6, 2022

View reviewed changes

packages/jest-jasmine2/src/jasmineAsyncInstall.ts Outdated Show resolved Hide resolved

michalwarda and others added 2 commits May 6, 2022 11:32

Throw when trying to use failing in jasmine

9faefc5

fix codeframe

a804799

SimenB approved these changes May 6, 2022

View reviewed changes

throw error for failing in jasmine

f8cc53d

mrazauskas reviewed May 6, 2022

View reviewed changes

packages/jest-jasmine2/src/jasmineAsyncInstall.ts Outdated Show resolved Hide resolved

SimenB and others added 2 commits May 6, 2022 12:08

Update packages/jest-jasmine2/src/jasmineAsyncInstall.ts

a02b159

Co-authored-by: Tom Mrazauskas <tom@mrazauskas.de>

sort

368eefd

SimenB changed the title ~~Add test.failing method.~~ feat: add test.failing method May 6, 2022

SimenB merged commit 34d15d1 into jestjs:main May 6, 2022

github-actions bot locked as resolved and limited conversation to collaborators Jun 6, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add `test.failing` method #12610

feat: add `test.failing` method #12610

michalwarda commented Mar 27, 2022 •

edited by SimenB

facebook-github-bot commented Mar 27, 2022

SimenB left a comment

mrazauskas commented Mar 29, 2022

michalwarda commented Apr 6, 2022

zpao commented Apr 18, 2022

facebook-github-bot commented Apr 19, 2022

facebook-github-bot commented Apr 19, 2022

SimenB commented Apr 21, 2022 •

edited

michalwarda commented Apr 26, 2022 •

edited

michalwarda commented Apr 26, 2022 •

edited

SimenB commented Apr 26, 2022

michalwarda commented Apr 29, 2022 •

edited

SimenB commented Apr 29, 2022

SimenB commented Apr 30, 2022

SimenB left a comment

SimenB Apr 30, 2022

michalwarda May 5, 2022

michalwarda May 5, 2022

SimenB May 6, 2022

SimenB commented May 2, 2022

michalwarda May 5, 2022

michalwarda commented May 5, 2022 •

edited

SimenB left a comment

github-actions bot commented Jun 6, 2022

feat: add test.failing method #12610

feat: add test.failing method #12610

Conversation

michalwarda commented Mar 27, 2022 • edited by SimenB

Summary

Test plan

facebook-github-bot commented Mar 27, 2022

Action Required

Process

SimenB left a comment

Choose a reason for hiding this comment

mrazauskas commented Mar 29, 2022

michalwarda commented Apr 6, 2022

zpao commented Apr 18, 2022

facebook-github-bot commented Apr 19, 2022

facebook-github-bot commented Apr 19, 2022

SimenB commented Apr 21, 2022 • edited

michalwarda commented Apr 26, 2022 • edited

michalwarda commented Apr 26, 2022 • edited

SimenB commented Apr 26, 2022

michalwarda commented Apr 29, 2022 • edited

SimenB commented Apr 29, 2022

SimenB commented Apr 30, 2022

SimenB left a comment

Choose a reason for hiding this comment

SimenB Apr 30, 2022

Choose a reason for hiding this comment

michalwarda May 5, 2022

Choose a reason for hiding this comment

michalwarda May 5, 2022

Choose a reason for hiding this comment

SimenB May 6, 2022

Choose a reason for hiding this comment

SimenB commented May 2, 2022

michalwarda May 5, 2022

Choose a reason for hiding this comment

michalwarda commented May 5, 2022 • edited

SimenB left a comment

Choose a reason for hiding this comment

github-actions bot commented Jun 6, 2022

feat: add `test.failing` method #12610

feat: add `test.failing` method #12610

michalwarda commented Mar 27, 2022 •

edited by SimenB

SimenB commented Apr 21, 2022 •

edited

michalwarda commented Apr 26, 2022 •

edited

michalwarda commented Apr 26, 2022 •

edited

michalwarda commented Apr 29, 2022 •

edited

michalwarda commented May 5, 2022 •

edited