Ensure unhandled rejections cause test failure. #1241

rwjblue · 2017-12-18T06:31:56Z

Prior to this change, an unhandled exception in a promise that was not returned from the test callback would not cause test failure.

For example, the following (using default out of the box settings) results in test failures:

test('native promises cause an unhandled rejection', function(assert) {
  // imagine this was "deep" inside various method calls
  setTimeout(function() {
    throw new Error('whoops!');
  });

  // other things that we are trying to test
  assert.deepEqual(someMethod(), ['one', 'two']);
});

And this (prior to this PR) would result in a passing test suite:

test('native promises cause an unhandled rejection', function(assert) {
  // imagine this was "deep" inside various method calls
  Promise.resolve().then(function() {
    throw new Error('whoops!');
  });

  // other things that we are trying to test
  assert.deepEqual(someMethod(), ['one', 'two']);
});

The reason that what we are doing already isn't good enough is:

HTML Reporter: the default unhandledRejection handler does not dispatch to window.onerror at all... 😩
Node CLI: unhandled rejections in Node 6.6 and higher emit a console warning, but nothing else

Krinkle · 2017-12-18T16:09:34Z

test/reporter-html/unhandled-rejection.js

@@ -0,0 +1,35 @@
+// Currently (2017-12-18) only Chrome has implemented `unhandledRejection`
+const IS_CHROME = !!window.chrome;


'onrejectionhandled' in window might be easier to maintain as a feature test, or are there false positives for that? That way when other browsers land support, our CI will verify them automatically.

Awesome, thank you for that. Hard coding to Chrome definitely felt bad, but it was late and I ran out of ideas. 😝

Also just confirmed that Safari also supports the unhandled rejection events, thanks again for the pointer on the feature detect!

Brings unhandled rejections in line with unhandled exceptions.

rwjblue · 2017-12-18T19:26:21Z

I took some time to experiment with the implementation (by polyfilling in emberjs/ember-qunit#306), and it seems that just using QUnit.onError was quite the right implementation. QUnit.onError is closely tied to window.onerror semantics (returning true or false for bubbling, the specific argument signature for stack trace, etc). Instead of making it much more complicated I created a QUnit.onRejectionHandler method to specifically handle this case.

I also updated the tests to ensure that we are testing both inside and outside of a single test context, and modified to use a more appropriate feature detection (instead of only testing on Chrome). The new tests pass properly on both Chrome and Safari (which are the only two browsers which have implemented onunhandledrejection to my knowledge).

* Properly feature detect the browsers ability to report unhandled rejections * Ensure unhandled rejections are tested for both inside and outside of a test

Unfortunately, we cannot use real errors here because the stack trace representation changes across Node versions (and therefore the expectations fail).

rwjblue · 2017-12-18T19:50:42Z

Note: I'm happy to squash the commits down when we are ready to go on this one. It is easier to push new commits (e.g. maintainers have an easier time of reviewing the subset of the diff that is new).

platinumazure

Just left a small comment. I promise to give this a full review later 😄

platinumazure · 2017-12-18T20:13:05Z

test/reporter-html/unhandled-rejection.js

+			var originalPushResult = assert.pushResult;
+			assert.pushResult = function( resultInfo ) {
+
+			// Inverts the result so we can test failing assertions


Curious to know why this comment is de-indented from the line below.

Just a mistake, though I'm surprised that the otherwise aggressive indentation linting rules didn't catch it...

No worries. The indent rule is slightly lenient about comments because in general it is hard to tell what lines of code a comment might be associated with, so ESLint generally allows comments to align with the next line of code, the previous line of code, or with the "correct" indentation (i.e., the indentation we would expect a token to actually be at if this were a token instead of a comment).

My guess is the indent rule is allowing this location due to aligning with line 9. That seems weird since line 10 is whitespace-only, so I wonder if we could improve the rule to not use the previous-line heuristic if the previous token is actually 2 lines above the comment instead of 1 (and similarly, not use the next-line heuristic if the next token is actually 2 lines below the comment instead of 1).

Totally makes sense! (also cleaned up the indentation)

platinumazure · 2017-12-18T21:15:49Z

For what it's worth, I've opened eslint/eslint#9733 about the comment indent issue.

trentmwillis

Overall, looks good to me. Just two minor comments.

trentmwillis · 2017-12-19T16:11:17Z

src/core/on-unhandled-rejection.js

+import { sourceFromStacktrace } from "./stacktrace";
+
+// Handle an unhandled rejection
+export default function onUnhandledRejection( reason ) {


Is reason just an Error instance?

No, reason is whatever the rejection value is. An Error instance is almost certainly the most common thing that reason would be, but it could also be a failed XHR, a string, undefined, null, etc. The value here is either whatever the argument to reject() or the argument tothrow is.

See small demo JSBin with the following content:

window.addEventListener('unhandledrejection', function(event) { console.log('reason:', event.reason); }) Promise.reject(); Promise.reject(null); Promise.reject("asdf"); Promise.resolve().then(() => { throw new Error('boom'); })

Ah okay, makes sense. Early-morning me couldn't quite make that connection.

trentmwillis · 2017-12-19T16:13:23Z

test/cli/fixtures/unhandled-rejection.js

+
+		const done = assert.async();
+
+		new Promise( function( resolve ) {


Correct me if I'm wrong, but can't this just be Promise.resolve().then( ... )? If so, can you change it here and below as well.

Yep, you are absolutely correct. I have been working too much with Ember's internal Ember.RSVP.Promise which is configured to resolve in an "odd" way, sorry about that!

Updated to be much more idiomatic...

platinumazure

LGTM, thanks!

trentmwillis · 2017-12-19T17:48:03Z

Looks good to me. @Krinkle any other feedback before I merge?

trentmwillis · 2017-12-20T16:15:45Z

Thanks @rwjblue!

CLI: Ensure an unhandled rejection results in a failed test.

336b32e

Krinkle requested changes Dec 18, 2017

View reviewed changes

rwjblue mentioned this pull request Dec 18, 2017

Update to QUnit 2.5.0 emberjs/ember-qunit#306

Merged

HTML Reporter: Ensure unhandled rejection fails.

bc23a65

Brings unhandled rejections in line with unhandled exceptions.

rwjblue force-pushed the ensure-unhandled-rejections-fail branch from 6249803 to bc23a65 Compare December 18, 2017 17:43

rwjblue force-pushed the ensure-unhandled-rejections-fail branch from 6d92642 to 93007ce Compare December 18, 2017 19:27

rwjblue added 3 commits December 18, 2017 14:28

Core: Add QUnit.onUnhandledRejection.

42eebf7

Tests: Update tests for unhandled rejection scenarios.

f3e30f4

* Properly feature detect the browsers ability to report unhandled rejections * Ensure unhandled rejections are tested for both inside and outside of a test

Core: Capture the proper stack trace in onUnhandledRejection.

9a77ca4

rwjblue force-pushed the ensure-unhandled-rejections-fail branch from 93007ce to bc32c4b Compare December 18, 2017 19:28

Tests: Update CLI test and test fixture to remove stacktrace ambiguity.

6a6fae8

Unfortunately, we cannot use real errors here because the stack trace representation changes across Node versions (and therefore the expectations fail).

rwjblue force-pushed the ensure-unhandled-rejections-fail branch from bc32c4b to 6a6fae8 Compare December 18, 2017 19:34

platinumazure reviewed Dec 18, 2017

View reviewed changes

Tests: Fixup comment indentation.

455ac83

rwjblue mentioned this pull request Dec 18, 2017

assert.async() + async function = gotcha #1229

Closed

trentmwillis requested changes Dec 19, 2017

View reviewed changes

platinumazure approved these changes Dec 19, 2017

View reviewed changes

Tests: Address feedback, use more idiomatic promise chaining.

4b955d1

rwjblue force-pushed the ensure-unhandled-rejections-fail branch from 97eb1c3 to 4b955d1 Compare December 19, 2017 17:02

trentmwillis approved these changes Dec 19, 2017

View reviewed changes

Krinkle approved these changes Dec 20, 2017

View reviewed changes

trentmwillis merged commit 81e0f69 into qunitjs:master Dec 20, 2017

rwjblue deleted the ensure-unhandled-rejections-fail branch December 20, 2017 17:19

rwjblue mentioned this pull request Jul 22, 2020

Any unhandled error causes acceptance tests to fail, making it impossible to test error states emberjs/ember-qunit#592

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ensure unhandled rejections cause test failure. #1241

Ensure unhandled rejections cause test failure. #1241

rwjblue commented Dec 18, 2017 •

edited

Krinkle Dec 18, 2017 •

edited

rwjblue Dec 18, 2017

rwjblue Dec 18, 2017

rwjblue commented Dec 18, 2017

rwjblue commented Dec 18, 2017

platinumazure left a comment

platinumazure Dec 18, 2017

rwjblue Dec 18, 2017

platinumazure Dec 18, 2017 •

edited

rwjblue Dec 18, 2017

platinumazure commented Dec 18, 2017

trentmwillis left a comment

trentmwillis Dec 19, 2017

rwjblue Dec 19, 2017 •

edited

trentmwillis Dec 19, 2017

trentmwillis Dec 19, 2017

rwjblue Dec 19, 2017

platinumazure left a comment

trentmwillis commented Dec 19, 2017

trentmwillis commented Dec 20, 2017

		@@ -0,0 +1,35 @@
		// Currently (2017-12-18) only Chrome has implemented `unhandledRejection`
		const IS_CHROME = !!window.chrome;


		const done = assert.async();

		new Promise( function( resolve ) {

Ensure unhandled rejections cause test failure. #1241

Ensure unhandled rejections cause test failure. #1241

Conversation

rwjblue commented Dec 18, 2017 • edited

Krinkle Dec 18, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rwjblue commented Dec 18, 2017

rwjblue commented Dec 18, 2017

platinumazure left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

platinumazure Dec 18, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

platinumazure commented Dec 18, 2017

trentmwillis left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rwjblue Dec 19, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

platinumazure left a comment

Choose a reason for hiding this comment

trentmwillis commented Dec 19, 2017

trentmwillis commented Dec 20, 2017

rwjblue commented Dec 18, 2017 •

edited

Krinkle Dec 18, 2017 •

edited

platinumazure Dec 18, 2017 •

edited

rwjblue Dec 19, 2017 •

edited