Core: ensure process-queue advances with rejected promises #1391

step2yeung · 2019-04-25T18:21:12Z

Problem:
processTaskQueue processes each task from the task queue as a promise. When the task throws an error, causing the promise to reject, the processing-queue does not continue to process the rest of the tasks queue. The user effect of this problem is qunit hanging, without continuing to process the rest of the task queue, meaning the test will not finish.

In our emberjs app that is uses qunit, the test execution will hang. Testem (the test runner) will recognize the browser has be hanging (using a timeout value) and exit the browser with some number of tests not executed.

Fix:

Make sure we advance() the processing-queue when a promise is rejected.
Make sure errors thrown from callbacks are handled, and subsequent callbacks are called

The fix will throw the error, and continue to process the tasks, so the test run will actually run to completion.

Fixes #1446.

src/core/processing-queue.js

reporter/html.js

src/core/processing-queue.js

src/core/logging.js

src/core/processing-queue.js

mikrostew · 2019-04-30T21:36:45Z

src/test.js

-			} );
+			};
+
+			return promiseChain.then( moduleStartCallback, moduleStartCallback );


Looks like this will swallow any errors at this point - is that the desired behavior here?

@mikrostew I assuming you are talking about const callbackPromises = notStartedModules.reduce(...); ?

if any of the promise from callbackPromises reject.. the whole promise should reject.. which should be handled on line 154:

return callbackPromises.then( testStartHandler, ( err ) => { return Promise.reject( err ).catch( ( err ) => { setTimeout( testStartHandler ); throw err; } ); } );

I was specifically looking at the line

return promiseChain.then( moduleStartCallback, moduleStartCallback );

If something in promiseChain resolves, then it calls the first function argument moduleStartCallback, no problem. But it looks like if it rejects (because of some error) it will call the second function argument, which is also moduleStartCallback, which doesn't do anything with the error that it receives. So that rejection will stop there, so that callbackPromises won't reject.

Read this again, this shouldn't be a problem, but I can understand why it looks like the error is swallowed.
For a bit more context.. const callbackPromises = notStartedModules.reduce(...); ensures the callbacks from notStartedModules are promisfied sequentually, so each callback that was made into a promise, an awaited for until the next callback is executed and made into a promise.
so You will get an chain of promises. In that chain, if there is one rejected promise, then the whole promise chain is rejected.

return promiseChain.then( moduleStartCallback, moduleStartCallback ); in this case, only ensures the next callback is promisfied even if the previous promise rejects. But as stated above, lien 154 will ensure the error is thrown.

I hope that makes more sense.. I think I should likely find a better way to code the explanation above up

src/core/processing-queue.js

rwjblue · 2019-05-01T14:25:49Z

@step2yeung - Would it be possible to split this into two changes (one for the actual bug being fixed along with its test, and another for the refactorings)?

stefanpenner · 2019-05-02T14:59:44Z

@step2yeung could you elaborate on the "problem" in the description above, can you relate your description to a user facing example. Currently, I'm having some trouble connecting the two.

step2yeung · 2019-05-03T19:19:54Z

@stefanpenner Updated the description, hope it helps clarify the user facing issue.

@rwjblue Separated out the fix and the refactor (will submit that afterwards)

stefanpenner · 2019-05-02T14:49:41Z

src/core/processing-queue.js

+				throw err;
+			};
+
+			// Without throwAndAdvance, qunit does not continue advanceing the processing queue if


advanceing -> advancing

stefanpenner · 2019-05-02T14:51:23Z

src/core/processing-queue.js

+
+			// Without throwAndAdvance, qunit does not continue advanceing the processing queue if
+			// task() throws an error
+			Promise.resolve( task() )


If task() throws a synchronous error, the catch bellow which would invoke throwAndAdvance won't be invoked. I'm guessing this isn't intentional, and regardless if task throws synchronously or asynchronously, we would expect the catchAndAdance to be invoked.

The following would rectify that issue.

new Promise(resolve => resolve(task()).then(processNextTaskOrAdance).catch(throwAndAdvance)

stefanpenner · 2019-05-02T14:53:00Z

src/core/processing-queue.js

+			// task() throws an error
+			Promise.resolve( task() )
+				.then( processNextTaskOrAdvance, throwAndAdvance )
+				.catch( throwAndAdvance );


Is there a reason to have throwAndAdvance both in the then's rejection handler, and in the subsequent catch?

As implemented if the throwAndAdvance on line 79 is invoked, it will also cause the throwAndAdvance on line 80 to be invoked.

I believe the intention here may be:

then(processNextTaskOrAdvance).catch(throwAndAdvance)

For added robustness, allow the DOM node to be missing at this point just in case. Extracted from #1391.

For added robustness, allow the DOM node to be missing at this point just in case. Extracted from #1391. Co-authored-by: step2yeung <syeung@linkedin.com>

Krinkle · 2021-07-04T22:56:01Z

src/core/logging.js

+		return promiseChain.then(
+			executeCallback,
+			( err ) => {
+				setTimeout( executeCallback );


Use of setTimeout must be conditional for non-browser environments such as SpiderMonkey (see other references). An alternate approach here can be to use Promise.resolve() as fallback for setTimeout, or to avoid this alltogether by using Promise.finally() for the executeCallback, which naturally forwards the original rejection/error.

Krinkle · 2021-07-04T23:34:46Z

src/core/logging.js

+			executeCallback,
+			( err ) => {
+				setTimeout( executeCallback );
+				throw err;


I suspect that this particular thenable chain might be better without a local error handler as otherwise it means that the last error will be reported instead of the first one. I general QUnit's test logic is modelled to continue after assertion failures, but for uncaught exceptions I think we generally want to stop after the first for any given sequence.

Krinkle · 2021-07-05T02:14:59Z

You may ignore these inline comments. I've uncovered a couple of other things that I'll summarise at #1446.

Krinkle · 2022-05-09T23:07:02Z

reporter/html.js


-		if ( !tests ) {
+		if ( !tests || !testItem ) {


I've extracted and applied this, with credit, via #1562 (released since then).

Krinkle · 2022-05-09T23:10:40Z

src/core.js

@@ -176,7 +176,10 @@ export function begin() {
 		runLoggingCallbacks( "begin", {
 			totalTests: Test.count,
 			modules: modulesLog
-		} ).then( unblockAndAdvanceQueue );
+		} ).then( unblockAndAdvanceQueue, function( err ) {
+			setTimeout( unblockAndAdvanceQueue );


I've added tests for this scenario in #1634, and addressed the bug this fixes (#1446) via a different approach in #1638. In a nutshell, we handle this via the unhandledrejection event in the browser and Node.js, same as before, but previously that didn't work right due to it creating a fake test which we no longer do.

Krinkle · 2022-05-11T17:49:05Z

The one part I believe we still haven't fixed is advancing the process-queue. It makes sense to advance the queue, but, I'd also like to make sure (to avoid future regressions) that the negative impact of this is quantified in an integration test.

For the QUnit CLI (TAP) this is probably not particularly significant as Node.js processes naturally end at this point, and the error is reported with TAP-formatted bail out. For browsers, I imagine the process would remain stuck pending timeout, exactly as before, as we'd not proceed and thus not reach the runEnd event or done() callback. This does mean that we'd run all the tests as well, which is potentially incorrect, but that's kind of the established pattern and before/beforeEach failures are handled the same way.

I'm adding this as a CLI fixture instead of an HTML fixture because we don't currently have a good way of expecting a failure in an HTML test run. I've confirmed that the "hanging" appearance described in #1391 is equivalent to the "Process exited before tests finished running" message reported. Once we switch to capturing TAP by default, this will make the HTML tests much easier to verify no matter whether the pass/fail outcome. Ref #1391.

rwjblue reviewed Apr 25, 2019

View reviewed changes

src/core/processing-queue.js Outdated Show resolved Hide resolved

step2yeung force-pushed the finally branch 4 times, most recently from 2b36e3f to 2f83202 Compare April 29, 2019 17:55

arthirm reviewed Apr 30, 2019

View reviewed changes

reporter/html.js Show resolved Hide resolved

arthirm reviewed Apr 30, 2019

View reviewed changes

src/core/processing-queue.js Outdated Show resolved Hide resolved

mikrostew reviewed Apr 30, 2019

View reviewed changes

stefanpenner suggested changes Apr 30, 2019

View reviewed changes

src/core/processing-queue.js Outdated Show resolved Hide resolved

src/core/processing-queue.js Outdated Show resolved Hide resolved

step2yeung force-pushed the finally branch from 2f83202 to e071239 Compare May 1, 2019 17:57

Core: ensure process-queue advance with rejected promises

091b03b

step2yeung force-pushed the finally branch from e071239 to 091b03b Compare May 3, 2019 18:53

stefanpenner suggested changes May 9, 2019

View reviewed changes

Krinkle force-pushed the master branch from f3744c0 to 6dc314b Compare July 4, 2020 23:52

Krinkle changed the title ~~Core: ensure process-queue advance with rejected promises~~ Core: ensure process-queue advances with rejected promises Nov 13, 2020

Krinkle pushed a commit that referenced this pull request Mar 13, 2021

HTML Reporter: Check for undefined testItem in testDone callback

a5eeac1

For added robustness, allow the DOM node to be missing at this point just in case. Extracted from #1391.

Krinkle mentioned this pull request Mar 13, 2021

HTML Reporter: Check for undefined testItem in testDone callback #1562

Merged

Krinkle pushed a commit that referenced this pull request Mar 13, 2021

HTML Reporter: Check for undefined testItem in testDone callback

900c66e

For added robustness, allow the DOM node to be missing at this point just in case. Extracted from #1391.

Base automatically changed from master to main March 18, 2021 18:59

Krinkle self-assigned this Jul 4, 2021

Krinkle mentioned this pull request Jul 4, 2021

Details of QUnit.begin error are lost #1446

Closed

Krinkle reviewed Jul 4, 2021

View reviewed changes

Krinkle mentioned this pull request Jul 5, 2021

Callback failures should result in cancellation of remaining tests #1633

Closed

Krinkle reviewed May 9, 2022

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Core: ensure process-queue advances with rejected promises #1391

Core: ensure process-queue advances with rejected promises #1391

step2yeung commented Apr 25, 2019 •

edited by Krinkle

mikrostew Apr 30, 2019

step2yeung May 1, 2019

mikrostew May 1, 2019

step2yeung May 1, 2019

rwjblue commented May 1, 2019

stefanpenner commented May 2, 2019

step2yeung commented May 3, 2019

stefanpenner May 2, 2019

stefanpenner May 2, 2019

stefanpenner May 2, 2019 •

edited

Krinkle Jul 4, 2021

Krinkle Jul 4, 2021

Krinkle commented Jul 5, 2021

Krinkle May 9, 2022

Krinkle May 9, 2022

Krinkle commented May 11, 2022 •

edited

Core: ensure process-queue advances with rejected promises #1391

Are you sure you want to change the base?

Core: ensure process-queue advances with rejected promises #1391

Conversation

step2yeung commented Apr 25, 2019 • edited by Krinkle

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rwjblue commented May 1, 2019

stefanpenner commented May 2, 2019

step2yeung commented May 3, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stefanpenner May 2, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Krinkle commented Jul 5, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Krinkle commented May 11, 2022 • edited

step2yeung commented Apr 25, 2019 •

edited by Krinkle

stefanpenner May 2, 2019 •

edited

Krinkle commented May 11, 2022 •

edited