Refactor yield await classification #12230

JLHwung · 2020-10-21T02:39:06Z

Q	A
Fixed Issues?	Fixes parsing error allowed in test262
Patch: Bug Fix?	Yes
Tests Added + Pass?	Yes
License	MIT

Currently @babel/parser uses parser state to track the position and possible declaration error of yield and await in ambiguous patterns: arrow head ((x)) and async arrow head async(x). Historically we have done several fixing attempts (#6877, #7727, #9405, #10469, #11148, etc.), however this approach is error-prone and can not handle edge cases especially if these patterns are nested: For example, many, if not most, test cases I added in this PR are failing on main but fixed by this PR.

We introduce an ExpressionScope to handle declaration errors in ambiguous patterns. See here for the design of ExpressionScope. The idea is to record such errors and only thrown at the time when such pattern is indeed an arrow function or an async arrow function. The stack based scope play nicely with nested structures: so errors will not be thrown too early or incorrectly reset when later a new structure is parsed.

babel-bot · 2020-10-21T02:43:04Z

Build successful! You can test your changes in the REPL here: https://babeljs.io/repl/build/30997/

codesandbox-ci · 2020-10-21T02:47:44Z

This pull request is automatically built and testable in CodeSandbox.

To see build info of the built libraries, click here or the icon next to each commit SHA.

Latest deployment of this branch, based on commit 46ba92f:

Sandbox	Source
babel-repl-custom-plugin	Configuration
babel-plugin-multi-config	Configuration

JLHwung · 2020-10-21T21:24:53Z

packages/babel-parser/src/parser/expression.js

-      // verify if the same logic is needed for yield.
-      if (oldYieldPos !== -1) this.state.yieldPos = oldYieldPos;
-
-      // Await is trickier than yield. When parsing a possible arrow function


@nicolo-ribaudo I really love reading these informative comments.

...ser/test/fixtures/es2015/yield/parameter-default-inside-arrow-inside-generator-5/output.json

JLHwung · 2020-10-21T22:42:12Z

.../babel-parser/test/fixtures/es2015/yield/parameter-name-arrow-inside-generator-1/output.json

@@ -2,7 +2,7 @@
  "type": "File",
  "start":0,"end":35,"loc":{"start":{"line":1,"column":0},"end":{"line":3,"column":1}},
  "errors": [
-    "SyntaxError: Can not use 'yield' as identifier inside a generator (2:3)",
+    "SyntaxError: yield is not allowed in generator parameters (2:3)",


babel/packages/babel-parser/test/fixtures/es2015/yield/parameter-default-inside-arrow-inside-generator-1/input.js

Lines 1 to 3 in d51aa6d

function* fn() {

(x = yield) => {};

}

The error message seems a bit misleading as it is not a generator, although it is indeed in function parameters. V8 has changed this error message to "Yield expression not allowed in formal parameter" (code), I suggest we align with V8 and update the error in following PRs.

JLHwung · 2020-10-21T22:47:47Z

...2017/async-functions/await-binding-inside-arrow-params-inside-async-arrow-params/output.json

@@ -2,7 +2,7 @@
  "type": "File",
  "start":0,"end":36,"loc":{"start":{"line":1,"column":0},"end":{"line":1,"column":36}},
  "errors": [
-    "SyntaxError: Can not use 'await' as identifier inside an async function (1:20)"
+    "SyntaxError: Can not use 'await' as identifier inside an async function (1:14)"


babel/packages/babel-parser/test/fixtures/es2017/async-functions/await-binding-inside-arrow-params-inside-async-arrow-params/input.js

Line 1 in d51aa6d

async (a = ({ await }) => {}) => {};

The position of await should be 14. An unexpected bugfix. 😄

…undary

packages/babel-parser/src/util/expression-scope.js

Co-authored-by: Nicolò Ribaudo <nicolo.ribaudo@gmail.com>

nicolo-ribaudo

This is awesome 🎉

JLHwung added PR: Bug Fix 🐛 A type of pull request used for our changelog categories pkg: parser labels Oct 21, 2020

JLHwung force-pushed the refactor-yield-await-tracking branch from 15773bb to ae9acf0 Compare October 21, 2020 20:50

JLHwung marked this pull request as ready for review October 21, 2020 21:20

JLHwung commented Oct 21, 2020

View reviewed changes

nicolo-ribaudo self-requested a review October 21, 2020 22:14

JLHwung commented Oct 21, 2020

View reviewed changes

...ser/test/fixtures/es2015/yield/parameter-default-inside-arrow-inside-generator-5/output.json Outdated Show resolved Hide resolved

JLHwung commented Oct 21, 2020

View reviewed changes

JLHwung added 12 commits October 23, 2020 21:11

fix: incomplete ParamKind declaration

ab70bde

refactor: add expression scope handler

bf3b3a2

test: update test262 allowlist

759af83

chore: cleanup

7086565

fix: push expression scope for function body

7751de4

fix: push new expression scope for initializer and static block

fdd7644

test: add more test cases

455dfdf

fix flow error

9aa7aaf

refactor: remove unecessary expression scope

29dfce9

fix: parameter initializer error should not cross expression scope bo…

b69b2da

…undary

chore: cleanup outdated comments

1c06e4e

fix: do not record async arrow error on ParameterDeclaration

8c92bd9

JLHwung force-pushed the refactor-yield-await-tracking branch from b0f87cf to 8c92bd9 Compare October 24, 2020 01:13

nicolo-ribaudo reviewed Oct 24, 2020

View reviewed changes

packages/babel-parser/src/util/expression-scope.js Outdated Show resolved Hide resolved

packages/babel-parser/src/util/expression-scope.js Outdated Show resolved Hide resolved

JLHwung and others added 2 commits October 24, 2020 20:49

Update packages/babel-parser/src/util/expression-scope.js

51bdc1a

Co-authored-by: Nicolò Ribaudo <nicolo.ribaudo@gmail.com>

polish: clear ancestry declaration error on validate

46ba92f

JLHwung requested a review from nicolo-ribaudo October 25, 2020 01:24

nicolo-ribaudo approved these changes Oct 25, 2020

View reviewed changes

existentialism approved these changes Oct 26, 2020

View reviewed changes

JLHwung merged commit 2782a54 into babel:main Oct 26, 2020

JLHwung deleted the refactor-yield-await-tracking branch October 26, 2020 15:42

JLHwung mentioned this pull request Oct 26, 2020

Polish parser errors #12258

Merged

JLHwung mentioned this pull request Nov 7, 2020

fix: disallow all parenthesized pattern except parsing LHS #12327

Merged

github-actions bot added the outdated A closed issue/PR that is archived due to age. Recommended to make a new issue label Jan 26, 2021

github-actions bot locked as resolved and limited conversation to collaborators Jan 26, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor yield await classification #12230

Refactor yield await classification #12230

JLHwung commented Oct 21, 2020 •

edited by gitpod-io bot

babel-bot commented Oct 21, 2020 •

edited

codesandbox-ci bot commented Oct 21, 2020 •

edited

JLHwung Oct 21, 2020

JLHwung Oct 21, 2020 •

edited

JLHwung Oct 21, 2020 •

edited

nicolo-ribaudo left a comment

Refactor yield await classification #12230

Refactor yield await classification #12230

Conversation

JLHwung commented Oct 21, 2020 • edited by gitpod-io bot

babel-bot commented Oct 21, 2020 • edited

codesandbox-ci bot commented Oct 21, 2020 • edited

JLHwung Oct 21, 2020

Choose a reason for hiding this comment

JLHwung Oct 21, 2020 • edited

Choose a reason for hiding this comment

JLHwung Oct 21, 2020 • edited

Choose a reason for hiding this comment

nicolo-ribaudo left a comment

Choose a reason for hiding this comment

JLHwung commented Oct 21, 2020 •

edited by gitpod-io bot

babel-bot commented Oct 21, 2020 •

edited

codesandbox-ci bot commented Oct 21, 2020 •

edited

JLHwung Oct 21, 2020 •

edited

JLHwung Oct 21, 2020 •

edited