Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tokenize keywords-like identifier as new tokens #13769

Merged
merged 6 commits into from Sep 23, 2021

Conversation

JLHwung
Copy link
Contributor

@JLHwung JLHwung commented Sep 15, 2021

Q                       A
Fixed Issues? Babel parser incorrectly allows interf\u{61}ce A {} when flows is enabled.
Patch: Bug Fix? Y
Major: Breaking Change?
Minor: New Feature?
Tests Added + Pass? Yes
Documentation PR Link
Any Dependency Changes?
License MIT

This PR is based on #13768. The tt.name is now exploded into multiple tokens, each of them represents an "interested" identifier name (such as async, static) which subjects to additional parsing logic. The tokens is not limited to ES, the Flow/TS keyword-likes are also tokenized differently for parsing convenience.

The isContextual, expectContextual now accept a token type instead of a string since these names are tokenized into different types. The verification is simplified because, e.g. token === tt._async already implies that token is an identifier token.

@codesandbox-ci
Copy link

codesandbox-ci bot commented Sep 15, 2021

This pull request is automatically built and testable in CodeSandbox.

To see build info of the built libraries, click here or the icon next to each commit SHA.

Latest deployment of this branch, based on commit 6fdea76:

Sandbox Source
babel-repl-custom-plugin Configuration
babel-plugin-multi-config Configuration

@babel-bot
Copy link
Collaborator

babel-bot commented Sep 15, 2021

Build successful! You can test your changes in the REPL here: https://babeljs.io/repl/build/48823/

@JLHwung JLHwung force-pushed the tokenize-keywords-like-identifier branch from 3bd0561 to 1ccd28b Compare September 17, 2021 14:59
@JLHwung JLHwung added pkg: parser PR: Bug Fix 🐛 A type of pull request used for our changelog categories labels Sep 17, 2021
@JLHwung JLHwung marked this pull request as ready for review September 17, 2021 15:14
lookahead.type === tt.string ||
lookahead.type === tt.num
) {
if (tokenIsLiteralPropertyName(lookahead.type)) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a bug fix: Flow allows any identifiers to be the accessor type property. e.g.

type A = { get if(): number }

is a valid Flow code.

this.match(tt.name) &&
this.state.value === "interface"
) {
if (this.state.strict && this.isContextual(tt._interface)) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a bug fix:

interf\u{61}ce A {}

is invalid Flow code.

@JLHwung JLHwung added PR: Spec Compliance 👓 A type of pull request used for our changelog categories and removed PR: Bug Fix 🐛 A type of pull request used for our changelog categories labels Sep 17, 2021
@nicolo-ribaudo
Copy link
Member

Ugh this title is going to mean nothing when it appears in the changelog under "Spec Compliance" 😬
We can label it both as Spec Compliance and Internal, and then I'll manually edit the changelog.

Copy link
Member

@existentialism existentialism left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

@JLHwung JLHwung merged commit 178d43f into babel:main Sep 23, 2021
@JLHwung JLHwung deleted the tokenize-keywords-like-identifier branch September 23, 2021 14:54
@github-actions github-actions bot added the outdated A closed issue/PR that is archived due to age. Recommended to make a new issue label Dec 24, 2021
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Dec 24, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
outdated A closed issue/PR that is archived due to age. Recommended to make a new issue pkg: parser PR: Spec Compliance 👓 A type of pull request used for our changelog categories
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants