Add ability to disable whitespace control: `Handlebars.parseWithoutProcessing` #1584

rwjblue · 2019-10-25T16:14:36Z

When authoring tooling that parses Handlebars files and emits Handlebars files, you often want to preserve the exact formatting of the input. The changes in this commit add a new method to the Handlebars namespace: parseWithoutProcessing. Unlike, Handlebars.parse (which will mutate the parsed AST to apply whitespace control) this method will parse the template and return it directly (without processing 😉).

For example, parsing the following template (feel free to review the AST on ASTExplorer):

 {{#foo}}
   {{~bar~}} {{baz~}}
 {{/foo}}

Using Handlebars.parse, the AST returned would have truncated the following whitespace:

The whitespace prior to the {{#foo}}
The newline following {{#foo}}
The leading whitespace before {{~bar~}}
The whitespace between {{~bar~}} and {{baz~}}
The newline after {{baz~}}
The whitespace prior to the {{/foo}}

When Handlebars.parse is used from Handlebars.precompile or Handlebars.compile, this whitespace stripping is very important (these behaviors are intentional, and generally lead to better rendered output).

When the same template is parsed with Handlebars.parseWithoutProcessing none of those modifications to the AST are made. This enables "codemod tooling" (e.g. prettier and ember-template-recast) to preserve the exact initial formatting.

Prior to these changes, those tools would have to manually reconstruct the whitespace that is lost prior to emitting source.

scalvert · 2019-10-25T16:20:05Z

This will definitely address a pain point WRT recast and whitespace, that I myself have definitely encountered. Thanks for this. 🙏

nknapp · 2019-10-25T20:04:44Z

I can see the need, but I don't think that a compile-option is the right way to go here. A compile-option would suggest that the use-case is compiling and executing a template with whitespace-control disabled.

But all you want is to build a clean, unmodified AST, correct?

I think a second "parse" function should be exposed for the purpose of building tools. Maybe there are other modifications that are useful for tooling, that could be built into the second function.

Consequently, the test-cases should not be along the lines of "shouldCompileTo" but more like the tests in the "ast.js"-spec: Build an AST and compare it to the expected one.

Does that make sense?

rwjblue · 2019-10-25T20:26:05Z

I don't think that a compile-option is the right way to go here. A compile-option would suggest that the use-case is compiling and executing a template with whitespace-control disabled.

The PR adds the option to parse, not compile, and the types call this object ParseOption, (which is different from CompileOptions). Am I thinking about the structure incorrectly? FWIW, I do find it very odd that handling whitespace control is part of "parsing" (as opposed to compiling). I just don't know how to fix that as it's a bit of a long standing thing.

I think a second "parse" function should be exposed for the purpose of building tools. Maybe there are other modifications that are useful for tooling, that could be built into the second function.

Sure, that seems totally fine to me as well. What do you think it should be called? Where should it go?

Consequently, the test-cases should not be along the lines of "shouldCompileTo" but more like the tests in the "ast.js"-spec: Build an AST and compare it to the expected one.

Sounds good, thank you for reviewing!

rwjblue · 2019-10-25T21:45:00Z

@nknapp - I just pushed another commit that separates parse into two functions (one that parses the input template string, and the other that leverages the raw parse output and adds whitespace control), as well as adding a number of additional tests to spec/ast.js.

Is that along the lines of what you were thinking? Do you have a better idea on the name of the (currently named parseAST) function?

nknapp · 2019-10-27T14:55:11Z

How about parseWithoutProcessing

But "raw" is already taken for {{{{raw-blocks}}}} {{{{/raw-blocks}}}}. This might be confusing.
I am not sure about this either. The name might be only temporary, because we have plans to extract the parser into a separate package. When we do that, we can choose proper names for the function or even leave the white-space processing out of the package completely.

But we don't have a timeline. I can't make one with the small amount of work I can put into this every day.

nknapp

I have added a review (this makes the current state more visible in the PR overview).

lib/handlebars/compiler/base.js

types/index.d.ts

rwjblue · 2019-10-27T16:31:04Z

Thanks for your time @nknapp! I've updated with your latest round of feedback:

Rename method to parseWithoutProcessing
Update types for new method
Add types/test.ts for new API

spec/ast.js

rwjblue · 2019-10-27T17:24:46Z

OK, just pushed another couple of changes:

Added documentation for parseWithoutProcessing to docs/compiler-api.md
Add explicit tests to spec/ast.js for using whitespace control (with parse and parseWithoutProcessing)

nknapp · 2019-10-27T19:13:03Z

Thank you. I will have another glance but all in all the PR looks good to me now.

nknapp

One single change, then we are done...

types/test.ts

When authoring tooling that parses Handlebars files and emits Handlebars files, you often want to preserve the **exact** formatting of the input. The changes in this commit add a new method to the `Handlebars` namespace: `parseWithoutProcessing`. Unlike, `Handlebars.parse` (which will mutate the parsed AST to apply whitespace control) this method will parse the template and return it directly (**without** processing :wink:). For example, parsing the following template: ```hbs {{#foo}} {{~bar~}} {{baz~}} {{/foo}} ``` Using `Handlebars.parse`, the AST returned would have truncated the following whitespace: * The whitespace prior to the `{{#foo}}` * The newline following `{{#foo}}` * The leading whitespace before `{{~bar~}}` * The whitespace between `{{~bar~}}` and `{{baz~}}` * The newline after `{{baz~}}` * The whitespace prior to the `{{/foo}}` When `Handlebars.parse` is used from `Handlebars.precompile` or `Handlebars.compile`, this whitespace stripping is **very** important (these behaviors are intentional, and generally lead to better rendered output). When the same template is parsed with `Handlebars.parseWithoutProcessing` none of those modifications to the AST are made. This enables "codemod tooling" (e.g. `prettier` and `ember-template-recast`) to preserve the **exact** initial formatting. Prior to these changes, those tools would have to _manually_ reconstruct the whitespace that is lost prior to emitting source.

rwjblue · 2019-10-27T23:22:20Z

Just pushed another round of updates:

Fixed issues in types/test.ts
Squashed commits down to one
Updated description and commit messages to match current implementation

@nknapp - Please let me know if there is anything else that needs some tweaks. 😸

nknapp

Thanks a lot for your patience and persistence in this.

I haven't decided on the date of the next release yet. Is this urgent for you or can it wait another couple of weeks?

nknapp · 2019-10-27T23:27:48Z

I would have done the squashing myself, but well OK.

I noticed that unsquashed commits are much easier to review.

rwjblue · 2019-10-27T23:28:12Z

I haven't decided on the date of the next release yet. Is this urgent for you or can it wait another couple of weeks.

There are some bugs that are a bit difficult to work around in the codemod tooling that I am maintaining (things like ember-template-lint/ember-template-recast#155) that this helps resolve (once updated in @glimmer/syntax). Now that I know how the future versions of handlebars will support this, I can probably do a (slightly hacky) work around while we wait for 4.5.0.

tldr; I think I can "polyfill" this new method in my codemod tooling, if I run into issues there though I will let you know...

rwjblue · 2019-10-27T23:31:46Z

I would have done the squashing myself, but well OK.

Gotcha. I also wanted to fix up the commit message a bit (the main commit message was still talking about the original option I proposed).

I noticed that unsquashed commits are much easier to review.

Yes, I whole heartedly agree. The GitHub UI makes reviewing the delta between commits super simple, but when folks continually squash / amend that workflow is broken. FWIW, I have had some luck with the new "mark file as viewed" functionality in the GitHub pull request UI (this works across squashes), but it's still not as nice as leaving it unsquashed.

nknapp · 2019-10-28T05:15:10Z

Before you start building workarounds... I would like wait for #1583 to be ginishrd. Then I could probably release 4.5 on the next weekend.

It won't be more than a couple of weeks at most. Maybe you can use a github dependency until then.

nknapp · 2019-10-28T05:16:46Z

If you want this for ember, you probably need a backport to 3.x, don't you?

nknapp · 2019-10-28T15:00:49Z

I might be able to release this evening. Do you know if you need the feature in 4.x or 3.x?

rwjblue · 2019-10-28T18:02:32Z

If you want this for ember, you probably need a backport to 3.x, don't you?

We are using v4 over in @glimmer/syntax (which is what Ember uses for parsing).

It won't be more than a couple of weeks at most.

Ya, definitely no worries on timing from me. Thank you again for all of your help getting this into a good state.

nknapp · 2019-10-28T18:29:11Z

We are using v4 over in @glimmer/syntax (which is what Ember uses for parsing).

Really? A while ago @wycats asked for my opinion on how to proceed with the parser. He said Ember was using an older version and he was thinking about how to converge the code-base again, because Handlebars has features that are not used by Ember and Ember has other features like components.

But OK. It's a good to know that you are using the current version.

nknapp · 2019-10-28T19:32:35Z

I have release 4.5 now. It was just good to have all this wrapped up. (will take a break now).

rwjblue mentioned this pull request Oct 25, 2019

If block with whitespace control is mutated during transform ember-template-lint/ember-template-recast#155

Closed

nknapp requested changes Oct 27, 2019

View reviewed changes

lib/handlebars/compiler/base.js Outdated Show resolved Hide resolved

types/index.d.ts Outdated Show resolved Hide resolved

nknapp reviewed Oct 27, 2019

View reviewed changes

spec/ast.js Show resolved Hide resolved

nknapp requested changes Oct 27, 2019

View reviewed changes

types/test.ts Outdated Show resolved Hide resolved

nknapp added this to the 4.5 milestone Oct 27, 2019

rwjblue force-pushed the allow-disabling-whitespace-control branch from 1f1e071 to b58a228 Compare October 27, 2019 23:16

rwjblue changed the title ~~Add ability to disable whitespace control.~~ Add ability to disable whitespace control: Handlebars.parseWithoutProcessing Oct 27, 2019

nknapp approved these changes Oct 27, 2019

View reviewed changes

nknapp merged commit 62ed3c2 into handlebars-lang:4.x Oct 27, 2019

rwjblue deleted the allow-disabling-whitespace-control branch October 27, 2019 23:31

This was referenced Oct 28, 2019

Ensure codemod mode ignores whitespace control. glimmerjs/glimmer-vm#983

Merged

[QUEST] Glimmer jgwhite/prettier#1

Closed

Support whitespace control (~) ember-codemods/ember-angle-brackets-codemod#46

Closed

nknapp mentioned this pull request Oct 28, 2019

Resolve deprecation warning message from eslint while running eslint #1586

Merged

8 tasks

rwjblue mentioned this pull request Oct 29, 2019

Update to handlebars@4.5.1. glimmerjs/glimmer-vm#986

Merged

nknapp mentioned this pull request Dec 17, 2019

Ability to pass more than one variable data to a template #1611

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add ability to disable whitespace control: `Handlebars.parseWithoutProcessing` #1584

Add ability to disable whitespace control: `Handlebars.parseWithoutProcessing` #1584

rwjblue commented Oct 25, 2019 •

edited

scalvert commented Oct 25, 2019

nknapp commented Oct 25, 2019

rwjblue commented Oct 25, 2019 •

edited

rwjblue commented Oct 25, 2019 •

edited

nknapp commented Oct 27, 2019 •

edited

nknapp left a comment

rwjblue commented Oct 27, 2019

rwjblue commented Oct 27, 2019

nknapp commented Oct 27, 2019

nknapp left a comment

rwjblue commented Oct 27, 2019

nknapp left a comment •

edited

nknapp commented Oct 27, 2019

rwjblue commented Oct 27, 2019

rwjblue commented Oct 27, 2019

nknapp commented Oct 28, 2019

nknapp commented Oct 28, 2019

nknapp commented Oct 28, 2019 •

edited

rwjblue commented Oct 28, 2019

nknapp commented Oct 28, 2019

nknapp commented Oct 28, 2019

Add ability to disable whitespace control: Handlebars.parseWithoutProcessing #1584

Add ability to disable whitespace control: Handlebars.parseWithoutProcessing #1584

Conversation

rwjblue commented Oct 25, 2019 • edited

scalvert commented Oct 25, 2019

nknapp commented Oct 25, 2019

rwjblue commented Oct 25, 2019 • edited

rwjblue commented Oct 25, 2019 • edited

nknapp commented Oct 27, 2019 • edited

nknapp left a comment

Choose a reason for hiding this comment

rwjblue commented Oct 27, 2019

rwjblue commented Oct 27, 2019

nknapp commented Oct 27, 2019

nknapp left a comment

Choose a reason for hiding this comment

rwjblue commented Oct 27, 2019

nknapp left a comment • edited

Choose a reason for hiding this comment

nknapp commented Oct 27, 2019

rwjblue commented Oct 27, 2019

rwjblue commented Oct 27, 2019

nknapp commented Oct 28, 2019

nknapp commented Oct 28, 2019

nknapp commented Oct 28, 2019 • edited

rwjblue commented Oct 28, 2019

nknapp commented Oct 28, 2019

nknapp commented Oct 28, 2019

Add ability to disable whitespace control: `Handlebars.parseWithoutProcessing` #1584

Add ability to disable whitespace control: `Handlebars.parseWithoutProcessing` #1584

rwjblue commented Oct 25, 2019 •

edited

rwjblue commented Oct 25, 2019 •

edited

rwjblue commented Oct 25, 2019 •

edited

nknapp commented Oct 27, 2019 •

edited

nknapp left a comment •

edited

nknapp commented Oct 28, 2019 •

edited