Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stringify stats using streaming approach #2190

Merged
merged 1 commit into from Dec 7, 2020
Merged

Stringify stats using streaming approach #2190

merged 1 commit into from Dec 7, 2020

Conversation

lahmatiy
Copy link
Contributor

@lahmatiy lahmatiy commented Dec 5, 2020

What kind of change does this PR introduce?

This PR changes the approach for stats stringify to use stream instead of JSON.stringify() + sync write. No API or functionality is changed.

Did you add tests for your changes?

No

If relevant, did you update the documentation?

Not needed

Summary

It makes stats available for huge projects. Changes has the follow effects:

  • Memory
    • Allows to avoid max string length limit of V8 engine (500MB). In other words, stats becomes available for huge projects, whose stats exceed 500MB.
    • Reduces memory consumption on serialization.
  • Adds time penalty. However, this make sense for huge stats only and seems to be an acceptable, given that otherwise stats may be unavailable (due max string limit hit or out of memory).
  • Adds new dependency "json-ext", which is dependency free and most effective solution for a huge JSON parse/stringify at the moment. Other solutions have troubles to handle huge JSON (see benchmark).

Does this PR introduce a breaking change?

No

Other information

Tested on AST Explorer (https://github.com/fkling/astexplorer)

Command (cwd website):

npm run build -- --json stats.json

BEFORE:

fs.writeFileSync(dest, JSON.stringify(stats.toJson(...)))

Time: 4583ms
Memory used: +1_850_027_831 (heap: +1_508_459_432, external: +341_568_399)

AFTER:

createJsonStringifyStream(stats.toJson(...)).pipe(fs.createWriteStream(dest))

Time: 5881ms
Memory used: -90_171_587 (heap: -32_339_984, external: -57_831_603)

So it's a bit slower, but much effective in memory consumption.

@lahmatiy lahmatiy requested a review from a team as a code owner December 5, 2020 17:52
@TrySound
Copy link

TrySound commented Dec 5, 2020

I think node 12+ support is too optimistic until April.

@lahmatiy
Copy link
Contributor Author

lahmatiy commented Dec 5, 2020

@TrySound Didn't know that requirement. Will fix it a bit later.

@anshumanv
Copy link
Member

anshumanv commented Dec 5, 2020

I've re-run the CI job, some random install failure Edit - node version incompatibility

Change the approach for stats stringify to use streams instead of JSON.stringify() + sync write. No API or functionality is changed.

It makes stats available for huge projects. Changes has the follow effects:

Memory
Allows to avoid max string length limit of V8 engine (500MB). In other words, stats becomes available for huge projects, whose stats exceed 500MB.
Reduces memory consumption on serialization.
Adds time penalty. However, this make sense for huge stats only and seems to be an acceptable, given that otherwise stats may be unavailable (due max string limit hit or out of memory).
Adds new dependency "json-ext", which is dependency free and most effective solution for a huge JSON parse/stringify at the moment, and other solutions have troubles to handle huge JSON (see https://github.com/discoveryjs/json-ext/blob/master/benchmarks/README.md#stream-stringifying).
@codecov
Copy link

codecov bot commented Dec 5, 2020

Codecov Report

Merging #2190 (9b5d04e) into master (3e5f946) will increase coverage by 2.23%.
The diff coverage is 66.66%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #2190      +/-   ##
==========================================
+ Coverage   66.09%   68.32%   +2.23%     
==========================================
  Files          72       72              
  Lines        2345     2346       +1     
  Branches      518      518              
==========================================
+ Hits         1550     1603      +53     
+ Misses        795      743      -52     
Impacted Files Coverage Δ
packages/webpack-cli/lib/webpack-cli.js 95.83% <66.66%> (+12.18%) ⬆️
packages/migrate/src/index.ts 90.24% <0.00%> (+4.87%) ⬆️
packages/utils/src/prop-types.ts 100.00% <0.00%> (+16.66%) ⬆️
packages/webpack-cli/lib/utils/to-kebab-case.js 100.00% <0.00%> (+33.33%) ⬆️
packages/webpack-cli/lib/utils/cli-flags.js 100.00% <0.00%> (+53.84%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3e5f946...9b5d04e. Read the comment docs.

@lahmatiy
Copy link
Contributor Author

lahmatiy commented Dec 5, 2020

Fixed, now json-ext supports Node.js 10. All tests passed

@alexander-akait alexander-akait merged commit 9bf4e92 into webpack:master Dec 7, 2020
@alexander-akait
Copy link
Member

Big thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants