webpack --profile --json "RangeError: Invalid string length" when outputting _huge_ JSON file

[simonflk]

Do you want to request a feature or report a bug?

Bug

What is the current behavior?

When running webpack with --profile --json flags, and the JSON data to be output is rather big, it fails with this error:

			process.stdout.write(JSON.stringify(stats.toJson(outputOptions), null, 2) + "\n");
			                          ^

RangeError: Invalid string length
    at join (native)
    at Object.stringify (native)
    at compilerCallback (/home/simon/dev/otrl/booking-engine/node_modules/webpack/bin/webpack.js:337:30)

If the current behavior is a bug, please provide the steps to reproduce.

Well, I guess you need a project with enough entry points, and a complicated enough module graph that the size of the profile is quite large. I will try to create a reproducible repo, but actually this is demonstrable with node & no dependencies:

nodejs/node-v0.x-archive#14170

What is the expected behavior?

The JSON should be output to stdout

If this is a feature request, what is motivation or use case for changing the behavior?

Please mention other relevant information such as the browser version, Node.js version, webpack version and Operating System.

• node v6.10.0
• webpack 2.3.3
• Ubuntu 16.04.1 LTS

[simonflk]

With the fix from #4630, I was able to output the JSON.
JSON file is 243MB.

[unsafecode]

Any chance of having this included in the next release? Manual patching webpack.js is definitely not the best approach...

[sokra]

v8 supports larger strings soon

https://twitter.com/bmeurer/status/889450001709096962

[Astridax]

So using Node v10, I managed to make one of these stats files. I had to give Node 16GB of ram and the resultant file was 870MB!

There must be a better way than this to analyse webpack bundles.

[kevinbarabash]

I'm still seeing this even after upgrading to node 10. The exact version I'm running is 10.15.0.

[cjlarose]

I have a large project with many entry points and I also get this error after upgrading to NodeJS 10.15.3.

I was able to modify webpack-cli so that it streams out the JSON representation of the Webpack stats object here: cjlarose/webpack-cli@0169c33.

In my case, that fork successfully generated a JSON file containing 1447518306 bytes. This is larger than the maximum string size supported by NodeJS 10 on a 64-bit platform (1073741799 bytes)

In short, I don't think that relying on larger string sizes in NodeJS 10 is an appropriate way to dismiss this issue. There are real projects that have stats objects that are really large, and many of those projects are exactly the kinds of projects that have a strong need to gather statistics from webpack in order to analyze and optimize bundles.

Would a change to write the stats object as JSON using a stream be a welcome change?

[davidnormo]

We have a 1.6GB stats file (with @cjlarose's nice streaming addition to webpack-cli) however all the analysis tools fail to load a stat file that big. I.e. getting the stats file is only half the battle - analysing something that big is the next challenge.

[DenyVeyten]

Having the same issue, node v14.15.1, webpack 4.43.0, webpack-cli 3.3.5:

mode_modules\webpack-cli\bin\cli.js:320
                                        stdout.write(JSON.stringify(stats.toJson(outputOptions), null, 2) + "\n");
                                                          ^

RangeError: Invalid string length

[alexander-akait]

fixed, will be available in the next release of webpack-cli

[cjlarose]

I think this is the PR that fixes the issue webpack/webpack-cli#2190