CLI: Cannot create a string longer than 0x1fffffe8 characters

[taras-turchenko-moc]

Issue description

We have front-end heavy application with ~40 entry points and i want to optimize build size

$ rm -rf ./tmp/webpack-stats.json

$ yarn build --progress --profile --json > ./tmp/webpack-stats.json && npx webpack-bundle-analyzer ./tmp/webpack-stats.json
Couldn't read webpack bundle stats from "/Users/tarasturchenko/Projects/work/ZipifyPages/tmp/webpack-stats.json":
Error: Cannot create a string longer than 0x1fffffe8 characters

$ ll ./tmp/webpack-stats.json
-rw-r--r--  1 tarasturchenko  staff   1.0G Feb  4 11:28 ./tmp/webpack-stats.json

Technical info

  System:
    OS: macOS 12.0.1
    CPU: (12) x64 Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
    Memory: 487.04 MB / 16.00 GB
    Shell: 5.8 - /bin/zsh
  Binaries:
    Node: 14.18.1 - ~/.nvm/versions/node/v14.18.1/bin/node
    Yarn: 1.22.17 - ~/.nvm/versions/node/v14.18.1/bin/yarn
    npm: 8.1.4 - ~/.nvm/versions/node/v14.18.1/bin/npm
  npmPackages:
    compression-webpack-plugin: ^9.0.0 => 9.0.0
    image-minimizer-webpack-plugin: ^3.1.2 => 3.1.2
    webpack: 5.64.1 => 5.64.1
    webpack-cli: 4.9.1 => 4.9.1
    webpack-manifest-plugin: ^4.0.2 => 4.0.2
    webpack-merge: ^5.8.0 => 5.8.0

Debug info

How do you use this module? As CLI utility or as plugin?

If CLI, what command was used? (e.g. webpack-bundle-analyzer -O path/to/stats.json)

npx webpack-bundle-analyzer ./tmp/webpack-stats.json

If plugin, what options were provided? (e.g. new BundleAnalyzerPlugin({ analyzerMode: 'disabled', generateStatsFile: true }))

What other Webpack plugins were used?

• image-minimizer-webpack-plugin
• mini-css-extract-plugin
• compression-webpack-plugin
• webpack-manifest-plugin
• IgnorePlugin & ProvidePlugin
• vue-loader

It would be nice to also attach webpack stats file.
It can be generated using these options:

new BundleAnalyzerPlugin({
  analyzerMode: 'disabled',
  generateStatsFile: true,
  // Excludes module sources from stats file so there won't be any sensitive data
  statsOptions: { source: false }
})

stats.json will be created in Webpack bundle output directory.

[valscion]

Oh wow, I've never seen an error like that 😅 sounds like something is generating an awfully long string 😅

I don't know even where to start to fix this. Would you be open to do some googling and figure out if this error has showed up elsewhere and if there might be some common code patterns that might cause this error?

[th0r]

I think the reason is webpack-stats.json size is 1gb and we're using something like readFileSync to read it as a string.

[th0r]

I think here are the problematic code:

webpack-bundle-analyzer/src/analyzer.js

Lines 164 to 168 in e5b617e

	function readStatsFromFile(filename) {
	return JSON.parse(
	fs.readFileSync(filename, 'utf8')
	);
	}

Maybe there is some package that parses JSON without reading it into a string first.
@sokra maybe you've already solved such problem?

[taras-turchenko-moc]

I'll try stream-json. Seems like this lib can do it but it takes too long. I started parsing 30 minutes ago and it's still running

Also i will try how works the webpack plugin with it

[taras-turchenko-moc]

It's finished and it takes ~30 minutes just to parse this json. Also i've tried the webpack plugin and it works fine

[sushantmakin]

Hi,
What is the way around this? the stats.json which is created is >512 MB (840 MB), hence it fails with the same error for me as well with the same error.

[taras-turchenko-moc]

@sushantmakin Webpack plugin works fine for me

https://github.com/webpack-contrib/webpack-bundle-analyzer#usage-as-a-plugin

[tgudroe]

Is there a solution to this? We also have a very large stats.json file that is failing when trying to analyze it.

[taras-turchenko-moc]

@tgudroe the webpack plugin works fine

[tgudroe]

Does that mean the CLI can't be used in my case and there is no hope of it working in the future?

[taras-turchenko-moc]

@tgudroe I'll take a look at this later. Maybe we can fix it by replacing fs.readfile to one of binary json readers

[joshunger]

When I upload the same stats.json to http://chrisbateman.github.io/webpack-visualizer/ I'm seeing this error. Is this related?

VM33:1 Uncaught SyntaxError: Unexpected end of JSON input
    at JSON.parse (<anonymous>)
    at e.handleFileUpload (build.js:18:5124)
    at n.onloadend (build.js:23:9878)

[tiberiuzuld]

Hello,
I encountered this issue with a large angular app.
Using the changes in this PR worked for me #423

[haven2world]

Same issue. Hope this can be fixed.

[valscion]

The workaround is to use the plugin and not CLI

[haven2world]

The workaround is to use the plugin and not CLI

I knew that. But sometimes I need to compare the dependency graph between branches. It takes time to switch branch, install dependency and build the project. In this case, it's much more convenient to leverage the existing stat.json

[valscion]

This should now be fixed in v4.8.0 thanks to #423! 🚀

Feel free to take it for a spin and let us know if the issue wasn't solved with that PR as expected.

[tiberiuzuld]

Hello,
A bit late to the party, but only now I had the time to test this with our application.
I tested this fix and it worked for me to load a 548 MB file.

Thank you for the fix.