Webpack 3 RC ModuleConcatenationPlugin crash, out of memory

[chrisui]

Do you want to request a feature or report a bug?

Bug

What is the current behavior?

Webpack crashes trying to build our application when using Webpack.optimize.ModuleConcatenationPlugin().

Also note up until the crash the build time is also showing a >60% increase in regards to build time with this plugin.

If the current behavior is a bug, please provide the steps to reproduce.

Building a large application (3000-4000 modules), with several entry point, a few commons chunks, pretty aggressive code-splitting using import() everywhere and uglified with source-maps.

What is the expected behavior?

No crash without having to bump memory usage. Note the build time increase is pretty significant too - wouldn't expect a single plugin to have such a drastic impact.

Please mention other relevant information such as the browser version, Node.js version, webpack version and Operating System.

OS: Mac OSX
Node.JS: v6.9.1
Webpack version: 3.0.0-rc.2

Stack:

<--- Last few GCs --->

  471727 ms: Mark-sweep 1343.8 (1443.3) -> 1343.2 (1443.3) MB, 1503.7 / 0.0 ms [allocation failure] [scavenge might not succeed].
  473238 ms: Mark-sweep 1343.2 (1443.3) -> 1343.2 (1443.3) MB, 1509.9 / 0.0 ms [allocation failure] [scavenge might not succeed].
  474819 ms: Mark-sweep 1343.2 (1443.3) -> 1343.2 (1427.3) MB, 1579.0 / 0.1 ms [last resort gc].
  476439 ms: Mark-sweep 1343.2 (1427.3) -> 1343.2 (1427.3) MB, 1619.3 / 0.0 ms [last resort gc].


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x24ab363cfb51 <JS Object>
    1: stringify [native json.js:~176] [pc=0x138f1ce98049] (this=0x24ab363c9111 <a JSON with map 0x8bbe1d0a0e1>,E=0x362ddef793a9 <an Object with map 0x26615909c261>,F=0x24ab36304381 <undefined>,S=0x24ab36304381 <undefined>)
    2: arguments adaptor frame: 1->3
    3: source(aka getSource) [/Users/chris/Projects/frontend/node_modules/stats-webpack-plugin/index.js:25] [pc=0x138f1d931920] (this=0x...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [/Users/chris/.nvm/versions/node/v6.9.1/bin/node]
 2: node::FatalException(v8::Isolate*, v8::Local<v8::Value>, v8::Local<v8::Message>) [/Users/chris/.nvm/versions/node/v6.9.1/bin/node]
 3: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/Users/chris/.nvm/versions/node/v6.9.1/bin/node]
 4: v8::internal::Factory::NewRawTwoByteString(int, v8::internal::PretenureFlag) [/Users/chris/.nvm/versions/node/v6.9.1/bin/node]
 5: v8::internal::IncrementalStringBuilder::Extend() [/Users/chris/.nvm/versions/node/v6.9.1/bin/node]
 6: v8::internal::BasicJsonStringifier::SerializeString(v8::internal::Handle<v8::internal::String>) [/Users/chris/.nvm/versions/node/v6.9.1/bin/node]
 7: v8::internal::BasicJsonStringifier::Result v8::internal::BasicJsonStringifier::Serialize_<true>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [/Users/chris/.nvm/versions/node/v6.9.1/bin/node]
 8: v8::internal::BasicJsonStringifier::Result v8::internal::BasicJsonStringifier::Serialize_<false>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [/Users/chris/.nvm/versions/node/v6.9.1/bin/node]
 9: v8::internal::BasicJsonStringifier::SerializeJSArraySlow(v8::internal::Handle<v8::internal::JSArray>, unsigned int, unsigned int) [/Users/chris/.nvm/versions/node/v6.9.1/bin/node]
10: v8::internal::BasicJsonStringifier::Result v8::internal::BasicJsonStringifier::Serialize_<true>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [/Users/chris/.nvm/versions/node/v6.9.1/bin/node]
11: v8::internal::BasicJsonStringifier::Result v8::internal::BasicJsonStringifier::Serialize_<false>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [/Users/chris/.nvm/versions/node/v6.9.1/bin/node]
12: v8::internal::BasicJsonStringifier::SerializeJSArraySlow(v8::internal::Handle<v8::internal::JSArray>, unsigned int, unsigned int) [/Users/chris/.nvm/versions/node/v6.9.1/bin/node]
13: v8::internal::BasicJsonStringifier::Result v8::internal::BasicJsonStringifier::Serialize_<true>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [/Users/chris/.nvm/versions/node/v6.9.1/bin/node]
14: v8::internal::BasicJsonStringifier::Result v8::internal::BasicJsonStringifier::Serialize_<false>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [/Users/chris/.nvm/versions/node/v6.9.1/bin/node]
15: v8::internal::BasicJsonStringifier::SerializeJSArraySlow(v8::internal::Handle<v8::internal::JSArray>, unsigned int, unsigned int) [/Users/chris/.nvm/versions/node/v6.9.1/bin/node]
16: v8::internal::BasicJsonStringifier::Result v8::internal::BasicJsonStringifier::Serialize_<true>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [/Users/chris/.nvm/versions/node/v6.9.1/bin/node]
17: v8::internal::BasicJsonStringifier::Result v8::internal::BasicJsonStringifier::Serialize_<false>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [/Users/chris/.nvm/versions/node/v6.9.1/bin/node]
18: v8::internal::Runtime_BasicJSONStringify(int, v8::internal::Object**, v8::internal::Isolate*) [/Users/chris/.nvm/versions/node/v6.9.1/bin/node]
19: 0x138f16c092a7
sh: line 1: 43838 Abort trap: 6           webpack --config ./webpack/build.webpack.config.js --bail --profile

npm ERR! Darwin 14.5.0
npm ERR! argv "/Users/chris/.nvm/versions/node/v6.9.1/bin/node" "/Users/chris/.nvm/versions/node/v6.9.1/bin/npm" "run" "build"
npm ERR! node v6.9.1
npm ERR! npm  v3.10.8
npm ERR! code ELIFECYCLE
npm ERR! frontend@0.209.0 build: `npm run clean; webpack --config ./webpack/build.webpack.config.js --bail --profile`
npm ERR! Exit status 134
npm ERR! 
npm ERR! Failed at the frontend@0.209.0 build script 'npm run clean; webpack --config ./webpack/build.webpack.config.js --bail --profile'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the frontend package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     npm run clean; webpack --config ./webpack/build.webpack.config.js --bail --profile
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs frontend
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!     npm owner ls frontend
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     /Users/chris/Projects/frontend/npm-debug.log

[chrisui]

On a second run with --max-old-space-size=8192 the build has completed with a 20% build time penalty.

Memory is quite limited on CI so going to play with how low we can get the memory until there's no crash for now.

[chrisui]

Probably for a separate issue but I actually see a slight increase (~1kb across vast majority of chunks) in our application using ModuleConcatenationPlugin after diving into the bundle analyser.

Wonder if there's an "off the top of head" reason why this may be?

[TheLarkInn]

If you can isolate and reproduce the bundle size increase we'd be happy to look at it. However overall, the huge improvement should be seen through runtime execution decrease.

[Furizaa]

I'm experiencing the same for a somewhat equal setup. Our build time goes up by 300% (I guess due to swap files) and the chunk size increases by 1kb for each split chunk.

[sokra]

On a second run with --max-old-space-size=8192 the build has completed with a 20% build time penalty.

v8 gets slow when near the memory limit (GC runs very often). A increased build time for the ModuleConcatenationPlugin is expected.

It's also expected that the ModuleConcatenationPlugin needs more memory.

Probably for a separate issue but I actually see a slight increase (~1kb across vast majority of chunks) in our application using ModuleConcatenationPlugin after diving into the bundle analyser.

That's maybe not wrong. Scope Hoisting doesn't decrease post-gzip size. pre-gzip size is usually a bit lower. pre-minimize size is ~ equal. You may get additional size savings while minimizing when the minimizer discovers some additional cross-module optimization, but that's rar. In this case post-gzip and pre-gzip size decrease.

In some cases post-gzip size is bigger, because function names are longer (more functions in a single scope but only limited number of letters in the alphabet).

[Manuelandro]

Think i'm experiencing the same without the error logs. After the first bundle, with ModuleConcatenationPlugin, it does not recognize any changes anymore, so i have to restart it.
It works like a charme without this plugin

[aseem2625]

For me it's bundling fine but the browser does't load the app giving error call of undefined

[christophehurpeau]

@aseem2625 then you should probably open another issue 😄

[ghostd]

@aseem2625 look at this html-webpack-plugin comment, it fixed my 'undefined' error (mainly back to webpack@2.6.1)

[emirdeliz]

+1

[emilio-martinez]

My two cents: The project I've been testing ModuleConcatenationPlugin in, I've mostly gotten this on situations where I have cache enabled, e.g. on watch. I simply disabled it when I'm on HMR mode, for example, and only kept it when producing a final bundle, which is where the optimizations matter to me for the most part.

[aseem2625]

@emilio-martinez As far as I rem., that's what is suggested by webpack's official medium post that ModuleConcatenationPlugin doesn't work with HMR.

[emilio-martinez]

@aseem2625 Yup, thanks, I realize that. I don't have it enabled with HMR, i.e., there are certain uses where I'm on "watch", no HMR, cache enabled for rebuilds, and I'm getting the error.

[tajo]

Ran into this issue as well. If you wonder how to give webpack more memory through that flag:

node --max-old-space-size=8192 node_modules/.bin/webpack

[JLHwung]

Could you reproduce this issue on 3.5.2? It seems that #5471 is possibly related to this issue.

[echenley]

@JLHwung Yes. I'm on 3.5.2 using --max-old-space-size=10240. The process consumes up to around 8gb of memory on a relatively large app (single output bundle is ~5mb, minified). Watching the process in activity monitor, it seems to be fine for a while, slowly growing to around 2gb, then towards the end it shoots up in increments as high as 1gb at a time, probably when the ModuleConcatenationPlugin kicks in. Setting devtool to false (as suggested in the other thread) does not have a noticeable effect on memory consumption.

Without ModuleConcatenationPlugin, the process maxes out around 2.5gb towards the end.

Basically this means that we need to upgrade our CI boxes from 4gb to 8 or even 16. That expense is difficult to justify for the marginal gains from this plugin.

Build speed isn't really an issue in a CI environment. I wonder if there could be a slower, disk-oriented version of the plugin that could run in the kind of low-memory environments common for CI?

[kzc]

ModuleConcatenationPlugin memory problem was isolated in #5992 (comment) and fixed in #5997.

[jjinux]

Is this card still relevant?