Change forEach to forEachLimit and purge cache to fix memory consumption issues #8609

dav-is · 2019-01-09T19:28:46Z

I did some memory profiling on a large app to find the issue. The app was crashing with memory usage of around 1.4GB.

The timelines are only from the "additional chunk assets processing (91%)" stage plus or minus some time. As a note, the profiling slows down the process considerably and I started the profiling manually close to the crash, so don't pay much attention to the lengths of time shown in the timelines.

Before applying this PR

Note that it allocates 550MB all at once (this is not the heap usage, only inside the profiling scope)

When applying the `forEachLimit`(15) change

I noticed the same amount of memory being allocated, except now it was adding 34MB of memory usage every frame. Note that each time it is storing _cachedSource

When setting `source._cachedSource = undefined` with `forEachLimit`

The heap memory usage went down drastically and stopped running out of memory. The _cachedSource should be removed after a file is emitted.

Performance Concerns

I ran time on two processes.

Before PR using max_old_space_size so node didn't run out of memory: 2m9s with 1.4GB peak memory usage
After PR using standard memory limts: 2m15s without running out of memory (other processes in the build use a lot of memory so the peak usage wouldn't be a useful metric here)

Accessing Private Variables

When I spoke to @sokra about this PR he had issue with the fact that I was using private variables and I can't depend on source being a CachedSource. This doesn't matter because I'm not accessing the variable, I'm just setting it to undefined. If the source isn't a CachedSource then it will have no effect and will not cause any errors.

The alternative of this is adding a method to CachedSource to purge the cache and checking that source is an instance of CachedSource and calling the method. There's no such thing as private variables in JavaScript and webpack doesn't use TypeScript, so I don't really see a reason to do this.

If anyone has another solution, I'd be open to suggestions.

The `forEachLimit` limit

I set the limit to 15 and it seems reasonable, but might need to be tweaked.

Thanks to @timneutkens for his help in diagnosing this issue with me.

jsf-clabot · 2019-01-09T19:28:49Z

Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

webpack-bot · 2019-01-09T19:28:51Z

For maintainers only:

This need to be documented (issue in webpack/webpack.js.org will be filed when merged)

sokra · 2019-01-09T19:49:33Z

You need to add the types in the declaration.d.ts file as we have no typings for neo-async

webpack-bot · 2019-01-09T20:04:26Z

Thank you for your pull request! The most important CI builds succeeded, we’ll review the pull request soon.

timneutkens · 2019-01-09T22:00:30Z

From testing so far it seems like this doesn't solve our issue.

webpack-bot · 2019-01-11T20:14:45Z

@dav-is Thanks for your update.

I labeled the Pull Request so reviewers will review it again.

@sokra Please review the new changes.

dav-is · 2019-01-16T04:32:15Z

@sokra I added more details to the initial post

sompylasar · 2019-01-16T06:35:21Z

lib/Compiler.js

@@ -340,6 +341,7 @@ class Compiler extends Tapable {
 						source.existsAt = targetPath;
 						source.emitted = true;
 						this.outputFileSystem.writeFile(targetPath, content, callback);
+						source._cachedSource = undefined;


writeFile is async and potentially may fail; does it change how _cachedSource should be cleared? (for ex., only on successful write, or only after the async operation is complete)

If writeFile fails, wouldn't the whole process fail?

That's an assumption this piece of code should not make.

Also this code currently seems to assume that writeFile uses _cachedSource synchronously which may not be the case depending on writeFile implementation.

Say a single file's cache wasn't cleared, it would have the memory footprint of a single file (not a big deal).

Say a single file's cache was prematurely pruged, webpack knows how to repopulate the cache so it wouldn't lose the source (it's only a cache)

What should be clear is that the cache is never purged and piles up to a GB+. Removing it right after it's written is the obvious end of life for a cache element.

What makes you think that writeFile could be asynchronous? The forEachLimit is the async part

The fact that a callback is passed to it. Is the writeFile call guaranteed to finish all its work synchronously despite receiving a callback?

What should be clear is that the cache is never purged and piles up to a GB+. Removing it right after it's written is the obvious end of life for a cache element.

Yes, I get that. That's why I'm trying to clarify when this "after it's written" event happens. Is it guaranteed to happen synchronously within the writeFile call? If so, the code is correct.

It's getting pretty late here (2am) I'll take another look at what you're saying in the morning 👍

It's getting pretty late here (2am) I'll take another look at what you're saying in the morning 👍

Yes, it's 12am on my end. Thanks for addressing the memory issue and paying attention to my comments!

@sompylasar I looked into it and it makes sense to purge the cache before writeFile and I tested it and the fix still works. Thanks for your feedback 👍

lib/Compiler.js

sokra · 2019-01-16T20:03:51Z

I can't easily take this change, as it would be a breaking change. While it probably improves memory/performance for you, it could affect other plugin negatively. Plugins could access the assets in the afterEmit/done hook. Purging the cache in emitAssets would make these plugins run the source generation again.

We need to put this behind an opt-in flag and enable it by default in the next major.

Instead of removing the _cachedSource it makes more sense (and is cleaner) to replace the source in assets with i. e. a SizeOnlySource (which only allows size() and throws on source()). This also has the benefit that the real Source can be GC'd and not only the cache of it. We should get rid of the existsAt and emitted hacks at the same time. I think a WeakMap<Source, { existsAt, sizeOnlySource }> plus a emittedAssets: Set<string> would work here. As removing existsAt and emitted is a breaking change, this should be moved to the next major.

dav-is · 2019-01-17T18:23:52Z

Closing in favor of #8642

@Meligy

This version (https://github.com/webpack/webpack/releases/tag/v4.29.0) includes a memory leak fix for assets (webpack/webpack#8609, webpack/webpack#8642). Thanks to @Meligy for pinging about this!

@Meligy

This version (https://github.com/webpack/webpack/releases/tag/v4.29.0) includes a memory leak fix for assets (webpack/webpack#8609, webpack/webpack#8642). Thanks to @Meligy for pinging about this!

@Meligy

This version (https://github.com/webpack/webpack/releases/tag/v4.29.0) includes a memory leak fix for assets (webpack/webpack#8609, webpack/webpack#8642). Thanks to @Meligy for pinging about this!

…timizations (#6114) We want our new memory optimizations (webpack/webpack#8609)

@Meligy

This version (https://github.com/webpack/webpack/releases/tag/v4.29.0) includes a memory leak fix for assets (webpack/webpack#8609, webpack/webpack#8642). Thanks to @Meligy for pinging about this!

forEach to forEachLimit

9cb4225

webpack-bot added the PR: unreviewed label Jan 9, 2019

webpack-bot added the PR: CI-not-ok label Jan 9, 2019

Use eachLimit instead even though forEachLimit apears to be an alias...

78abf04

Add ts declarations

80514cc

webpack-bot added PR: CI-ok and removed PR: CI-not-ok labels Jan 9, 2019

sokra approved these changes Jan 11, 2019

View reviewed changes

webpack-bot added PR: reviewed-approved and removed PR: unreviewed labels Jan 11, 2019

Remove cache after outputting

c26191d

webpack-bot added PR: review-outdated and removed PR: reviewed-approved labels Jan 11, 2019

dav-is changed the title ~~Change forEach to forEachLimit to fix memory consumption issues~~ Change forEach to forEachLimit and purge cache to fix memory consumption issues Jan 16, 2019

Retrigger build

f5ab496

sompylasar reviewed Jan 16, 2019

View reviewed changes

dav-is added 3 commits January 16, 2019 01:07

Check that _cachedSource is set

30d1010

Move cache purging before write

3ac9d6e

remove useless space

03331c9

sokra mentioned this pull request Jan 16, 2019

Change forEach to forEachLimit when emitting assets #8639

Merged

timneutkens mentioned this pull request Jan 17, 2019

How to reduce memory usage for next.js production build vercel/next.js#5575

Closed

guybedford mentioned this pull request Jan 17, 2019

Too high memory usage. vercel/ncc#165

Closed

dav-is closed this Jan 17, 2019

dav-is deleted the change-foreach-to-foreachlimit branch January 17, 2019 18:57

filipesilva mentioned this pull request Jan 22, 2019

feat(@angular-devkit/build-angular): update webpack to 4.29 angular/angular-cli#13494

Merged

dav-is mentioned this pull request Jan 24, 2019

Upgrade Webpack to 4.29.0 with FutureEmitAssets for Massive Memory Optimizations vercel/next.js#6114

Merged

timneutkens pushed a commit to vercel/next.js that referenced this pull request Jan 24, 2019

Upgrade Webpack to 4.29.0 with FutureEmitAssets for Massive Memory Op…

f28833c

…timizations (#6114) We want our new memory optimizations (webpack/webpack#8609)

shiftkey mentioned this pull request Mar 1, 2019

investigate Terser use for minification desktop/desktop#6652

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Change forEach to forEachLimit and purge cache to fix memory consumption issues #8609

Change forEach to forEachLimit and purge cache to fix memory consumption issues #8609

dav-is commented Jan 9, 2019 •

edited

jsf-clabot commented Jan 9, 2019 •

edited

webpack-bot commented Jan 9, 2019

sokra commented Jan 9, 2019

webpack-bot commented Jan 9, 2019

timneutkens commented Jan 9, 2019

webpack-bot commented Jan 11, 2019

dav-is commented Jan 16, 2019

sompylasar Jan 16, 2019

dav-is Jan 16, 2019

sompylasar Jan 16, 2019

sompylasar Jan 16, 2019

dav-is Jan 16, 2019

dav-is Jan 16, 2019

sompylasar Jan 16, 2019

dav-is Jan 16, 2019

sompylasar Jan 16, 2019

dav-is Jan 16, 2019

sokra commented Jan 16, 2019

dav-is commented Jan 17, 2019

Change forEach to forEachLimit and purge cache to fix memory consumption issues #8609

Change forEach to forEachLimit and purge cache to fix memory consumption issues #8609

Conversation

dav-is commented Jan 9, 2019 • edited

Before applying this PR

When applying the forEachLimit(15) change

When setting source._cachedSource = undefined with forEachLimit

Performance Concerns

Accessing Private Variables

The forEachLimit limit

jsf-clabot commented Jan 9, 2019 • edited

webpack-bot commented Jan 9, 2019

sokra commented Jan 9, 2019

webpack-bot commented Jan 9, 2019

timneutkens commented Jan 9, 2019

webpack-bot commented Jan 11, 2019

dav-is commented Jan 16, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sokra commented Jan 16, 2019

dav-is commented Jan 17, 2019

dav-is commented Jan 9, 2019 •

edited

When applying the `forEachLimit`(15) change

When setting `source._cachedSource = undefined` with `forEachLimit`

The `forEachLimit` limit

jsf-clabot commented Jan 9, 2019 •

edited