Concurrency is limited to patterns, not files. (Error: EMFILE, too many open files)

[Tmunayyer]

Documentation Is:

• Missing
• Needed
• Confusing
• Not Sure?

Please Explain in Detail...

The concurrency option's description is as follows:

Limits the number of simultaneous requests to fs.

This was a bit misleading. I was running into errors of Error: EMFILE, too many open files while using this plugin even changing concurrency down to a value of 1.

The actual code in the plugin is as follow:

await Promise.all(this.patterns.map((item, index) => limit(async () => { ...

This means that if you pass an entire directory like node_modules, this plugin will go ham opening files which might cause an error if your process's open file limit is too low.

Maybe its a misuse of the plugin but it might be worth being more specific in the docs.

Your Proposal for Changes

I would specify this limits the number of concurrent patterns that will be running but the number of files opened might still be quite large if you pass a directory. Perhaps something like:

Limits the number of simultaneous patterns that will run concurrently. This has no affect on limiting the number of files opened for directories.

[alexander-akait]

For files you should use https://github.com/mrmlnc/fast-glob#common-options

[alexander-akait]

I was running into errors of Error: EMFILE, too many open files while using this plugin even changing concurrency down to a value of 1.

I think problem in other place, because if you have 16 cores (on thread per core), and you have 16 patterens with many files, maximum opened files will be 16 (from fast-glob) files * 16 patterns, we can't control count of files here, it is job of fast-glob

Also you don't provide a version of plugin, old version use node-glob (no protect from EMFILE).

Please provide steps to reproduce or as minim stack trace

[Tmunayyer]

Sorry I wasnt trying to report a bug but rather a clarification in the documentation.

The limit function only applied to how many patterns are used concurrently, not how many requests are sent to fs at once. So for example if I pass a directory like I was doing, the plugin was spinning up tons of fs calls to move all the files.

I think there should just be a clarification in the documentation that one should not pass a directory to this plugin and if they do, the plugin will work as fast as possible and the "limit" imposed by the concurrent option will have no affect on the calls to fs.

[Tmunayyer]

to get around this, I just moved the functionality to copy the folder out of webpack and used fs-extra package.

[alexander-akait]

PR welcome

[alexander-akait]

I still think problem no in plugin, because it is impossible to get Error: EMFILE, too many open files, note - you files will not minimizer/compressed/etc

[melenudo]

I've also got this error when a directory with a bunch of files is copied. It's difficult to reproduce (in my team slow windows systems got the problem).

If you analyze the code you are getting all entries from the glob and run a Promise.all on all those entries:

copy-webpack-plugin/src/index.js

Line 443 in 32bdafa

copiedResult = await Promise.all(

For each entry you are running a readFile:

copy-webpack-plugin/src/index.js

Line 585 in 32bdafa

data = await readFile(inputFileSystem, absoluteFilename);

In this scenario you could call readFile concurrently too many times reaching the file system limits (for example if readFile is slow because file system operations are slow).