[bug] EMFILE: too many open files

[konradpabjan]

What happened?

Stumbled on a failure that happened with v4. Apparently this doesn't happen with v3

Example run can be found here: https://github.com/spacetelescope/jdaviz/actions/runs/7266578163/job/19798598137

What did you expect to happen?

Successful artifact upload

How can we reproduce it?

Not sure, but I think the key here is a large number of files. In the logs it shows 8824 files which is quite a bit. Maybe some read stream isn't being closed somewhere 🤔

Anything else we need to know?

Nope

What version of the action are you using?

v4

What are your runner environments?

windows

Are you on GitHub Enterprise Server? If so, what version?

No

[zaikunzhang]

Also seen in my workflow.

Run actions/upload-artifact@v4.0.0
With the provided path, there will be [7](https://github.com/primalib/prima/actions/runs/7338125515/job/19983281711#step:4:8)3166 files uploaded
Artifact name is valid!
Root directory input is valid!
Beginning upload of artifact content to blob storage
node:events:492
      throw er; // Unhandled 'error' event
      ^

Error: EMFILE: too many open files, open '/home/runner/work/prima/prima/prima_prima_quadruple.1_20.ubln.single.231227_1211_start/HS63'
Emitted 'error' event on ReadStream instance at:
    at emitErrorNT (node:internal/streams/destroy:151:[8](https://github.com/primalib/prima/actions/runs/7338125515/job/19983281711#step:4:9))
    at emitErrorCloseNT (node:internal/streams/destroy:[11](https://github.com/primalib/prima/actions/runs/7338125515/job/19983281711#step:4:12)6:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
  errno: -24,
  code: 'EMFILE',
  syscall: 'open',
  path: '/home/runner/work/prima/prima/prima_prima_quadruple.1_20.ubln.single.23[12](https://github.com/primalib/prima/actions/runs/7338125515/job/19983281711#step:4:13)27_1211_start/HS63'
}

Node.js v[20](https://github.com/primalib/prima/actions/runs/7338125515/job/19983281711#step:4:21).8.1

[robherley]

We can possibly fix with the file method in archiver:

• https://www.archiverjs.com/docs/archiver#file

Appends a file given its filepath using a lazystream wrapper to prevent issues with open file limits.

Right now we are just using the append operation and creating a read stream per file:

• https://github.com/actions/toolkit/blob/5430c5d84832076372990c7c27f900878ff66dc9/packages/artifact/src/internal/upload/zip.ts#L47-L49

[SMoraisAnsys]

Bumping as I'm facing the same issue on a repo :)

[sungaila]

Same issue here when I tried to upload 9,834 files for a single artifact. The compression-level made no difference.

Had to downgrade to upload-artifact@v3 and download-artifact@v3.

[qwerttvv]

actions/upload-artifact@main
  with:
    name: MSYS
    path: C:\MSYS
    if-no-files-found: warn
    compression-level: 6
    overwrite: false
  env:
    MSYSTEM: MINGW64
With the provided path, there will be 51574 files uploaded
Artifact name is valid!
Root directory input is valid!
Beginning upload of artifact content to blob storage
node:events:492
      throw er; // Unhandled 'error' event
      ^

Error: EMFILE: too many open files, open 'C:\MSYS\msys64\mingw64\lib\python3.11\distutils\tests\__pycache__\test_cygwinccompiler.cpython-311.pyc'
Emitted 'error' event on ReadStream instance at:
    at emitErrorNT (node:internal/streams/destroy:151:8)
    at emitErrorCloseNT (node:internal/streams/destroy:116:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
  errno: -4066,
  code: 'EMFILE',
  syscall: 'open',
  path: 'C:\\MSYS\\msys64\\mingw64\\lib\\python3.11\\distutils\\tests\\__pycache__\\test_cygwinccompiler.cpython-311.pyc'
}

Node.js v20.8.1

same bug at latest https://github.com/actions/upload-artifact/tree/3a8048248f2f288c271830f8ecf2a1c5d8eb0e9a

actions/upload-artifact@v3 is good but warning

Warning: There are over 10,000 files in this artifact, consider creating an archive before upload to improve the upload performance.

[zaikunzhang]

Is this fixed yet?

[ehuelsmann]

Running into this with Linux too: https://github.com/ehuelsmann/LedgerSMB/actions/runs/8121634001/job/22200227067#step:6:26

[pcfreak30]

Ditto, just ran into the same problem :/

[rmunn]

I've switched actions/toolkit#1723 (proposed fix for retaining file permissions in uploaded .zip files) to use zip.file(file.sourcePath) instead of zip.append(createReadStream(file.sourcePath)) so that that PR can also fix this bug at the same time. It needs a review from repo maintainers before it can go anywhere, though.

[marcodali]

Is there a command that we can run in the docker container that allow us to increment the number of open file descriptors to 10k lets say for example??

Wouldn't that fix this issue?