file write timing issue

[szilvasics]

I found an error in the "file-flow":

MW config:
app.use(
fileUpload({
useTempFiles: true,
tempFileDir: "/tmp/",
debug: true,
safeFileNames: true
})
);

The file seems OK when I check with:
if (Object.keys(req.files).length == 0 || !req.files.file) {
return console.log("No file uploaded.");
}

but sometimes, I get an error when I try to move the file to a new path
{ [Error: ENOENT: no such file or directory, open '/tmp/tmp1575948653152']
errno: -2,
code: 'ENOENT',
syscall: 'open',
path: '/tmp/tmp1575948653152' }

I use this code to move:
req.files.file.mv(/data/${prefix}_${req.files.file.name}, err => {
if (err) {
return console.log("Error in file moving", err);
}
...
});

I checked the file and exists in /tmp

The problem was fixed, when I changed my code to wait a second (with setTimeout) before move the file.

[RomanBurunkov]

Hi,

That is interesting, can u share middleware debug logging for the issue, when u do that without timeout?

Also could u clarify whether /data and /tmp located on the same fs and what type of hardware u are using when getting this issue.

[szilvasics]

Hi,

I'm using Docker on Ubuntu 18.04 host.
The container come from node:lts-alpine
/data folder is a local folder on host, disk formatted to ext4 and folder attached to container fs
/tmp is created by container default

before timeout fix, the debug log was something like that
Uploaded 12537 bytes for file customersync_19_1210_0612_25.xml
Error in file moving { [Error: ENOENT: no such file or directory, open '/tmp/tmp1575954947640']
errno: -2,
code: 'ENOENT',
syscall: 'open',
path: '/tmp/tmp1575954947640' }

after timeout fix, I saw uploaded file saved by multiple chunks sometimes
Uploaded 64946 bytes for file productsync_19_1210_1640_41.xml
Uploaded 1180 bytes for file productsync_19_1210_1640_41.xml
Processing file: /data/20191210-16442350_productsync_19_1210_1640_41xml
The real file size is in the folder 66126 bytes.

Is it possible these files was wrong before timeout?
It may happen, express middleware callback start immediately after first chunk was saved?

[RomanBurunkov]

The output you showed up looks strange for me, what version of middleware are you using?

Answering your question about callback immediate start.
I can't see any ways that can happens without any errors during upload...

[szilvasics]

I'm using
express-fileupload@1.1.3-alpha.1
express@4.16.4

[RomanBurunkov]

There were several changes in file handling since 1.1.3.
Try the latest 1.1.6 and post debug log if issue still appears with1.1.6 version

[RomanBurunkov]

Hi @szilvasics , any updates about this issue?

[caritasverein]

We encountered a similar issue in our application and tracked it down to the fact, that the middleware does not wait for the tempfile to flush / finish.
As the upload-files stream ends (https://github.com/richardgirges/express-fileupload/blob/master/lib/processMultipart.js#L80), the output-files stream is ended as well through its complete function (https://github.com/richardgirges/express-fileupload/blob/master/lib/tempFileHandler.js#L33).
Now the middleware should wait for both the upload- and the output-file to fire the finish-event, but the upload-files finish-handler calls next() immediately (https://github.com/richardgirges/express-fileupload/blob/master/lib/processMultipart.js#L112).
Depending on how long it takes for the output-file to (flush and then) finish, other code that might rely on the files existence might be running too soon.

[RomanBurunkov]

Hi @caritasverein ,

Thanks for your investigation.

Could you also clarify which version of middleware you use and if it's the latest one share the output when debug option set to true.

[somewind]

i meet the same bug with Docker Centos.

version 1.1.5

[pdumais]

I just had that same problem. I was moving the file using process.exec("mv ...") (because mv() was broken in my old release). The file would get moved but it was 0 bytes. If I add a sleep before moving, everything is fine. and I've seen that repeatedly but only on one system.

I think the file should get flushed before calling next(). Or maybe do a validation of the file size/existance before proceeding.

[somewind]

I just had that same problem. I was moving the file using process.exec("mv ...") (because mv() was broken in my old release). The file would get moved but it was 0 bytes. If I add a sleep before moving, everything is fine. and I've seen that repeatedly but only on one system.

I think the file should get flushed before calling next(). Or maybe do a validation of the file size/existance before proceeding.

I released a package called express-fileupload-temp-file to temporarily fix it.

[pdumais]

@RomanBurunkov are you ok with @somewind's PR? Or were you thinking about another solution?

[RomanBurunkov]

@pdumais

I have some doubts about it, but still haven't had a chance to look into because of having no free time =(

[somewind]

@pdumais

I have some doubts about it, but still haven't had a chance to look into because of having no free time =(

The previous reply did not express clearly, I have modified it as you said, please check the PR again 😃

[Shahmeerm]

@somewind

We recently hit the same error when API benchmarking start for our project. Almost 13% of the times file uploads fail with the error Error: ENOENT: no such file or directory, open '/tmp/tmp-4-1585274797122'

We see that your commit has been merged. We are on 1.1.6.

NPM Registry suggests that 1.1.6 updated 4 months ago. How can we access your fix ? Please guide.

[Shahmeerm]

@RomanBurunkov Can you please guide ? I have placed a comment 6 Days ago.

[larsandersson-ifpi]

Having same problem.
anyone that can suggest a replacement for express-fileupload?

[RomanBurunkov]

Hi there,

Going to publish into npm 1.1.7-alpha.1 with @somewind fix for this issue.