uploader.cache! loads entire contents of file into memory

[vergenzt]

I've been encountering this in production with 1-2 GB uploads and it's causing background processors to crash. :( (They cache! the file first from AWS in order to do local processing.)

The load happens at this line:

def sanitized_file
  _content = file.read

  # ...

end

I've been trying to look around and see if I can figure out how to fix it, but I've gotten a bit lost in the codebase and am not sure where this needs to be fixed.

Why does the whole file get read into the sanitized file? What's the purpose of the sanitized file here in the first place?

[vergenzt]

Related to:

• GIF Upload Memory Leak breaking entire site (App Crashed Error on Heroku)  #2016
• large file is going to blow memory #1780
• Streaming of SanitizedFile. #1529
• Read streams in chunks. #1523
• DOS vector #795
• Add support for fog streaming uploads #467
• Large Files #413

[thauburger]

+1

[oponce1611]

+1

[a-barbieri]

+1

[thucnguyen]

+1