Streaming requests: Write a file directly to disk

[ellcs]

I want to upload a file and stream the body directly to disk, without buffering the body in a tempfile first. I tried multiple variations and could not figure out how to do it with rack < 3.0. I read within Github discussions that a #rewind method has been dropped since rack 3.0. I thought with the dropped #rewind the body/file would not be buffered before anymore.

I could not find anything in the documentation and not within Github discussions. My questions are the following:

• Is it even possible to stream the request? Does it work with rack < 3.0 or is rack 3.0 mandatory?
• If it works, could you please provide a minimal example?
• Do i have to use a certain web server (puma, thin, unicorn, ...)?

My current assumption, please correct me if I am wrong: Only after Rack version 3.0, streaming a file directly to disk is possible, because it will be always written to /tmp. It will be written to /tmp even before call(env) will occur. The web server must be compatible to Rack 3.0.

Thank you in advance!

[ioquatix]

The input is not buffered on Falcon, so env['rack.input'].each{|chunk| ...} will stream from the incoming request.

However, if you use multipart, this won't work as you imagine, since the multiple parts need to be buffered - this part is Rack.

[ellcs]

Thank you! Based on your commend i wrote following App:

require "rack"
require 'securerandom'
require 'fileutils'

class App
  def call(env)
    puts "starting upload"
    body_stream(env) do |body|
      FileUtils.mkdir_p("/ondisk/upload")
      File.open("/ondisk/upload/#{SecureRandom.uuid}", "wb") do |file|
        IO.copy_stream(body, file)
      end
    end
    puts "done uploading"
    [200, {}, ["uploaded"]]
  end

  def body_stream(env, &block)
    stream = env['rack.input']
    block.call(stream)
  ensure
    stream.close
  end
end

I had to run it with falcon -p 9292 and made sure rack > 3.0 was installed. The config.ru required the app and simply used run App.new. I allocated a bigfile which was bigger than the RAM i had and could trigger a request: curl -v https://localhost:9292 --insecure -d @bigfile https://localhost:9292. I could upload multiple files bigger than the RAM i had at the same time.

[ioquatix]

That's awesome, well done. In theory, the same could be true for Puma, but that's just not how it's designed, IIUC, to avoid slow clients hogging worker threads.

[md-work]

With Apache+Passenger you can avoid this. Even with older software versions like ruby-2.5 and rack-2.0.8 on openSUSE-15.4.

 

Apache config:

<VirtualHost *:80>
	DocumentRoot /srv/www/htdocs/public
	
	# Allow requests up to 4 GiB.
	LimitRequestBody 4294967296
	
	# "on" writes the body to: ${PassengerDataBufferDir}/buffer.XXXXXXXXXX
        # On openSUSE-15.4 PassengerDataBufferDir is: /run
	#
	# That file will be invisible, because it's deleted and only the
	# process "Passenger core" still has an open file descriptor
	# for it.
	# ls -lAhs /proc/$(pidof -s 'Passenger core')"/fd/
	PassengerBufferUpload off
</VirtualHost>

 

Minimal /srv/www/htdocs/config.ru:

app = Proc.new do |env|
  request = Rack::Request.new(env)
  
  # Will read all data to /tmp/PassengerTeeInput-* if
  # "PassengerBufferUpload off" is set in Apache config.
  # This is needed to make the data rewind'able. (method "rewind")
  #
  # This will NOT respect PassengerDataBufferDir. Instead it uses this:
  # https://rubygems.org/gems/tmpdir
  #   require 'tmpdir'; Dir::tmpdir
  #
  # That file will be invisible, because it's deleted and only the
  # process "Passenger RubyApp: /var/www/html (production)" still
  # has an open file descriptor for it.
  # ls -lAhs /proc/$(pidof -s 'Passenger RubyApp: /var/www/html (production)')"/fd/
  #body_data = request.body
  
  # Access underlying socket and read data directly without rewind.
  body_data = request.body.instance_variable_get('@socket')
  
  size = 0
  while !body_data.eof? do
    size += body_data.read(1024).size
  end
  [200, {'Content-Type' => 'text/plain'}, 'body size was: ' + size.to_s]
end

run(app)

[ioquatix]

It would be better if rack.input was streaming in passenger.