Support streaming requests (Transfer-Encoding: chunked)

[dshorthouse]

Apologies in advance for the inquiry, but am not sure where else (or how else) to pose my inquiry. I make use of a OAuth2::Client with Faraday.default_adapter and am trying to figure out how to do a PUT with a chunked transfer encoding to stream a file to an endpoint. I can use File.read to put a file in memory and then PUT it in the body of a request, but this defeats the purpose. Is it possible to use Faraday in conjunction with OAuth2::Client to do such a thing? Is there perhaps some Faraday middleware module comparable to Faraday::Multipart::FilePart that would/could construct a chunked body for a large file to PUT? Thanks for any pointers.

[olleolleolle]

Meta: Hi, in order to turn this from an Issue into a Help Discussion, I pushed a button and made it so.

[dshorthouse]

I think I've managed to sort this out and it really has nothing to do with Faraday. In essence, the solution here (I think) is multiple PUT requests with the appropriate headers for Content-Range, Content-Length and Digest within a block like this

chunk_size = 2*1024*1024
io = File.new(file_path, "r")
io.each_with_index(nil, chunk_size) do |chunk, index|
   # set the headers here & send a PUT request
end

In any case, it might perhaps be interesting to implement some sort of a module that would/could magically do this.

[iMacTia]

Hi @dshorthouse, thank you for raising this!
To be honest this is the first time I see someone need to implement a streaming request in Faraday, and as you've already realised we don't currently have native or even middleware support for that.
We do support streaming responses, but in my experience the very few APIs that can accept a streaming request normally offer a library that can do that for you (e.g. AWS S3).

I don't have enough experience with streaming requests to know how much work would be needed, especially on the adapters' side, but I'd be happy to review a design proposal if anyone would be interested to work on it 🙂

[HoneyryderChuck]

You're better off probably just using an http client which supports what you want.

For example, httpx supports streaming requests, either by using the :body parameter, or if you really need to, setting the "chunked" header.