Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bypass copying the request body when it is too large #650

Open
vinaypunera opened this issue May 2, 2023 · 3 comments
Open

Bypass copying the request body when it is too large #650

vinaypunera opened this issue May 2, 2023 · 3 comments

Comments

@vinaypunera
Copy link

vinaypunera commented May 2, 2023

As of now the request body is by default read and copied to an in memory buffer. This behaviour increases the memory footprint in case we are uploading large data. Targeting use cases specific to passing large data in request body.
Ideally, in case of large data we want to rely on http1.1 chunking or http2 streaming, for that we need to bypass reading the data in memory for efficiency.

This copying happens at two places in request middlewares(in sequence) -

  1. https://github.com/go-resty/resty/blob/master/middleware.go#L445
  2. https://github.com/go-resty/resty/blob/master/middleware.go#L208

Possible solution -

  1. We can assume that for large data the request body will be an io.Reader and change code paths accordingly. We can introduce a flag notCopyRequestBody similar to notParseResponse

This issue is similar to - #495

@n0v3xx
Copy link

n0v3xx commented May 13, 2023

Think this is the same problem: #309

@vinaypunera
Copy link
Author

It is similar, more towards the request body. But the idea is to not buffer data in memory.

@kevin19930919
Copy link

Recently try to read chunk-based data from a 3-party API, but need to fall back to using the standard net/http
Looking forward to seeing resty support this feature.

@jeevatkm jeevatkm added this to the v3.0.0 Milestone milestone Oct 1, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

No branches or pull requests

4 participants