Retry streaming S3 object downloads using range #2326
Labels
feature-request
A feature should be added or improved.
help wanted
We are asking the community to submit a PR to resolve this issue.
Is your feature request related to a problem? Please describe.
Copying from #1617 :
When downloading S3 objects with a block, aws-sdk-s3 internally creates a BlockIO to be the writable response target. Normally aws-sdk-s3 retries any failed downloads in case of network errors, but not for streaming downloads. This is because BlockIO is not truncatable, unlike a response target like StringIO or Tempfile, and aws-sdk-s3 needs it to be because it retries the whole download from the start.
Describe the solution you'd like
s3 get_object supports the range parameter. On retry of streaming bodies for the get_object request, we can use the number of received so far to add a range parameter, thus avoiding re-downloading data that has already been sent.
The text was updated successfully, but these errors were encountered: