Skip to content

Commit

Permalink
feature/s3/manager: Suppress WriterAt which can bypass Windows optimi…
Browse files Browse the repository at this point in the history
…zations. (#1791)

Fixes issue with Go 1.19 discovered in the S3 transfer manager's windows upload and downloader buffer pools. It looks like the update to Go 1.19 changes the code path that is used by the pool because io.NopCloser was updated to forward WriterTo if the underlying Writer implemented it. Looks likes we'd made some assumptions about this behavior not being present. This PR suppresses the WriterTo of the buffer pool to prevent the unexpected usage.
  • Loading branch information
skmcgrail committed Aug 5, 2022
1 parent 876fd54 commit 8d6ca65
Show file tree
Hide file tree
Showing 2 changed files with 16 additions and 2 deletions.
10 changes: 8 additions & 2 deletions feature/s3/manager/download.go
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,8 @@ func WithDownloaderClientOptions(opts ...func(*s3.Options)) func(*Downloader) {
// interface.
//
// Example:
// // Load AWS Config
//
// // Load AWS Config
// cfg, err := config.LoadDefaultConfig(context.TODO())
// if err != nil {
// panic(err)
Expand Down Expand Up @@ -153,6 +154,7 @@ func NewDownloader(c DownloadAPIClient, options ...func(*Downloader)) *Downloade
// and GC runs.
//
// Example:
//
// // pre-allocate in memory buffer, where headObject type is *s3.HeadObjectOutput
// buf := make([]byte, int(headObject.ContentLength))
// // wrap with aws.WriteAtBuffer
Expand Down Expand Up @@ -397,7 +399,11 @@ func (d *downloader) tryDownloadChunk(params *s3.GetObjectInput, w io.Writer) (i
}
d.setTotalBytes(resp) // Set total if not yet set.

n, err := io.Copy(w, resp.Body)
var src io.Reader = resp.Body
if d.cfg.BufferProvider != nil {
src = &suppressWriterAt{suppressed: src}
}
n, err := io.Copy(w, src)
resp.Body.Close()
if err != nil {
return n, &errReadingBody{err: err}
Expand Down
8 changes: 8 additions & 0 deletions feature/s3/manager/writer_read_from.go
Original file line number Diff line number Diff line change
Expand Up @@ -73,3 +73,11 @@ func (p *PooledBufferedReadFromProvider) GetReadFrom(writer io.Writer) (r Writer
}
return r, cleanup
}

type suppressWriterAt struct {
suppressed io.Reader
}

func (s *suppressWriterAt) Read(p []byte) (n int, err error) {
return s.suppressed.Read(p)
}

0 comments on commit 8d6ca65

Please sign in to comment.