Skip to content

Efficient Go Reader for large AWS S3 Objects

License

Notifications You must be signed in to change notification settings

aiopsclub/aws-s3-reader

 
 

Repository files navigation

AWS S3 Reader

codecov Go Report Card Go Reference OpenSSF Scorecard

Efficient reader for large S3 files.

  • Seek() via Byte-Range HTTP offsets12
  • zero-memory copy
  • early HTTP Body termination
s3client := s3.New(session.Must(session.NewSession(
    aws.NewConfig().WithRegion("ap-southeast-1"),
)))

r := awss3reader.NewS3ReadSeeker(
    s3client,
    "nikolaydubina-blog-public",
    "videos/2024-02-22.mov",
    awss3reader.FixedChunkSizePolicy{Size: 1 << 20 * 40},
)
defer r.Close()

r.Seek(100, io.SeekCurrent)

res, err := io.ReadAll(r)

Related Work

Footnotes

  1. https://docs.aws.amazon.com/whitepapers/latest/s3-optimizing-performance-best-practices/use-byte-range-fetches.html

  2. https://developer.mozilla.org/en-US/docs/Web/HTTP/Range_requests

About

Efficient Go Reader for large AWS S3 Objects

Resources

License

Security policy

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Go 100.0%