AsyncResponseTransformer.toBlockingStream() infinitely prefetches data.

[nikron]

Describe the bug

From reading the code and looking at behavior of when using the SDK for S3, I believe AsyncResponseTransformer.toBlockingStream() produces a stream that will store the all the data in memory if not consumed fast enough. This is a problem when downloading lots of data from S3 without being able to write it fast enough, and it causes out of memory issues. I believe there should be a limit on how much ByteBufferStoringSubscriber will accumulate.

Expected Behavior

When reading faster than consuming with an input stream, I don't expect the input stream to buffer the entire remaining amount.

Current Behavior

Input streams can buffer the full amount of data if not written fast enough.

Reproduction Steps

I haven't tried this, but if my theory is right:

1. Attempt to download a download a large object with S3 using blocking input stream
2. Don't consume the input stream.
3. The full data should reside in memory.

Possible Solution

ByteBufferStoringSubscriber or it's delegate should set a max number of bytes to store along with a minimum.

Additional Information/Context

It's possible that I'm getting out of memory errors from downloading too much data from S3 for a different reason, but this is what my debugging has lead me to.

AWS Java SDK version used

2.24.1

JDK version used

21

Operating System and version

Ubuntu 22.04

[nikron]

I might be wrong about this bug report because ByteBufferStoringSubscriber does try to limit to the buffer. However, if the a single event exceeds the 4MB buffer there can be more than 4MB stored.

[nikron]

I think this is more a problem that the crt client sets the initial read buffer high (80mb) by default, so lots of medium size files can quickly get put into memory. Closing this as my memory issues when away after setting that much lower. I think the default should probably much lower than 80mb.

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.