S3 CRT client is almost an order of magnitude slower than the Netty client

[pwinckles]

Describe the bug

While converting a project to use the new TransferManager, and thereby pulling in the CRT client, I noticed a substantial hit to performance. A test suite that used to run in 5 minutes, now takes 30 minutes. I have only tested the get object api in isolation while looking for the source of the slow down, but it is significantly slower than the Netty client the point that it's unusable.

Expected Behavior

I expected the CRT client to be as fast or faster than the Netty client.

Current Behavior

It's really slow.

Reproduction Steps

The following test that demonstrates the problem:

import org.HdrHistogram.Histogram;
import software.amazon.awssdk.core.async.AsyncRequestBody;
import software.amazon.awssdk.core.async.AsyncResponseTransformer;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3AsyncClient;
import software.amazon.awssdk.services.s3.model.GetObjectRequest;

import java.io.PrintStream;
import java.util.stream.IntStream;

public class ReadTest {

    private static final String BUCKET = "YOUR_BUCKET_HERE";
    private static final String KEY = "test-small";
    private static final int ITERATIONS = 50;

    public static void main(String[] args) throws Exception {
        try (var s3 = S3AsyncClient.crtBuilder().region(Region.US_EAST_2).build()) {
            s3.putObject(b -> b.bucket(BUCKET).key(KEY).build(), AsyncRequestBody.fromString("testing\n"))
                    .get();
            runTest(s3);
        }
    }

    private static void runTest(S3AsyncClient s3) {
        var histogram = new Histogram(3);
        IntStream.range(0, ITERATIONS).forEach(i -> {
            var start = System.nanoTime();
            s3.getObject(
                            GetObjectRequest.builder()
                                    .bucket(BUCKET)
                                    .key(KEY)
                                    .build(),
                            AsyncResponseTransformer.toBytes())
                    .join();
            var stop = System.nanoTime();
            histogram.recordValue(stop - start);
        });
        histogram.outputPercentileDistribution(new PrintStream(System.out), 1_000_000.0);
    }

}

Histogram provided by HdrHistogram. Output unit is milliseconds.

The above test is for the CRT client, simply change the builder to S3AsyncClient.builder() to test the Netty client.

Possible Solution

No response

Additional Information/Context

Here are the results I saw on my system when I ran the above test (unit is milliseconds):

CRT

       Value     Percentile TotalCount 1/(1-Percentile)

     499.122 0.000000000000          2           1.00
     618.136 0.100000000000          6           1.11
     648.544 0.200000000000         10           1.25
     729.809 0.300000000000         15           1.43
     830.996 0.400000000000         20           1.67
     919.077 0.500000000000         25           2.00
     932.708 0.550000000000         28           2.22
     951.058 0.600000000000         30           2.50
    1020.264 0.650000000000         33           2.86
    1022.886 0.700000000000         35           3.33
    1125.122 0.750000000000         38           4.00
    1127.219 0.775000000000         39           4.44
    1129.316 0.800000000000         40           5.00
    1225.785 0.825000000000         42           5.71
    1227.882 0.850000000000         44           6.67
    1227.882 0.875000000000         44           8.00
    1228.931 0.887500000000         45           8.89
    1228.931 0.900000000000         45          10.00
    1248.854 0.912500000000         46          11.43
    1272.971 0.925000000000         47          13.33
    1272.971 0.937500000000         47          16.00
    1349.517 0.943750000000         48          17.78
    1349.517 0.950000000000         48          20.00
    1349.517 0.956250000000         48          22.86
    1432.355 0.962500000000         49          26.67
    1432.355 0.968750000000         49          32.00
    1432.355 0.971875000000         49          35.56
    1432.355 0.975000000000         49          40.00
    1432.355 0.978125000000         49          45.71
    1703.936 0.981250000000         50          53.33
    1703.936 1.000000000000         50
#[Mean    =      924.938, StdDeviation   =      256.060]
#[Max     =     1703.936, Total count    =           50]
#[Buckets =           21, SubBuckets     =         2048]

Netty

       Value     Percentile TotalCount 1/(1-Percentile)

      59.736 0.000000000000          1           1.00
      99.680 0.100000000000          5           1.11
     100.467 0.200000000000         11           1.25
     100.925 0.300000000000         15           1.43
     101.319 0.400000000000         20           1.67
     101.908 0.500000000000         25           2.00
     103.023 0.550000000000         28           2.22
     131.269 0.600000000000         30           2.50
     159.252 0.650000000000         33           2.86
     203.031 0.700000000000         35           3.33
     302.776 0.750000000000         38           4.00
     303.301 0.775000000000         39           4.44
     304.611 0.800000000000         40           5.00
     305.660 0.825000000000         44           5.71
     305.660 0.850000000000         44           6.67
     305.660 0.875000000000         44           8.00
     306.446 0.887500000000         47           8.89
     306.446 0.900000000000         47          10.00
     306.446 0.912500000000         47          11.43
     306.446 0.925000000000         47          13.33
     306.446 0.937500000000         47          16.00
     306.708 0.943750000000         48          17.78
     306.708 0.950000000000         48          20.00
     306.708 0.956250000000         48          22.86
     307.233 0.962500000000         49          26.67
     307.233 0.968750000000         49          32.00
     307.233 0.971875000000         49          35.56
     307.233 0.975000000000         49          40.00
     307.233 0.978125000000         49          45.71
    1343.226 0.981250000000         50          53.33
    1343.226 1.000000000000         50
#[Mean    =      186.467, StdDeviation   =      187.214]
#[Max     =     1343.226, Total count    =           50]
#[Buckets =           21, SubBuckets     =         2048]

AWS Java SDK version used

2.25.27 (CRT 0.29.14)

JDK version used

21.0.2

Operating System and version

Linux Fedora 39

[zoewangg]

Hi @pwinckles, thanks for reaching out. The AWS CRT-based S3 client provides enhanced performance for transferring objects of size 8MB+ by utilizing multipart upload and ranged GET to achieve parallel transfers. There is some overhead involved (extra API calls such as HeadObject to get the size), so if you are transferring smaller objects, the overhead may not be paid off, and thus it may be slower.

In your case, if you want to improve performance, I'd suggest using the standard Java S3 client with CRT HTTP client. https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/http-configuration-crt.html

[pwinckles]

@zoewangg thanks for the response.

It sounds like you're saying that CRT client is not a good general purpose client and I should use two clients -- CRT for the TransferManager and Netty for everything else. Is that accurate?

[zoewangg]

AWS CRT-based S3 client is preferred in the following scenarios:

• if your application transfers large objects (> 8MB) and you want maximized performance
• if you want to upload objects with unknown content length

Side note: we are working on improving our dev guide to compare different flavors of S3 clients so that users know how to choose them based on their use-case :)

[pwinckles]

Thanks!

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.