DynamoDb Enhanced perf testing #1726

dagnir · 2020-03-19T16:38:11Z

Description

Tests overhead of GET and PUT item over the low level client, as well as
performance of those operations compared to the V1 mapper.

Motivation and Context

Testing

Screenshots (if appropriate)

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)

Checklist

License

I confirm that this pull request can be released under the Apache 2 license

dagnir · 2020-03-19T17:19:54Z

Benchmark                                       (testItem)   Mode  Cnt      Score    Error  Units
EnhancedClientPutOverheadBenchmark.enhPutHuge          N/A  thrpt    8   6178.310 ± 21.855  ops/s
EnhancedClientPutOverheadBenchmark.enhPutSmall         N/A  thrpt    8  21834.816 ± 62.910  ops/s
EnhancedClientPutOverheadBenchmark.enhPutTiny          N/A  thrpt    8  24995.407 ± 49.975  ops/s
EnhancedClientPutOverheadBenchmark.llPut              TINY  thrpt    8  25125.061 ± 71.506  ops/s
EnhancedClientPutOverheadBenchmark.llPut             SMALL  thrpt    8  23217.554 ± 89.430  ops/s
EnhancedClientPutOverheadBenchmark.llPut              HUGE  thrpt    8   8534.843 ± 25.766  ops/s

dagnir · 2020-03-19T20:58:29Z

Benchmark                                       (testItem)   Mode  Cnt      Score     Error  Units
EnhancedClientGetOverheadBenchmark.enhGet             TINY  thrpt    8  21985.693 ±  70.213  ops/s
EnhancedClientGetOverheadBenchmark.enhGet            SMALL  thrpt    8  18645.558 ± 129.404  ops/s
EnhancedClientGetOverheadBenchmark.enhGet             HUGE  thrpt    8   4751.360 ±  12.167  ops/s
EnhancedClientGetOverheadBenchmark.llGet              TINY  thrpt    8  23480.263 ±  79.170  ops/s
EnhancedClientGetOverheadBenchmark.llGet             SMALL  thrpt    8  21336.071 ±  63.168  ops/s
EnhancedClientGetOverheadBenchmark.llGet              HUGE  thrpt    8   6619.252 ±  20.321  ops/s
EnhancedClientPutOverheadBenchmark.enhPutHuge          N/A  thrpt    8   5905.032 ±  25.948  ops/s
EnhancedClientPutOverheadBenchmark.enhPutSmall         N/A  thrpt    8  20660.983 ±  63.780  ops/s
EnhancedClientPutOverheadBenchmark.enhPutTiny          N/A  thrpt    8  23994.051 ±  74.239  ops/s
EnhancedClientPutOverheadBenchmark.llPut              TINY  thrpt    8  24335.511 ±  88.384  ops/s
EnhancedClientPutOverheadBenchmark.llPut             SMALL  thrpt    8  21922.118 ±  31.407  ops/s
EnhancedClientPutOverheadBenchmark.llPut              HUGE  thrpt    8   8064.792 ±  22.985  ops/s

spfink · 2020-03-24T22:11:41Z

test/sdk-benchmarks/pom.xml

+        <dependency>
+            <groupId>software.amazon.awssdk</groupId>
+            <artifactId>dynamodb-enhanced</artifactId>
+            <version>${awsjavasdk.version}-PREVIEW</version>


Add a TODO here so we don't forget?

spfink · 2020-03-24T22:12:11Z

...a/software/amazon/awssdk/benchmark/enhanced/dynamodb/EnhancedClientGetOverheadBenchmark.java

+    private static final AwsJsonProtocolFactory JSON_PROTOCOL_FACTORY = AwsJsonProtocolFactory
+            .builder()
+            .clientConfiguration(SdkClientConfiguration.builder()
+                    .option(ENDPOINT, URI.create("https://dynamodb.amazonaws.com"))


What's this endpoint?

This was to stop the numerous profile loader related log messages

spfink · 2020-03-24T22:12:42Z

...a/software/amazon/awssdk/benchmark/enhanced/dynamodb/EnhancedClientGetOverheadBenchmark.java

+
+
+    @Benchmark
+    public Object llGet(TestState s) {


I think we can write out better method names here

spfink · 2020-03-24T22:13:15Z

...benchmarks/src/main/java/software/amazon/awssdk/benchmark/enhanced/dynamodb/ItemFactory.java

+        b.setListAttr(Arrays.asList(randomS(),
+                randomS(),
+                randomS(),
+


Why the seemingly random new lines? Also can we just for loop something here?

The lines break them up in to powers of 2. I can change to using a loop

spfink · 2020-03-24T22:15:09Z

test/sdk-benchmarks/pom.xml

+        <dependency>
+            <groupId>com.amazonaws</groupId>
+            <artifactId>aws-java-sdk-dynamodb</artifactId>
+            <version>1.11.748</version>


Do we really need v1 here long term? Once we have a good baseline for v2, can we just track v2?

We can remove it in the future, but while we're iterating on the performance we should have this in here.

zoewangg · 2020-03-24T22:40:14Z

buildspecs/benchmarks.yml

intended change?

just while I'm testing these changes in codebuild

zoewangg · 2020-03-24T22:41:39Z

test/sdk-benchmarks/src/main/java/software/amazon/awssdk/benchmark/BenchmarkRunner.java

@@ -74,6 +83,7 @@ public static void main(String... args) throws RunnerException, JsonProcessingEx
        benchmarksToRun.addAll(ASYNC_BENCHMARKS);
        benchmarksToRun.addAll(PROTOCOL_BENCHMARKS);
        benchmarksToRun.addAll(COLD_START_BENCHMARKS);
+        benchmarksToRun.addAll(MAPPER_BENCHMARKS);


Should we add ddb enhanced client data to baseline.json for regression detection?

See previous comment for getting baseline

Tests overhead of GET and PUT item over the low level client, as well as performance of those operations compared to the V1 mapper.

sonarqubecloud · 2020-03-26T22:55:59Z

SonarCloud Quality Gate failed.

0 Bugs
0 Vulnerabilities (and 4 Security Hotspots to review)
20 Code Smells

0.0% Coverage
5.9% Duplication

…36b8fe8a9 Pull request: release <- staging/ad96181a-3fdb-429c-89f6-a2836b8fe8a9

dagnir force-pushed the ddb-enh-perf-testing branch from 8c7102a to f02489a Compare March 19, 2020 16:41

dagnir force-pushed the ddb-enh-perf-testing branch from f02489a to 32fb142 Compare March 19, 2020 20:14

dagnir force-pushed the ddb-enh-perf-testing branch 2 times, most recently from 868da82 to c801b53 Compare March 24, 2020 17:24

dagnir changed the title ~~[Do Not Merge]: WIP DynamoDb Enhanced perf testing~~ DynamoDb Enhanced perf testing Mar 24, 2020

dagnir marked this pull request as ready for review March 24, 2020 17:27

dagnir force-pushed the ddb-enh-perf-testing branch 2 times, most recently from 4786064 to 77948bb Compare March 24, 2020 20:49

dagnir requested a review from spfink March 24, 2020 20:57

dagnir force-pushed the ddb-enh-perf-testing branch from 77948bb to 667b130 Compare March 24, 2020 22:05

spfink reviewed Mar 24, 2020

View reviewed changes

dagnir force-pushed the ddb-enh-perf-testing branch from 667b130 to edd33dc Compare March 24, 2020 22:18

zoewangg reviewed Mar 24, 2020

View reviewed changes

dagnir added 2 commits March 24, 2020 15:47

DynamoDb Enhanced perf testing

8113a10

Tests overhead of GET and PUT item over the low level client, as well as performance of those operations compared to the V1 mapper.

Review comments

967340f

dagnir force-pushed the ddb-enh-perf-testing branch from 695d7f0 to 967340f Compare March 24, 2020 22:47

Update baseline

f2f9c58

dagnir force-pushed the ddb-enh-perf-testing branch from 36f641b to f2f9c58 Compare March 26, 2020 22:34

Merge branch 'master' into ddb-enh-perf-testing

f362ac5

spfink approved these changes Mar 26, 2020

View reviewed changes

dagnir merged commit 5ea0d1d into aws:master Mar 26, 2020

dagnir deleted the ddb-enh-perf-testing branch March 26, 2020 23:11

aws-sdk-java-automation added a commit that referenced this pull request Sep 30, 2021

Merge pull request #1726 from aws/staging/ad96181a-3fdb-429c-89f6-a28…

9b101f2

…36b8fe8a9 Pull request: release <- staging/ad96181a-3fdb-429c-89f6-a2836b8fe8a9

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DynamoDb Enhanced perf testing #1726

DynamoDb Enhanced perf testing #1726

dagnir commented Mar 19, 2020 •

edited

Loading

dagnir commented Mar 19, 2020

dagnir commented Mar 19, 2020

spfink Mar 24, 2020

dagnir Mar 24, 2020

spfink Mar 24, 2020

dagnir Mar 24, 2020

spfink Mar 24, 2020

spfink Mar 24, 2020

dagnir Mar 24, 2020

spfink Mar 24, 2020

dagnir Mar 24, 2020

zoewangg Mar 24, 2020

dagnir Mar 24, 2020

zoewangg Mar 24, 2020

dagnir Mar 24, 2020

sonarqubecloud bot commented Mar 26, 2020

DynamoDb Enhanced perf testing #1726

DynamoDb Enhanced perf testing #1726

Conversation

dagnir commented Mar 19, 2020 • edited Loading

Description

Motivation and Context

Testing

Screenshots (if appropriate)

Types of changes

Checklist

License

dagnir commented Mar 19, 2020

dagnir commented Mar 19, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sonarqubecloud bot commented Mar 26, 2020

dagnir commented Mar 19, 2020 •

edited

Loading