Skip to content

Commit

Permalink
[dynamodb] AWS SDK updated to 2.17.102 (openhab#11914)
Browse files Browse the repository at this point in the history 
* [dynamodb] Update guide for updating the SDK

Some steps were obsolete and required corrections.

Signed-off-by: Sami Salonen <ssalonen@gmail.com>

* [dynamodb] Update AWS SDK to 2.17.102

- Added NOTICE with all compiled dependencies. Dependencies listed in
  feature.xml are omitted.
- jackson dependency is now provided by AWS SDK as a 'shaded' copy
- log4j version used in tests updated
- Specifying DefaultsMode=STANDARD to make the SDK utilize latest
  recommended parameters. RetryMode is also controlled by DefaultsMode,
  so we avoid setting it explicitly unless overridden by the user.
- Fixed test regressions due to SDK changes

Signed-off-by: Sami Salonen <ssalonen@gmail.com>
Signed-off-by: Nick Waterton <n.waterton@outlook.com>
  • Loading branch information
@ssalonen @NickWaterton
ssalonen authored and NickWaterton committed Apr 27, 2022
1 parent e7b07f6 commit 01f1a9c
Show file tree
Hide file tree
Showing 9 changed files with 112 additions and 106 deletions.
31 changes: 31 additions & 0 deletions bundles/org.openhab.persistence.dynamodb/NOTICE
View file
Expand Up @@ -11,3 +11,34 @@ https://www.eclipse.org/legal/epl-2.0/.
== Source Code

https://github.com/openhab/openhab-addons

== Third-party Content

com.typesafe.netty: netty-reactive-streams-http
* License: Apache 2.0 License
* Project: https://github.com/playframework/netty-reactive-streams
* Source: https://github.com/playframework/netty-reactive-streams

com.typesafe.netty: netty-reactive-streams
* License: Apache 2.0 License
* Project: https://github.com/playframework/netty-reactive-streams
* Source: https://github.com/playframework/netty-reactive-streams

org.reactivestreams: reactive-streams
* License: Public Domain (CC0)
* Project: http://www.reactive-streams.org/
* Source: https://github.com/reactive-streams/reactive-streams-jvm/tree/v1.0.3

software.amazon.awssdk: annotations, auth, aws-core, aws-json-protocol,
dynamodb-enhanced, dynamodb, http-client-spi,
json-utils, metrics-spi, netty-nio-client, profiles,
protocol-core, regions, sdk-core,
third-party-jackson-core, utils
* License: Apache 2.0 License
* Project: https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/home.html
* Source: https://github.com/aws/aws-sdk-java-v2/

software.amazon.eventstream: eventstream
* License: Apache 2.0 License
* Project: https://github.com/awslabs/aws-eventstream-java/
* Source: https://github.com/awslabs/aws-eventstream-java/
10 changes: 5 additions & 5 deletions bundles/org.openhab.persistence.dynamodb/README.md
View file
Expand Up @@ -95,7 +95,7 @@ Modify the policy accordingly if needed.
<!-- markdownlint-disable ol-prefix -->
4. Click _Next: Tags_
5. Click _Next: Review_
6. Enter `openhab-dynamodb-policy` as the _Name_ol-prefix -->
6. Enter `openhab-dynamodb-policy` as the _Name_
7. Click _Create policy_ to finish policy creation
<!-- markdownlint-enable ol-prefix -->

Expand Down Expand Up @@ -224,10 +224,10 @@ Similar caveat applies for DynamoDB Time to Live (TTL) setting `expireDays`.

### Updating Amazon SDK

1. Clean `lib/*`
2. Update SDK version in `scripts/fetch_sdk_pom.xml`. You can use the [maven online repository browser](https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-dynamodb) to find the latest version available online.
3. `scripts/fetch_sdk.sh`
4. Copy printed dependencies to `pom.xml`
1. Update SDK version and `netty-nio-client` version in `scripts/fetch_sdk_pom.xml`. You can use the [maven online repository browser](https://mvnrepository.com/artifact/software.amazon.awssdk/dynamodb-enhanced) to find the latest version available online.
2. `scripts/fetch_sdk.sh`
3. Copy printed dependencies to `pom.xml`. If necessary, adjust feature.xml, bnd.importpackage and dep.noembedding as well (probably rarely needed but [it happens](https://aws.amazon.com/blogs/developer/the-aws-sdk-for-java-2-17-removes-its-external-dependency-on-jackson/)).
4. Check & update `NOTICE` file with all the updated, new and removed dependencies.

After these changes, it's good practice to run integration tests (against live AWS DynamoDB) in `org.openhab.persistence.dynamodb.test` bundle.
See README.md in the test bundle for more information how to execute the tests.
Expand Down
69 changes: 32 additions & 37 deletions bundles/org.openhab.persistence.dynamodb/pom.xml
View file
Expand Up @@ -19,10 +19,10 @@
<dependencies> -->
<bnd.importpackage>!com.amazonaws.*,!com.sun.org.apache.xpath.*,!kotlin,!org.apache.log.*,!org.bouncycastle.*,!org.joda.convert.*,!scala.util.*,!software.amazon.*,!org.reactivestreams,!com.typesafe.netty</bnd.importpackage>
<!-- We do not want to embed/compile in dependencies that are declared as OSGi imports (feature.xml). This includes e.g.
netty & jackson. Let's ensure by listing relevant packages with dep.noembedding -->
<dep.noembedding>netty-common,netty-transport,netty-transport-native-epoll,netty-transport-native-unix-common,netty-buffer,netty-resolver,netty-codec,netty-codec-http,netty-codec-http2,netty-handler,jackson-core,jackson-annotations,jackson-dataformat-cbor,jackson-databind</dep.noembedding>
<!-- netty version matching the openhab.tp-netty feature version -->
<slf4j.version>1.7.21</slf4j.version>
netty. Let's ensure by listing relevant packages with dep.noembedding -->
<dep.noembedding>netty-common,netty-transport,netty-transport-native-epoll,netty-transport-native-unix-common,netty-buffer,netty-resolver,netty-codec,netty-codec-http,netty-codec-http2,netty-handler</dep.noembedding>
<!-- slf4j version matching the version specified in openhab-core/pom.xml -->
<slf4j.version>1.7.32</slf4j.version>
</properties>

<!--Custom repository for DynamoDBLocal -->
Expand Down Expand Up @@ -61,7 +61,7 @@

<!-- Let's ensure the correct versions with dependencyManagement.
We want to run our tests and compilations using netty and jackson version used in the runtime (provided as OSGi features).
We want to run our tests and compilations using netty version used in the runtime (provided as OSGi features).
slf4j-api version is locked to core-version. Also: slf4j comes via openHAB logging, so setting it here as provided to
have the right OSGi imports.
-->
Expand All @@ -73,21 +73,6 @@
<version>${slf4j.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-annotations</artifactId>
<version>${jackson.version}</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-core</artifactId>
<version>${jackson.version}</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>${jackson.version}</version>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-buffer</artifactId>
Expand Down Expand Up @@ -213,87 +198,97 @@
<dependency>
<groupId>com.typesafe.netty</groupId>
<artifactId>netty-reactive-streams-http</artifactId>
<version>2.0.4</version>
<version>2.0.5</version>
</dependency>
<dependency>
<groupId>com.typesafe.netty</groupId>
<artifactId>netty-reactive-streams</artifactId>
<version>2.0.4</version>
<version>2.0.5</version>
</dependency>
<dependency>
<groupId>org.reactivestreams</groupId>
<artifactId>reactive-streams</artifactId>
<version>1.0.2</version>
<version>1.0.3</version>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>annotations</artifactId>
<version>2.15.56</version>
<version>2.17.102</version>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>auth</artifactId>
<version>2.15.56</version>
<version>2.17.102</version>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>aws-core</artifactId>
<version>2.15.56</version>
<version>2.17.102</version>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>aws-json-protocol</artifactId>
<version>2.15.56</version>
<version>2.17.102</version>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>dynamodb-enhanced</artifactId>
<version>2.15.56</version>
<version>2.17.102</version>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>dynamodb</artifactId>
<version>2.15.56</version>
<version>2.17.102</version>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>http-client-spi</artifactId>
<version>2.15.56</version>
<version>2.17.102</version>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>json-utils</artifactId>
<version>2.17.102</version>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>metrics-spi</artifactId>
<version>2.15.77</version>
<version>2.17.102</version>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>netty-nio-client</artifactId>
<version>2.15.77</version>
<version>2.17.102</version>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>profiles</artifactId>
<version>2.15.56</version>
<version>2.17.102</version>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>protocol-core</artifactId>
<version>2.15.56</version>
<version>2.17.102</version>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>regions</artifactId>
<version>2.15.56</version>
<version>2.17.102</version>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>sdk-core</artifactId>
<version>2.15.56</version>
<version>2.17.102</version>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>third-party-jackson-core</artifactId>
<version>2.17.102</version>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>utils</artifactId>
<version>2.15.56</version>
<version>2.17.102</version>
</dependency>
<dependency>
<groupId>software.amazon.eventstream</groupId>
Expand Down
16 changes: 2 additions & 14 deletions bundles/org.openhab.persistence.dynamodb/scripts/fetch_sdk_pom.xml
View file
Expand Up @@ -10,31 +10,19 @@
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>dynamodb-enhanced</artifactId>
<version>2.15.56</version>
<version>2.17.102</version>
<exclusions>
<!-- exclude artifacts available via openhab jackson feature -->
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
</exclusion>
<exclusion>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-annotations</artifactId>
</exclusion>
<exclusion>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-core</artifactId>
</exclusion>
<exclusion>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>netty-nio-client</artifactId>
<version>2.15.77</version>
<version>2.17.102</version>
<exclusions>
<!-- exclude artifacts available via openhab netty feature, or otherwise added in feature.xml -->
<exclusion>
Expand Down
1 change: 0 additions & 1 deletion bundles/org.openhab.persistence.dynamodb/src/main/feature/feature.xml
View file
Expand Up @@ -4,7 +4,6 @@

<feature name="openhab-persistence-dynamodb" description="DynamoDB Persistence" version="${project.version}">
<feature>openhab-runtime-base</feature>
<feature dependency="true">openhab.tp-jackson</feature>
<feature dependency="true">openhab.tp-netty</feature>
<bundle start-level="80">mvn:org.openhab.addons.bundles/org.openhab.persistence.dynamodb/${project.version}</bundle>
<configfile finalname="${openhab.conf}/services/dynamodb.cfg" override="false">mvn:org.openhab.addons.features.karaf/org.openhab.addons.features.karaf.openhab-addons-external/${project.version}/cfg/dynamodb</configfile>
Expand Down
43 changes: 23 additions & 20 deletions ...ence.dynamodb/src/main/java/org/openhab/persistence/dynamodb/internal/DynamoDBConfig.java
View file
Expand Up @@ -49,14 +49,13 @@ public class DynamoDBConfig {
public static final String DEFAULT_TABLE_NAME = "openhab";
public static final long DEFAULT_READ_CAPACITY_UNITS = 1;
public static final long DEFAULT_WRITE_CAPACITY_UNITS = 1;
public static final RetryMode DEFAULT_RETRY_MODE = RetryMode.STANDARD;
private static final Logger LOGGER = LoggerFactory.getLogger(DynamoDBConfig.class);

private long readCapacityUnits;
private long writeCapacityUnits;
private Region region;
private AwsCredentials credentials;
private RetryPolicy retryPolicy;
private Optional<RetryPolicy> retryPolicy;
private ExpectedTableSchema tableRevision;
private String table;
private String tablePrefixLegacy;
Expand All @@ -81,7 +80,7 @@ public class DynamoDBConfig {
}
region = Region.of(regionName);

RetryMode retryMode = RetryMode.STANDARD;
Optional<RetryMode> retryMode = Optional.empty();
AwsCredentials credentials;
String accessKey = (String) config.get("accessKey");
String secretKey = (String) config.get("secretKey");
Expand Down Expand Up @@ -110,11 +109,12 @@ public class DynamoDBConfig {
return Optional.of(value);
}
}
LOGGER.warn("Unknown retry_mode '{}' in profile. Ignoring and using default {} retry mode.",
retry_mode, DEFAULT_RETRY_MODE);
LOGGER.warn(
"Unknown retry_mode '{}' in profile. Ignoring and using SDK default retry mode.",
retry_mode);
return Optional.empty();

}).orElse(DEFAULT_RETRY_MODE);
});
LOGGER.debug("Retry mode {}", retryMode);
}

Expand Down Expand Up @@ -169,18 +169,19 @@ public class DynamoDBConfig {
switch (tableRevision) {
case NEW:
LOGGER.debug("Using new DynamoDB table schema");
return DynamoDBConfig.newSchema(region, credentials, AwsRetryPolicy.forRetryMode(retryMode), table,
readCapacityUnits, writeCapacityUnits, expireDays);
return DynamoDBConfig.newSchema(region, credentials, retryMode.map(AwsRetryPolicy::forRetryMode),
table, readCapacityUnits, writeCapacityUnits, expireDays);
case LEGACY:
LOGGER.warn(
"Using legacy DynamoDB table schema. It is recommended to transition to new schema by defining 'table' parameter and not configuring 'tablePrefix'");
return DynamoDBConfig.legacySchema(region, credentials, AwsRetryPolicy.forRetryMode(retryMode),
return DynamoDBConfig.legacySchema(region, credentials, retryMode.map(AwsRetryPolicy::forRetryMode),
tablePrefixLegacy, readCapacityUnits, writeCapacityUnits);
case MAYBE_LEGACY:
LOGGER.debug(
"Unclear whether we should use new legacy DynamoDB table schema. It is recommended to explicitly define new 'table' parameter. The correct table schema will be detected at runtime.");
return DynamoDBConfig.maybeLegacySchema(region, credentials, AwsRetryPolicy.forRetryMode(retryMode),
table, tablePrefixLegacy, readCapacityUnits, writeCapacityUnits, expireDays);
return DynamoDBConfig.maybeLegacySchema(region, credentials,
retryMode.map(AwsRetryPolicy::forRetryMode), table, tablePrefixLegacy, readCapacityUnits,
writeCapacityUnits, expireDays);
default:
throw new IllegalStateException("Unhandled enum. Bug");
}
Expand All @@ -190,26 +191,28 @@ public class DynamoDBConfig {
}
}

private static DynamoDBConfig newSchema(Region region, AwsCredentials credentials, RetryPolicy retryPolicy,
String table, long readCapacityUnits, long writeCapacityUnits, @Nullable Integer expireDays) {
private static DynamoDBConfig newSchema(Region region, AwsCredentials credentials,
Optional<RetryPolicy> retryPolicy, String table, long readCapacityUnits, long writeCapacityUnits,
@Nullable Integer expireDays) {
return new DynamoDBConfig(region, credentials, retryPolicy, table, "", ExpectedTableSchema.NEW,
readCapacityUnits, writeCapacityUnits, expireDays);
}

private static DynamoDBConfig legacySchema(Region region, AwsCredentials credentials, RetryPolicy retryPolicy,
String tablePrefixLegacy, long readCapacityUnits, long writeCapacityUnits) {
private static DynamoDBConfig legacySchema(Region region, AwsCredentials credentials,
Optional<RetryPolicy> retryPolicy, String tablePrefixLegacy, long readCapacityUnits,
long writeCapacityUnits) {
return new DynamoDBConfig(region, credentials, retryPolicy, "", tablePrefixLegacy, ExpectedTableSchema.LEGACY,
readCapacityUnits, writeCapacityUnits, null);
}

private static DynamoDBConfig maybeLegacySchema(Region region, AwsCredentials credentials, RetryPolicy retryPolicy,
String table, String tablePrefixLegacy, long readCapacityUnits, long writeCapacityUnits,
@Nullable Integer expireDays) {
private static DynamoDBConfig maybeLegacySchema(Region region, AwsCredentials credentials,
Optional<RetryPolicy> retryPolicy, String table, String tablePrefixLegacy, long readCapacityUnits,
long writeCapacityUnits, @Nullable Integer expireDays) {
return new DynamoDBConfig(region, credentials, retryPolicy, table, tablePrefixLegacy,
ExpectedTableSchema.MAYBE_LEGACY, readCapacityUnits, writeCapacityUnits, expireDays);
}

private DynamoDBConfig(Region region, AwsCredentials credentials, RetryPolicy retryPolicy, String table,
private DynamoDBConfig(Region region, AwsCredentials credentials, Optional<RetryPolicy> retryPolicy, String table,
String tablePrefixLegacy, ExpectedTableSchema tableRevision, long readCapacityUnits,
long writeCapacityUnits, @Nullable Integer expireDays) {
this.region = region;
Expand Down Expand Up @@ -251,7 +254,7 @@ public long getWriteCapacityUnits() {
return writeCapacityUnits;
}

public RetryPolicy getRetryPolicy() {
public Optional<RetryPolicy> getRetryPolicy() {
return retryPolicy;
}

Expand Down
4 changes: 3 additions & 1 deletion ...b/src/main/java/org/openhab/persistence/dynamodb/internal/DynamoDBPersistenceService.java
View file
Expand Up @@ -63,6 +63,7 @@

import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
import software.amazon.awssdk.awscore.AwsRequestOverrideConfiguration;
import software.amazon.awssdk.awscore.defaultsmode.DefaultsMode;
import software.amazon.awssdk.core.async.SdkPublisher;
import software.amazon.awssdk.core.client.config.ClientAsyncConfiguration;
import software.amazon.awssdk.core.client.config.ClientOverrideConfiguration;
Expand Down Expand Up @@ -125,7 +126,7 @@ void overrideConfig(ClientOverrideConfiguration.Builder config) {
DynamoDBConfig localDbConfig = dbConfig;
config.apiCallAttemptTimeout(TIMEOUT_API_CALL_ATTEMPT).apiCallTimeout(TIMEOUT_API_CALL);
if (localDbConfig != null) {
config.retryPolicy(localDbConfig.getRetryPolicy());
localDbConfig.getRetryPolicy().ifPresent(config::retryPolicy);
}
}

Expand Down Expand Up @@ -220,6 +221,7 @@ private boolean ensureClient() {
return true;
}
DynamoDbAsyncClientBuilder lowlevelClientBuilder = DynamoDbAsyncClient.builder()
.defaultsMode(DefaultsMode.STANDARD)
.credentialsProvider(StaticCredentialsProvider.create(localDbConfig.getCredentials()))
.httpClient(NettyNioAsyncHttpClient.builder().maxConcurrency(MAX_CONCURRENCY).build())
.asyncConfiguration(
Expand Down

0 comments on commit 01f1a9c

Please sign in to comment.