Concurrent DynamoDB Client Creating Excessive KMS API Calls #2129
Comments
jshlbrd
added
bug
|
Hi @jshlbrd , I see you opened a discussion and also an issue. I will close the discussion in favor of tracking it here. I'm a bit confused by the problem you are describing. When creating a table on Dynamo, you can specify which key you want to use for encryption at rest (whether its AWS managed or a CMK), the encryption is happening on the service side as described in this doc:
In other words, the SDK does not know / care about encryption at rest for dynamo. To make sure I'm not wrong, I provisioned Dynamodb tables with CMK per your description: $ aws dynamodb describe-table --table-name aws_sdk_go_v2_2128
{
"Table": {
"AttributeDefinitions": [
{
"AttributeName": "pk",
"AttributeType": "S"
}
],
"TableName": "aws_sdk_go_v2_2128",
"KeySchema": [
{
"AttributeName": "pk",
"KeyType": "HASH"
}
],
"TableStatus": "ACTIVE",
"CreationDateTime": "2023-05-15T15:46:02.519000-07:00",
"ProvisionedThroughput": {
"NumberOfDecreasesToday": 0,
"ReadCapacityUnits": 1,
"WriteCapacityUnits": 1
},
"TableSizeBytes": 0,
"ItemCount": 0,
"TableArn": "arn:aws:dynamodb:us-east-1:REDACTED:table/aws_sdk_go_v2_2128",
"TableId": "REDACTED",
"SSEDescription": {
"Status": "ENABLED",
"SSEType": "KMS",
"KMSMasterKeyArn": "arn:aws:kms:us-east-1:REDACTED"
},
"TableClassSummary": {
"TableClass": "STANDARD"
},
"DeletionProtectionEnabled": false
}
}Using the code you have provided I'm not seeing any network calls to KMS. ctx := context.TODO()
cfg, err := config.LoadDefaultConfig(ctx, config.WithClientLogMode(aws.LogResponseWithBody|aws.LogRequestWithBody))
if err != nil {
panic(err)
}You will see 25 network calls to the Dynamodb service, but no calls are made to KMS. Can you provide some more information about how you noticed the difference? From playing around with Cloudtrail I wasn't able to identify any Thanks, |
RanVaknin
added
response-requested
|
Hey @RanVaknin, thanks for taking a look at this! The API calls are not made by the client, they're made by the DynamoDB service -- but the number of KMS calls made by DynamoDB appear to be the direct result of using concurrency with the Go client. Referencing this documentation may help clarify (this is from the section about the GetItem API call):
Here's an example CloudTrail event that shows the behavior I'm describing (removed and redacted private information): {
"eventVersion": "1.08",
"userIdentity": {
"type": "AssumedRole",
},
"invokedBy": "dynamodb.amazonaws.com"
},
"eventTime": "2023-05-16T00:19:11Z",
"eventSource": "kms.amazonaws.com",
"eventName": "Decrypt",
"awsRegion": "us-east-1",
"sourceIPAddress": "dynamodb.amazonaws.com",
"userAgent": "dynamodb.amazonaws.com",
"requestParameters": {
"encryptionContext": {
"aws:dynamodb:tableName": "aws_sdk_go_v2_2128_concurrency",
"aws:dynamodb:subscriberId": `AWS ACCOUNT ID`
},
"encryptionAlgorithm": "SYMMETRIC_DEFAULT"
},
"responseElements": null,
"requestID": `REQUEST ID`
"eventID": `EVENT ID`
"readOnly": true,
"resources": [
{
"accountId": `AWS ACCOUNT ID`
"type": "AWS::KMS::Key",
"ARN": `KMS Key ARN`
}
],
"eventType": "AwsApiCall",
"managementEvent": true,
"recipientAccountId": `AWS ACCOUNT ID`
"eventCategory": "Management"
}I see approximately 25 events like the above for the concurrent client. One thing worth noting, I'm using role assumption and each Decrypt call has a unique access key ID. This is a guess, but is it possible that DynamoDB is treating each goroutine as its own client due to how authentication is handled when using a concurrent client? |
|
Hi @jshlbrd , Thanks for additional info.
When you say "unique access key ID" are you referring to How are you assuming the role? Are you reading it from env variables? ctx := context.TODO()
cfg, err := config.LoadDefaultConfig(ctx, config.WithClientLogMode(aws.LogResponseWithBody|aws.LogRequestWithBody))
if err != nil {
panic(err)
}
client := sts.NewFromConfig(cfg)
appCreds := stscreds.NewAssumeRoleProvider(client, "arn:aws:iam::REDACTED:role/role_name")
cfgCopy := cfg.Copy()
cfgCopy.Credentials = aws.NewCredentialsCache(appCreds)
svc := dynamodb.NewFromConfig(cfgCopy)
// ... more code hereWhen testing this code by enabling the request logs I can examine the request sent by each Goroutine and I notice in the Authorization header, that all access keys are the same. Should look something like this: SDK 2023/05/16 11:40:33 DEBUG Request
POST / HTTP/1.1
Host: dynamodb.us-east-1.amazonaws.com
User-Agent: aws-sdk-go-v2/1.18.0 os/macos lang/go/1.19.1 md/GOOS/darwin md/GOARCH/arm64 api/dynamodb/1.19.7
Content-Length: 69
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: REDACTED
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential={{Secret Access ID}}/20230516/us-east-1/dynamodb/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-date;x-amz-security-token;x-amz-target, Signature=REDACTED
Content-Type: application/x-amz-json-1.0
X-Amz-Date: 20230516T184033Z
X-Amz-Security-Token: REDACTED
X-Amz-Target: DynamoDB_20120810.GetItem
{"Key":{"pk":{"S":"b"}},"TableName":"aws_sdk_go_v2_2128_concurrency"}
This should be the same as the implicit assume role you are probably doing through injecting env variable but I thought we can try anyway. Another theory I can think of, is because you are sending all of these requests at once, the Dynamo server is not able to effectively cache the requests - resulting in multiple At any rate, this seems like a service side limitation. If none of these methods work for you, we will have to upstream this to the Dynamodb team internally. I can do this on your behalf, or if you file a support ticket, you'll get assigned a dedicated person that will help navigate this request to the service team - they usually have a little more tools than us to help expedite things. If you choose the latter, just make sure to have them CC me on the ticket using my alias Thanks again, |
|
Thanks @RanVaknin!
Not quite, it's the {
"userIdentity": {
"type": "AssumedRole",
"principalId": "REDACTED",
"arn": "REDACTED",
"accountId": "REDACTED",
"accessKeyId": "REDACTED",
"sessionContext": {
"sessionIssuer": {
"type": "Role",
"principalId": "REDACTED",
"arn": "REDACTED",
"accountId": "REDACTED",
"userName": "REDACTED"
}
},
"invokedBy": "dynamodb.amazonaws.com"
}
}Based on your suggestions I confirmed the following:
If you can file a request with the DynamoDB team then that would help. My feeling is that, for some reason, the DynamoDB service does not recognize that the Go client is a single requester when it is used concurrently; it may be treating each goroutine as a unique client, which causes the key caching feature to fail. |
|
Hi @jshlbrd, Thanks for trying to implement the suggestions. Can you please provide me with 2 separate The service team will need that to identify your requests and investigate. Thanks! |
|
@RanVaknin Here are the 25 requests from my latest run with the concurrent requests:
|
|
Hi @jshlbrd , Thanks for that. Just to triple check, when you enable the request logging, you see the same principal / AWS Secret Access Key on sent on the request? The doc you linked specifies that the principal is what influences the caching mechanism. Principal should look something like this: In my last provided log: SDK 2023/05/16 11:40:33 DEBUG Request
POST / HTTP/1.1
...
...
...
Authorization: AWS4-HMAC-SHA256 Credential={{ PRINCIPAL }}/20230516/us-east-1/dynamodb/aws4_request,
...
...
...Is the principal sent here with every request the same? (In my repro code it is the same) because if it is not, then that's what is throwing Dynamo off. Thanks again for your prompt responses. All the best, |
|
@RanVaknin That's correct, the principal is the same in every request. Thank you! |
|
Hi @jshlbrd, I created an internal ticket, we now have to wait for a response. My suggestion would be to use the iterative approach for now so you can save on billing costs. Additionally if you see this is taking too long, feel free to reach out to AWS Support, they might be able to expedite this faster than I can. I will keep you updated when I learn anything new.
|
RanVaknin
added
service-api
|
Hi @jshlbrd , I finally heard back from the dynamo team:
This is further explained by this little nuance (in bold) from the doc you shared:
Since the goroutine will establish a new connection on a different host, in memory caching will occur per host. To avoid the excess decrypt call I'd stick with invoking those operations sequentially rather than concurrently. Thanks again for your patience. All the best, |
|
This issue is now closed. Comments on closed issues are hard for our team to see. |
Describe the bug
When the DynamoDB client is used concurrently with a customer-managed key it creates a
DecryptAPI call per goroutine / thread instead of creating only one API call for the client. More detail is described here: #2128.Expected Behavior
The DynamoDB client makes 1
DecryptAPI call up to every 5 minutes (based on the documentation here) for a single client used in any number of goroutines.Current Behavior
The DynamoDB client makes N
DecryptAPI call up to every 5 minutes for a single client based on the number of goroutines. For example, if the client is used in 25 goroutines, then close to 25 Decrypt calls are made.Reproduction Steps
#2128 (comment)
Possible Solution
No response
Additional Information/Context
No response
AWS Go SDK V2 Module Versions Used
github.com/aws/aws-sdk-go-v2 v1.18.0
github.com/aws/aws-sdk-go-v2/config v1.18.25
github.com/aws/aws-sdk-go-v2/credentials v1.13.24
github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue v1.10.25
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.13.3
github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.33
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.27
github.com/aws/aws-sdk-go-v2/internal/ini v1.3.34
github.com/aws/aws-sdk-go-v2/service/dynamodb v1.19.7
github.com/aws/aws-sdk-go-v2/service/dynamodbstreams v1.14.11
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.9.11
github.com/aws/aws-sdk-go-v2/service/internal/endpoint-discovery v1.7.27
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.27
github.com/aws/aws-sdk-go-v2/service/sso v1.12.10
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.14.10
github.com/aws/aws-sdk-go-v2/service/sts v1.19.0
github.com/aws/aws-sdk-go-v2/service/xray v1.16.11
Compiler and Version used
go1.19.6 linux/arm64
Operating System and version
Debian 11
The text was updated successfully, but these errors were encountered: