remote_storage: AWS_PROFILE with endpoint overrides in ~/.aws/config (updates AWS SDKs) #7664
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… (updates AWS SDKs) Before this PR, using the AWS SDK profile feature for running against minio didn't work, * either because our SDK versions were too old and didn't include awslabs/aws-sdk-rust#1060 * or because we didn't massage the s3 client config builder correctly. This PR * udpates all the AWS SDKs we use to, respectively, the latest version I could find on crates.io (Is there a better process?) * changes the way remote_storage constructs the S3 client. Regarding the latter, if one reads the SDK docs, it is clear that the recommended way is to use `aws_config::from_env` because that sources all the default, then customize where needed. What we were doing instead is to use the `aws_sdk_s3` builder directly.
It fails with
```
2024-05-08T18:41:09.230477Z INFO attach{tenant_id=ccd34c1399035a50e09bbc8b05de940a shard_id=0000 gen=00000001}:preload: listing remote timelines
2024-05-08T18:41:12.337353Z INFO attach{tenant_id=ccd34c1399035a50e09bbc8b05de940a shard_id=0000 gen=00000001}:preload:lazy_load_identity: constructed abstract provider from config file chain=ProfileChain { base: AccessKey(Credentials { provider_name: "ProfileFile", access_key_id: "foobar123", secret_access_key: "** redacted **", expires_after: "never" }), chain: [] }
2024-05-08T18:41:12.337430Z INFO attach{tenant_id=ccd34c1399035a50e09bbc8b05de940a shard_id=0000 gen=00000001}:preload:lazy_load_identity: loaded base credentials creds=Credentials { provider_name: "ProfileFile", access_key_id: "foobar123", secret_access_key: "** redacted **", expires_after: "never" }
2024-05-08T18:41:12.342052Z INFO attach{tenant_id=ccd34c1399035a50e09bbc8b05de940a shard_id=0000 gen=00000001}:preload: list identifiers in prefix tenants/ccd34c1399035a50e09bbc8b05de940a/timelines/ failed, will retry (attempt 1): Failed to download a remote file: Failed to list S3 prefixes
Caused by:
0: dispatch failure
1: io error
2: error trying to connect: dns error: failed to lookup address information: Name or service not known
3: dns error: failed to lookup address information: Name or service not known
4: failed to lookup address information: Name or service not known
```
This reverts commit db2011d.
it fails with
```
2024-05-08T18:46:08.713500Z INFO attach{tenant_id=ff8ceb3c7e9b0963650bee7c29cb74e4 shard_id=0000 gen=00000001}:preload: listing remote timelines
2024-05-08T18:46:11.778847Z WARN attach{tenant_id=ff8ceb3c7e9b0963650bee7c29cb74e4 shard_id=0000 gen=00000001}:preload:lazy_load_identity: profile `services local-minio-services` ignored because `services local-minio-services` was not a valid identifier
2024-05-08T18:46:11.779070Z INFO attach{tenant_id=ff8ceb3c7e9b0963650bee7c29cb74e4 shard_id=0000 gen=00000001}:preload:lazy_load_identity: constructed abstract provider from config file chain=ProfileChain { base: AccessKey(Credentials { provider_name: "ProfileFile", access_key_id: "foobar123", secret_access_key: "** redacted **", expires_after: "never" }), chain: [] }
2024-05-08T18:46:11.779102Z INFO attach{tenant_id=ff8ceb3c7e9b0963650bee7c29cb74e4 shard_id=0000 gen=00000001}:preload:lazy_load_identity: first credentials will be loaded from AccessKey(Credentials { provider_name: "ProfileFile", access_key_id: "foobar123", secret_access_key: "** redacted **", expires_after: "never" }) base=AccessKey(Credentials { provider_name: "ProfileFile", access_key_id: "foobar123", secret_access_key: "** redacted **", expires_after: "never" })
2024-05-08T18:46:11.779129Z INFO attach{tenant_id=ff8ceb3c7e9b0963650bee7c29cb74e4 shard_id=0000 gen=00000001}:preload:lazy_load_identity: loaded base credentials creds=Credentials { provider_name: "ProfileFile", access_key_id: "foobar123", secret_access_key: "** redacted **", expires_after: "never" }
2024-05-08T18:46:11.779168Z INFO attach{tenant_id=ff8ceb3c7e9b0963650bee7c29cb74e4 shard_id=0000 gen=00000001}:preload:lazy_load_identity: identity cache miss occurred; added new identity (took 692.001<C2><B5>s) new_expiration=2024-05-08T19:01:11.778442Z valid_for=899.999275768s partition=IdentityCachePartition(6)
2024-05-08T18:46:11.782016Z INFO attach{tenant_id=ff8ceb3c7e9b0963650bee7c29cb74e4 shard_id=0000 gen=00000001}:preload: list identifiers in prefix tenants/ff8ceb3c7e9b0963650bee7c29cb74e4/timelines/ failed, will retry (attempt 1): Failed to download a remote file: Failed to list S3 prefixes
Caused by:
0: dispatch failure
1: io error
2: error trying to connect: dns error: failed to lookup address information: Name or service not known
3: dns error: failed to lookup address information: Name or service not known
4: failed to lookup address information: Name or service not known
```
This reverts commit 5517893.
This was referenced May 8, 2024
koivunej
approved these changes
May 8, 2024
3024 tests run: 2892 passed, 0 failed, 132 skipped (full report)Code coverage* (full report)
* collected from Rust tests only The comment gets automatically updated with the latest test results
3e13ac4 at 2024-05-09T08:34:14.513Z :recycle: |
Can you link to what you're referring to here? I'm not following |
|
Have you checked whether the commands written in #6202 still work with this PR applied? |
|
Yep, still works. Extended the docs in this PR. Verified manually that the tests actually use the env-var-configured bucket / credentials. Side note: the |
That's probably because the bucket doesn't have versioning enabled. It needs to be enabled explicitly (also in S3 but those buckets are long lived and the configuration lives outside of this repo). For #6533's pytest changes I had to add a |
a-masterov
pushed a commit
that referenced
this pull request
May 20, 2024
…(updates AWS SDKs) (#7664) Before this PR, using the AWS SDK profile feature for running against minio didn't work because * our SDK versions were too old and didn't include awslabs/aws-sdk-rust#1060 and * we didn't massage the s3 client config builder correctly. This PR * udpates all the AWS SDKs we use to, respectively, the latest version I could find on crates.io (Is there a better process?) * changes the way remote_storage constructs the S3 client, and * documents how to run the test suite against real S3 & local minio. Regarding the changes to `remote_storage`: if one reads the SDK docs, it is clear that the recommended way is to use `aws_config::from_env`, then customize. What we were doing instead is to use the `aws_sdk_s3` builder directly. To get the `local-minio` in the added docs working, I needed to update both the SDKs and make the changes to the `remote_storage`. See the commit history in this PR for details. Refs: * byproduct: smithy-lang/smithy-rs#3633 * follow-up on deprecation: #7665 * follow-up for scrubber S3 setup: #7667
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Before this PR, using the AWS SDK profile feature for running against minio didn't work because
Service Specific Configuration Override awslabs/aws-sdk-rust#1060 and
This PR
Regarding the changes to
remote_storage: if one reads the SDK docs, it is clear that the recommended way is to useaws_config::from_env, then customize.What we were doing instead is to use the
aws_sdk_s3builder directly.To get the
local-minioin the added docs working, I needed to update both the SDKs and make the changes to theremote_storage. See the commit history in this PR for details.Refs: