Skip to content

Commit

Permalink
remote_storage: AWS_PROFILE with endpoint overrides in ~/.aws/config …
Browse files Browse the repository at this point in the history
…(updates AWS SDKs) (#7664)

Before this PR, using the AWS SDK profile feature for running against
minio didn't work because
* our SDK versions were too old and didn't include
  awslabs/aws-sdk-rust#1060 and 
* we didn't massage the s3 client config builder correctly.

This PR
* udpates all the AWS SDKs we use to, respectively, the latest version I
could find on crates.io (Is there a better process?)
* changes the way remote_storage constructs the S3 client, and
* documents how to run the test suite against real S3 & local minio.

Regarding the changes to `remote_storage`: if one reads the SDK docs, it
is clear that the recommended way is to use `aws_config::from_env`, then
customize.
What we were doing instead is to use the `aws_sdk_s3` builder directly.

To get the `local-minio` in the added docs working, I needed to update
both the SDKs and make the changes to the `remote_storage`. See the
commit history in this PR for details.

Refs:
* byproduct: smithy-lang/smithy-rs#3633
* follow-up on deprecation:
#7665
* follow-up for scrubber S3 setup:
#7667
  • Loading branch information
problame committed May 9, 2024
1 parent d5399b7 commit ab10523
Show file tree
Hide file tree
Showing 6 changed files with 287 additions and 78 deletions.
88 changes: 54 additions & 34 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

14 changes: 7 additions & 7 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -52,14 +52,14 @@ azure_storage_blobs = "0.19"
flate2 = "1.0.26"
async-stream = "0.3"
async-trait = "0.1"
aws-config = { version = "1.1.4", default-features = false, features=["rustls"] }
aws-sdk-s3 = "1.14"
aws-config = { version = "1.3", default-features = false, features=["rustls"] }
aws-sdk-s3 = "1.26"
aws-sdk-iam = "1.15.0"
aws-smithy-async = { version = "1.1.4", default-features = false, features=["rt-tokio"] }
aws-smithy-types = "1.1.4"
aws-credential-types = "1.1.4"
aws-sigv4 = { version = "1.2.0", features = ["sign-http"] }
aws-types = "1.1.7"
aws-smithy-async = { version = "1.2.1", default-features = false, features=["rt-tokio"] }
aws-smithy-types = "1.1.9"
aws-credential-types = "1.2.0"
aws-sigv4 = { version = "1.2.1", features = ["sign-http"] }
aws-types = "1.2.0"
axum = { version = "0.6.20", features = ["ws"] }
base64 = "0.13.0"
bincode = "1.3"
Expand Down

1 comment on commit ab10523

@github-actions
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

3105 tests run: 2957 passed, 2 failed, 146 skipped (full report)


Failures on Postgres 14

  • test_storage_controller_many_tenants[github-actions-selfhosted]: release
  • test_bulk_tenant_create[github-actions-selfhosted-5]: release
# Run all failed tests locally:
scripts/pytest -vv -n $(nproc) -k "test_storage_controller_many_tenants[release-pg14-github-actions-selfhosted] or test_bulk_tenant_create[release-pg14-github-actions-selfhosted-5]"
Flaky tests (3)

Postgres 16

  • test_vm_bit_clear_on_heap_lock: release

Postgres 15

  • test_statvfs_pressure_usage: debug

Postgres 14

  • test_partial_evict_tenant[relative_equal]: release

Code coverage* (full report)

  • functions: 31.4% (6315 of 20126 functions)
  • lines: 47.3% (47594 of 100686 lines)

* collected from Rust tests only


The comment gets automatically updated with the latest test results
ab10523 at 2024-05-09T10:37:09.093Z :recycle:

Please sign in to comment.