Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

blob/s3blob: Support S3 server side encryption headers for Write and Copy #3340

Merged
merged 6 commits into from Feb 25, 2024

Conversation

tristan-newmann
Copy link
Contributor

Adds support to s3blob for setting AWS S3 server side encryption headers when making requests that require such headers. The additional settings can be specified with the ssetype and kmskeyid URL params, similar to the other configuration settings

Fixes #3337

Local testing

SDKv2

Bucket policy enforces KMS headers - none provided (effectively the issue described in #3337)
Expect: AccessDenied failure
image

Bucket policy enforces KMS headers - values provided
Expect: No errors. Object is uploaded
image

Bucket policy doesn't enforce KMS - no values provided
Expect: No errors. Object is uploaded
image

SDKv1

Bucket policy enforces KMS headers - none provided (effectively the issue described in #3337)
Expect: AccessDenied failure
image

Bucket policy enforces KMS headers - values provided
Expect: No errors. Object is uploaded
image

Bucket policy doesn't enforce KMS - no values provided
Expect: No errors. Object is uploaded
image

Copy link

google-cla bot commented Nov 21, 2023

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

Adds support to s3blob for setting AWS S3 server side encryption headers when making requests that require such headers. The additional settings can be specified with the `ssetype` and `kmskeyid` URL params, similar to the other configuration settings
@tristan-newmann tristan-newmann force-pushed the enhancement/s3-support-sse-headers branch from c8d1a71 to 2017a81 Compare November 21, 2023 06:13
aws/aws.go Outdated Show resolved Hide resolved
blob/s3blob/s3blob.go Outdated Show resolved Hide resolved
blob/s3blob/s3blob.go Outdated Show resolved Hide resolved
blob/s3blob/s3blob_test.go Show resolved Hide resolved
Copy link

codecov bot commented Nov 21, 2023

Codecov Report

Attention: Patch coverage is 60.00000% with 16 lines in your changes are missing coverage. Please review.

Project coverage is 73.13%. Comparing base (6d5d289) to head (1cb7fd0).
Report is 19 commits behind head on master.

Files Patch % Lines
blob/s3blob/s3blob.go 60.00% 8 Missing and 8 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #3340      +/-   ##
==========================================
- Coverage   77.47%   73.13%   -4.34%     
==========================================
  Files         104      113       +9     
  Lines       13933    14825     +892     
==========================================
+ Hits        10794    10843      +49     
- Misses       2378     3210     +832     
- Partials      761      772      +11     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@tristan-newmann
Copy link
Contributor Author

@vangent Thanks for the feedback! I implemented the requested changes. I left a couple unresolved since I wasn't sure if I did it exactly how you had in mind. This is ready for re-review

Let me know if I can force push to fix the CLA check, one of the commits has a slightly incorrect version of my email due to a local misconfiguration

@vangent
Copy link
Contributor

vangent commented Nov 30, 2023

Please go ahead and force-push and I'll re-review.

Updates the EncryptionType field of s3blob.Options to use the AWS type. An error is thrown at initialisation time if an invalid value is provided for the ssetype parameter
@tristan-newmann tristan-newmann force-pushed the enhancement/s3-support-sse-headers branch from 84f7555 to a7a2899 Compare December 1, 2023 00:31
@tristan-newmann
Copy link
Contributor Author

Ready for re-review

blob/s3blob/s3blob.go Outdated Show resolved Hide resolved
blob/s3blob/s3blob.go Outdated Show resolved Hide resolved
blob/s3blob/s3blob.go Outdated Show resolved Hide resolved
blob/s3blob/s3blob.go Outdated Show resolved Hide resolved
blob/s3blob/s3blob.go Outdated Show resolved Hide resolved
blob/s3blob/s3blob.go Outdated Show resolved Hide resolved
blob/s3blob/s3blob.go Outdated Show resolved Hide resolved
@tristan-newmann
Copy link
Contributor Author

@vangent Sorry for the delay on this one, priorities forced my to put this aside for a moment, but I am back now. This is ready for re-review, and I re-did the smoke testing as well.

Unit tests pass

image

Redoing the smoke testing after the adjustments:

SDKv2

Bucket policy enforces KMS headers - none provided (effectively the issue described in #3337)

Expect: AccessDenied failure
image

Bucket policy enforces KMS headers - values provided

Expect: No errors. Object is uploaded
image
image

Bucket policy doesn't enforce KMS - no values provided

Expect: No errors. Object is uploaded
image

SDKv1

Bucket policy enforces KMS headers - none provided (effectively the issue described in #3337)

Expect: AccessDenied failure
image

Bucket policy enforces KMS headers - values provided

Expect: No errors. Object is uploaded
image

Bucket policy doesn't enforce KMS - no values provided

Expect: No errors. Object is uploaded
image

blob/s3blob/s3blob.go Outdated Show resolved Hide resolved
@vangent vangent merged commit fb4e4b9 into google:master Feb 25, 2024
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

blob/s3: Support Server Side encryption settings in bucket url
2 participants