feat(storage): refactor to use transport-agnostic interface #6465
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Change production code paths for Composer to use the new transport agnostic interface. Required a few setup steps to make tests pass, which will be needed for other integrations too: * Add initialization of tc to NewClient * Add new useGRPC flag to remove switches on tc in Reader/Writer Also a few cleanups: * Use EncryptionKey from dstObject * Use defaultGen instead of hard coded int Also, a unit test checks that setting Generation with composer causes a failure. Currently this is enforced by applyConds. I added a separate validation for this instead, but I could do a pass- through of the Generation through the interface -- it's just confusing looking because setting Generation on a new object is never valid.
This takes out a few lines of boilerplate that will be present in all top-level methods.
Migrate Copier to use the new transport-agnostic interface. I found one additional piece of data that needs to be passed back through the interface in the process (object size). I removed one integration test. This test is old and currently calls rewriteObject in a way that an external library user could not accomplish. I don't think this test adds any value.
* chore(storage): migrate GetServiceAccount * migrate ListBuckets * migrate CreateBucket and refactor interface method * update tests and BucketIterator with retryConfig * Revert "update tests and BucketIterator with retryConfig" This reverts commit b7662b4. * remove unused field and internal method upon migration
* chore(storage): migrate IAM operations * remove unused field
Migrate ACL ops to the new transport agnostic interface. No interface changes were necessary for this one and tests pass.
Migrates the Notification APIs to the transport agnostic surface. Integration tests passed and added a check to ensure that AddNotification returns a Notification with a non-empty ID.
* test: update continuous.sh for storage-refactor * add todo to revert change Co-authored-by: Chris Cotter <cjcotter@google.com> Co-authored-by: cojenco <cathyo@google.com>
* chore(storage): migrate GetBucket and DeleteBucket * migrate remaining bucket methods * fix merge conflict Co-authored-by: Chris Cotter <cjcotter@google.com>
Migrate Object metadata operations to transport-agnostic interface. Integration tests pass locally - GetObject - UpdateObject - DeleteObject
Migrate Writer to use the transport-agnostic interface. A few minor fixes: * Pass in retry, idempotency and user project via settings rather than determining below interface. * Use default generation since this can't be sent for this call.
Migrate the Reader to use the new transport-agnostic interface. A few minor fixes were required: * Move validations above interface * Fix up settings for call * Add param for sending with accept-encoding: gzip (HTTP only) * Fix up reader resumption test
product-auto-label
bot
added
size: xl
noahdietz
changed the title
noahdietz
commented
Aug 2, 2022
tritone
approved these changes
Aug 10, 2022
noahdietz
added
the
automerge
gcf-merge-on-green
bot
removed
the
automerge
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.