Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(storage): refactor to use transport-agnostic interface #6465

Merged
merged 18 commits into from Aug 10, 2022

Commits on Jul 26, 2022

  1. chore(storage): integrate Compose in new interface (#6414)

    Change production code paths for Composer to use the new transport
    agnostic interface.
    
    Required a few setup steps to make tests pass, which will be needed for
    other integrations too:
    * Add initialization of tc to NewClient
    * Add new useGRPC flag to remove switches on tc in Reader/Writer
    
    Also a few cleanups:
    * Use EncryptionKey from dstObject
    * Use defaultGen instead of hard coded int
    
    Also, a unit test checks that setting Generation with composer causes
    a failure. Currently this is enforced by applyConds. I added
    a separate validation for this instead, but I could do a pass-
    through of the Generation through the interface -- it's just
    confusing looking because setting Generation on a new object
    is never valid.
    tritone committed Jul 26, 2022
    Copy the full SHA
    be23ac8 View commit details
    Browse the repository at this point in the history
  2. chore(storage): add makeStorageOpts helper (#6426)

    This takes out a few lines of boilerplate that will be
    present in all top-level methods.
    tritone committed Jul 26, 2022
    Copy the full SHA
    6cf62f8 View commit details
    Browse the repository at this point in the history
  3. Copy the full SHA
    6ad9f3f View commit details
    Browse the repository at this point in the history

Commits on Jul 27, 2022

  1. chore(storage): migrate Copier (#6435)

    Migrate Copier to use the new transport-agnostic interface.
    I found one additional piece of data that needs to be passed back
    through the interface in the process (object size).
    
    I removed one integration test. This test is old and currently
    calls rewriteObject in a way that an external library user could
    not accomplish. I don't think this test adds any value.
    tritone committed Jul 27, 2022
    Copy the full SHA
    46f968a View commit details
    Browse the repository at this point in the history
  2. chore(storage): migrate top-level methods (#6433)

    * chore(storage): migrate GetServiceAccount
    
    * migrate ListBuckets
    
    * migrate CreateBucket and refactor interface method
    
    * update tests and BucketIterator with retryConfig
    
    * Revert "update tests and BucketIterator with retryConfig"
    
    This reverts commit b7662b4.
    
    * remove unused field and internal method upon migration
    cojenco committed Jul 27, 2022
    Copy the full SHA
    3a14187 View commit details
    Browse the repository at this point in the history

Commits on Jul 28, 2022

  1. chore(storage): migrate IAM operations (#6437)

    * chore(storage): migrate IAM operations
    
    * remove unused field
    cojenco committed Jul 28, 2022
    Copy the full SHA
    5df3a32 View commit details
    Browse the repository at this point in the history
  2. chore(storage): migrate ACL operations (#6440)

    Migrate ACL ops to the new transport agnostic interface. No
    interface changes were necessary for this one and tests pass.
    tritone committed Jul 28, 2022
    Copy the full SHA
    52985ce View commit details
    Browse the repository at this point in the history
  3. chore(storage): migrate Notification methods (#6442)

    Migrates the Notification APIs to the transport agnostic surface. Integration tests passed and added a check to ensure that AddNotification returns a Notification with a non-empty ID.
    noahdietz committed Jul 28, 2022
    Copy the full SHA
    76bf3fa View commit details
    Browse the repository at this point in the history

Commits on Jul 29, 2022

  1. test: update continuous.sh for storage-refactor (#6443)

    * test: update continuous.sh for storage-refactor
    
    * add todo to revert change
    
    Co-authored-by: Chris Cotter <cjcotter@google.com>
    Co-authored-by: cojenco <cathyo@google.com>
    3 people committed Jul 29, 2022
    Copy the full SHA
    996daa6 View commit details
    Browse the repository at this point in the history
  2. chore(storage): migrate bucket methods (#6438)

    * chore(storage): migrate GetBucket and DeleteBucket
    
    * migrate remaining bucket methods
    
    * fix merge conflict
    
    Co-authored-by: Chris Cotter <cjcotter@google.com>
    cojenco and tritone committed Jul 29, 2022
    Copy the full SHA
    4fa01ba View commit details
    Browse the repository at this point in the history
  3. chore(storage): migrate object metadata operations (#6445)

    Migrate Object metadata operations to transport-agnostic interface. Integration tests pass locally
    - GetObject
    - UpdateObject
    - DeleteObject
    cojenco committed Jul 29, 2022
    Copy the full SHA
    7bfd4ec View commit details
    Browse the repository at this point in the history

Commits on Aug 1, 2022

  1. Copy the full SHA
    5ee0de3 View commit details
    Browse the repository at this point in the history
  2. chore(storage): migrate Writer (#6452)

    Migrate Writer to use the transport-agnostic interface. A few
    minor fixes:
    
    * Pass in retry, idempotency and user project via settings rather
    than determining below interface.
    * Use default generation since this can't be sent for this call.
    tritone committed Aug 1, 2022
    Copy the full SHA
    728ab63 View commit details
    Browse the repository at this point in the history

Commits on Aug 2, 2022

  1. chore(storage): migrate Reader (#6457)

    Migrate the Reader to use the new transport-agnostic interface.
    A few minor fixes were required:
    * Move validations above interface
    * Fix up settings for call
    * Add param for sending with accept-encoding: gzip (HTTP only)
    * Fix up reader resumption test
    tritone committed Aug 2, 2022
    Copy the full SHA
    a29cb5d View commit details
    Browse the repository at this point in the history
  2. Copy the full SHA
    875fd0f View commit details
    Browse the repository at this point in the history

Commits on Aug 8, 2022

  1. Copy the full SHA
    669450f View commit details
    Browse the repository at this point in the history

Commits on Aug 10, 2022

  1. Copy the full SHA
    0c10144 View commit details
    Browse the repository at this point in the history
  2. Copy the full SHA
    9757864 View commit details
    Browse the repository at this point in the history