Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(bigquery/storage/managedwriter): support append retries #6695

Merged
merged 19 commits into from Sep 27, 2022

Conversation

shollyman
Copy link
Contributor

@shollyman shollyman commented Sep 19, 2022

This PR adds support for automatic retry of failed appends. Failures are evaluated both from the perspective of any receive errors getting the response, as well as any response that may be embedded in response from the service. Previous behavior was to simply re-attempt failures when issuing the append.

This PR also adds a new WriterOption (DisableWriteRetries(disable bool)) to control this behavior (default is to have retries enabled).

For users sensitive to write duplication, this PR also exposes a new TotalAttempts method on the AppendResult, which will indicate the total number of times this write was attempted.

This also tries to clean up retries in general a bit more. The generated client will already retry unary RPCs, subject to the service config present when generating the storage API.

We specifically clarify/introduce two additional kinds of retries above that: a unary retry and a stateless retry.

The unary retry is used to (re)open the underlying bidi network connection which appends are sent upon, as we want to be resilient to reconnection. The unary retry is effectively stateful for the operation of reopening the connection, and thus uses a gax-based backoff that backs off with increasing intervals.

The stateless retry is used when processing the responses returning from the backend on the bidi stream, where backing off incrementally doesn't make sense. Instead, we use a base backoff and jitter, and for cases where a more severe backoff is warranted (throughput exhaustion) we use a multiplication factor. The intent here is to provide backpressure which will eventually saturate the append queue and cause blocking/rejection of writes until the backlog recovers.

@product-auto-label product-auto-label bot added size: s Pull request size is small. api: bigquery Issues related to the BigQuery API. labels Sep 19, 2022
@product-auto-label product-auto-label bot added size: l Pull request size is large. and removed size: s Pull request size is small. labels Sep 21, 2022
@shollyman shollyman marked this pull request as ready for review September 22, 2022 17:20
@shollyman shollyman requested review from a team September 22, 2022 17:20
@shollyman shollyman requested review from a team as code owners September 22, 2022 17:20
@shollyman shollyman requested review from chalmerlowe, prash-mi, yirutang and alvarowolfx and removed request for chalmerlowe and prash-mi September 22, 2022 17:20
@shollyman shollyman added the automerge Merge the pull request once unit tests and other checks pass. label Sep 27, 2022
@gcf-merge-on-green gcf-merge-on-green bot merged commit 6ae9c67 into googleapis:main Sep 27, 2022
@gcf-merge-on-green gcf-merge-on-green bot removed the automerge Merge the pull request once unit tests and other checks pass. label Sep 27, 2022
// Any other non-status based errors treated as retryable.
return r.bo.Pause(), true
// All other non-status errors are treated as non-retryable (including context errors).
return
}
switch s.Code() {
case codes.Aborted,
codes.Canceled,
codes.DeadlineExceeded,
codes.Internal,
codes.Unavailable:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, missed this, please retry on failed_precondition.

kimihrr pushed a commit to kimihrr/google-cloud-go-kimia that referenced this pull request Sep 28, 2022
…pis#6695)

This PR adds support for automatic retry of failed appends.  Failures are evaluated both from the perspective of any receive errors getting the response, as well as any response that may be embedded in response from the service.  Previous behavior was to simply re-attempt failures when issuing the append.

This PR also adds a new WriterOption (`DisableWriteRetries(disable bool)`) to control this behavior (default is to have retries enabled).

For users sensitive to write duplication, this PR also exposes a new TotalAttempts method on the AppendResult, which will indicate the total number of times this write was attempted.

This also tries to clean up retries in general a bit more.  The generated client will already retry unary RPCs, subject to the [service config](https://github.com/googleapis/googleapis/blob/master/google/cloud/bigquery/storage/v1/bigquerystorage_grpc_service_config.json) present when generating the storage API.

We specifically clarify/introduce two additional kinds of retries above that: a unary retry and a stateless retry.

The unary retry is used to (re)open the underlying bidi network connection which appends are sent upon, as we want to be resilient to reconnection.  The unary retry is effectively stateful for the operation of reopening the connection, and thus uses a gax-based backoff that backs off with increasing intervals.

The stateless retry is used when processing the responses returning from the backend on the bidi stream, where backing off incrementally doesn't make sense.  Instead, we use a base backoff and jitter, and for cases where a more severe backoff is warranted (throughput exhaustion) we use a multiplication factor.  The intent here is to provide backpressure which will eventually saturate the append queue and cause blocking/rejection of writes until the backlog recovers.
gcf-merge-on-green bot pushed a commit that referenced this pull request Oct 24, 2022
🤖 I have created a release *beep* *boop*
---


## [1.43.0](https://togithub.com/googleapis/google-cloud-go/compare/bigquery/v1.42.0...bigquery/v1.43.0) (2022-10-24)


### Features

* **bigquery/analyticshub:** rename nodejs analyticshub library package name ([52dddd1](https://togithub.com/googleapis/google-cloud-go/commit/52dddd1ed89fbe77e1859311c3b993a77a82bfc7))
* **bigquery/storage/managedwriter:** Enable field name indirection ([#6247](https://togithub.com/googleapis/google-cloud-go/issues/6247)) ([1969273](https://togithub.com/googleapis/google-cloud-go/commit/19692735b0fbafa176d0315bda923528e1eedf6d))
* **bigquery/storage/managedwriter:** Retry on FailedPrecondition ([#6761](https://togithub.com/googleapis/google-cloud-go/issues/6761)) ([d1a444d](https://togithub.com/googleapis/google-cloud-go/commit/d1a444d769c9578b586bef608d343b4b0abd3658))
* **bigquery/storage/managedwriter:** Support append retries ([#6695](https://togithub.com/googleapis/google-cloud-go/issues/6695)) ([6ae9c67](https://togithub.com/googleapis/google-cloud-go/commit/6ae9c670a11d80b34872cb05fda933303b81851d))
* **bigquery/storage/managedwriter:** Switch to opt-in retry ([#6765](https://togithub.com/googleapis/google-cloud-go/issues/6765)) ([a3e97a6](https://togithub.com/googleapis/google-cloud-go/commit/a3e97a6f15ad1989ef815b9bd5838192f9f226f1))
* **bigquery:** Add remote function options to routine metadata ([#6702](https://togithub.com/googleapis/google-cloud-go/issues/6702)) ([d9a437d](https://togithub.com/googleapis/google-cloud-go/commit/d9a437de75a5f5151cd000d8f9a6b7fc567d8551))


### Bug Fixes

* **bigquery/storage/managedwriter:** Address possible resource leak ([#6775](https://togithub.com/googleapis/google-cloud-go/issues/6775)) ([979440b](https://togithub.com/googleapis/google-cloud-go/commit/979440b43573c1cfd744d3f63d0c633c7319ad46))
* **bigquery:** Avoid stack overflow on query param with recursive types ([#6890](https://togithub.com/googleapis/google-cloud-go/issues/6890)) ([854ccfc](https://togithub.com/googleapis/google-cloud-go/commit/854ccfca259d747759d283fc0d0053893f3c8f8d)), refs [#6884](https://togithub.com/googleapis/google-cloud-go/issues/6884)
* **bigquery:** Bq connection auth scopes ([#6752](https://togithub.com/googleapis/google-cloud-go/issues/6752)) ([8e09288](https://togithub.com/googleapis/google-cloud-go/commit/8e09288185f721d90288c3aa873980fc44b98613)), refs [#6744](https://togithub.com/googleapis/google-cloud-go/issues/6744)


### Documentation

* **bigquery/storage/managedwriter/adapt:** Typo in error string ([#6729](https://togithub.com/googleapis/google-cloud-go/issues/6729)) ([bb26153](https://togithub.com/googleapis/google-cloud-go/commit/bb26153d38475cd9784edbf241df84c368f5a166))
* **bigquery/storage/managedwriter:** Add retry info to package docs ([#6803](https://togithub.com/googleapis/google-cloud-go/issues/6803)) ([81e52e5](https://togithub.com/googleapis/google-cloud-go/commit/81e52e59dcf3c4a44108e039fb0e3a0e2ce8284f))

---
This PR was generated with [Release Please](https://togithub.com/googleapis/release-please). See [documentation](https://togithub.com/googleapis/release-please#release-please).
kimihrr pushed a commit to kimihrr/google-cloud-go-kimia that referenced this pull request Oct 25, 2022
🤖 I have created a release *beep* *boop*
---


## [1.43.0](https://togithub.com/googleapis/google-cloud-go/compare/bigquery/v1.42.0...bigquery/v1.43.0) (2022-10-24)


### Features

* **bigquery/analyticshub:** rename nodejs analyticshub library package name ([52dddd1](https://togithub.com/googleapis/google-cloud-go/commit/52dddd1ed89fbe77e1859311c3b993a77a82bfc7))
* **bigquery/storage/managedwriter:** Enable field name indirection ([googleapis#6247](https://togithub.com/googleapis/google-cloud-go/issues/6247)) ([1969273](https://togithub.com/googleapis/google-cloud-go/commit/19692735b0fbafa176d0315bda923528e1eedf6d))
* **bigquery/storage/managedwriter:** Retry on FailedPrecondition ([googleapis#6761](https://togithub.com/googleapis/google-cloud-go/issues/6761)) ([d1a444d](https://togithub.com/googleapis/google-cloud-go/commit/d1a444d769c9578b586bef608d343b4b0abd3658))
* **bigquery/storage/managedwriter:** Support append retries ([googleapis#6695](https://togithub.com/googleapis/google-cloud-go/issues/6695)) ([6ae9c67](https://togithub.com/googleapis/google-cloud-go/commit/6ae9c670a11d80b34872cb05fda933303b81851d))
* **bigquery/storage/managedwriter:** Switch to opt-in retry ([googleapis#6765](https://togithub.com/googleapis/google-cloud-go/issues/6765)) ([a3e97a6](https://togithub.com/googleapis/google-cloud-go/commit/a3e97a6f15ad1989ef815b9bd5838192f9f226f1))
* **bigquery:** Add remote function options to routine metadata ([googleapis#6702](https://togithub.com/googleapis/google-cloud-go/issues/6702)) ([d9a437d](https://togithub.com/googleapis/google-cloud-go/commit/d9a437de75a5f5151cd000d8f9a6b7fc567d8551))


### Bug Fixes

* **bigquery/storage/managedwriter:** Address possible resource leak ([googleapis#6775](https://togithub.com/googleapis/google-cloud-go/issues/6775)) ([979440b](https://togithub.com/googleapis/google-cloud-go/commit/979440b43573c1cfd744d3f63d0c633c7319ad46))
* **bigquery:** Avoid stack overflow on query param with recursive types ([googleapis#6890](https://togithub.com/googleapis/google-cloud-go/issues/6890)) ([854ccfc](https://togithub.com/googleapis/google-cloud-go/commit/854ccfca259d747759d283fc0d0053893f3c8f8d)), refs [googleapis#6884](https://togithub.com/googleapis/google-cloud-go/issues/6884)
* **bigquery:** Bq connection auth scopes ([googleapis#6752](https://togithub.com/googleapis/google-cloud-go/issues/6752)) ([8e09288](https://togithub.com/googleapis/google-cloud-go/commit/8e09288185f721d90288c3aa873980fc44b98613)), refs [googleapis#6744](https://togithub.com/googleapis/google-cloud-go/issues/6744)


### Documentation

* **bigquery/storage/managedwriter/adapt:** Typo in error string ([googleapis#6729](https://togithub.com/googleapis/google-cloud-go/issues/6729)) ([bb26153](https://togithub.com/googleapis/google-cloud-go/commit/bb26153d38475cd9784edbf241df84c368f5a166))
* **bigquery/storage/managedwriter:** Add retry info to package docs ([googleapis#6803](https://togithub.com/googleapis/google-cloud-go/issues/6803)) ([81e52e5](https://togithub.com/googleapis/google-cloud-go/commit/81e52e59dcf3c4a44108e039fb0e3a0e2ce8284f))

---
This PR was generated with [Release Please](https://togithub.com/googleapis/release-please). See [documentation](https://togithub.com/googleapis/release-please#release-please).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: bigquery Issues related to the BigQuery API. size: l Pull request size is large.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants