feat(internal/gensupport): wrap retry failures with context and prev error #1684

matheuscscp · 2022-09-02T14:34:40Z

I have a use case in which my context is cancelling a sendAndRetry() operation, and the error being returned is the error returned by the Google API in the previous attempt of sendAndRetry() instead of ctx.Err(), which is misleading because I was investigating why the Google Storage Go library was giving up on a 503 error (which is not expected according to the retry/idempotency docs: https://cloud.google.com/storage/docs/retry-strategy).

closes: #1685

google-cla · 2022-09-02T14:34:45Z

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

codyoss · 2022-09-02T14:40:53Z

@matheuscscp could you please file an issue for discussion before I take a look at the PR. Thanks

matheuscscp · 2022-09-02T16:09:43Z

@matheuscscp could you please file an issue for discussion before I take a look at the PR. Thanks

@codyoss here it is: #1685

sorry for not starting with that!

quartzmo · 2022-09-06T19:49:22Z

internal/gensupport/send.go

@@ -98,10 +98,11 @@ func sendAndRetry(ctx context.Context, client *http.Client, req *http.Request, r
 		case <-ctx.Done():
 			// If we got an error, and the context has been canceled,
 			// the context's error is probably more useful.


Would another possible solution to #1685 be simply to remove this entire block, and let the context raise the deadline exceeded error on its own?

do you mean just return nil, ctx.Err()? otherwise, where would you suggest to put something equivalent to case <-time.After(pause): (the backoff)?

@tritone Can you comment on this?

Hey @quartzmo , thanks for flagging. I think the right answer is to return a wrapped error that includes both the context cancelation and the final error from the service, if available -- otherwise it can be pretty hard to debug either way. We implemented this in google-cloud-go: https://github.com/googleapis/google-cloud-go/blob/0da9ab0831540569dc04c0a23437b084b1564e15/internal/retry.go#L36

I can make a PR to implement this if folks agree.

(Also I see that this PR already basically takes that approach -- happy to work with you on completing it yourself if you want; the main change I would ask for is to implement a custom error type that implements Is(error) bool so that we can preserve the validity of calls to errors.Is in user code.

yep I'd like to fix that as you suggested, I will do it tomorrow

sounds good, thank you!

it's there, PTAL. I basically copied the wrappedCallErr struct from your other repo with the exception that I did Is() like this:

func (e wrappedCallErr) Is(target error) bool { return errors.Is(e.ctxErr, target) || errors.Is(e.wrappedErr, target) }

basically leveraging errors.Is() instead of using plain == comparison. if that's too much pls let me know!

I also seized the opportunity to use this wrapper in all the other occurrences of some ctx+svc error inside this file

tritone

Generally looks good, aside from change to send which I think we should hold off on.

tritone · 2022-09-07T16:10:11Z

internal/gensupport/send.go

@@ -98,10 +119,10 @@ func sendAndRetry(ctx context.Context, client *http.Client, req *http.Request, r
 		case <-ctx.Done():
 			// If we got an error, and the context has been canceled,


This comment is no longer accurate, I would rephrase to something like:

// If we got an error and the context has been canceled, return an error acknowledging // both the context cancelation and the service error.

tritone · 2022-09-07T16:52:08Z

internal/gensupport/send.go

@@ -43,7 +64,7 @@ func send(ctx context.Context, client *http.Client, req *http.Request) (*http.Re
 	if err != nil {
 		select {
 		case <-ctx.Done():
-			err = ctx.Err()
+			err = wrappedCallErr{ctx.Err(), err}


Let's not make this change in send() for now. Reasons:

The case that the call returned a non-context error but the context is expired/canceled is not a very common case without retries being in the picture. And the string from wrappedCallErr is specific to retries being at the root of the issue.

send is much more widely used than sendAndRetry including by the manual clients built on top of this library. I would want to audit these to make sure they can handle this error (meaning that they are upgraded to use errors.Is rather than checking for equality) if they have error handling logic at that layer.

tritone

LGTM. @codyoss could you also give this a look before we merge?

codyoss

LGTM

matheuscscp requested review from a team and yoshi-approver as code owners September 2, 2022 14:34

quartzmo reviewed Sep 6, 2022

View reviewed changes

tritone reviewed Sep 7, 2022

View reviewed changes

tritone added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Sep 7, 2022

kokoro-team removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Sep 7, 2022

matheuscscp force-pushed the dont-swallow-context-error branch from c0bed43 to a61f463 Compare September 8, 2022 10:47

codyoss added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Sep 8, 2022

kokoro-team removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Sep 8, 2022

matheuscscp force-pushed the dont-swallow-context-error branch from a61f463 to c2ec69f Compare September 12, 2022 21:07

tritone approved these changes Sep 12, 2022

View reviewed changes

codyoss approved these changes Sep 19, 2022

View reviewed changes

codyoss changed the title ~~bugfix: do not swallow context error when context is done~~ feat(internal/gensupport): do not swallow context error when context is done Sep 19, 2022

codyoss changed the title ~~feat(internal/gensupport): do not swallow context error when context is done~~ feat(internal/gensupport): wrap retry failures with context and prev error Sep 19, 2022

tritone added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Sep 19, 2022

kokoro-team removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Sep 19, 2022

matheuscscp added 4 commits September 19, 2022 23:00

bugfix: do not swallow context error when context is done

7fa2b39

bugfix: wrap both ctx and service call errors

78006b1

fix comment as suggested by @tritone

a247f2d

hold send() change as suggested by @tritone

9a8dddd

matheuscscp force-pushed the dont-swallow-context-error branch from 65c3967 to 9a8dddd Compare September 19, 2022 22:00

tritone added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Sep 19, 2022

kokoro-team removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Sep 19, 2022

tritone merged commit f427ee3 into googleapis:main Sep 20, 2022

release-please bot mentioned this pull request Sep 20, 2022

chore(main): release 0.97.0 #1694

Merged

codyoss mentioned this pull request Oct 18, 2022

all: do not hide original error code googleapis/gax-go#235

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(internal/gensupport): wrap retry failures with context and prev error #1684

feat(internal/gensupport): wrap retry failures with context and prev error #1684

matheuscscp commented Sep 2, 2022 •

edited by quartzmo

google-cla bot commented Sep 2, 2022

codyoss commented Sep 2, 2022

matheuscscp commented Sep 2, 2022

quartzmo Sep 6, 2022

matheuscscp Sep 6, 2022

quartzmo Sep 6, 2022

tritone Sep 6, 2022

tritone Sep 6, 2022

matheuscscp Sep 6, 2022

tritone Sep 7, 2022

matheuscscp Sep 7, 2022 •

edited

tritone left a comment

tritone Sep 7, 2022

matheuscscp Sep 7, 2022

tritone Sep 7, 2022

matheuscscp Sep 7, 2022

tritone left a comment

codyoss left a comment

		@@ -98,10 +119,10 @@ func sendAndRetry(ctx context.Context, client http.Client, req http.Request, r
		case <-ctx.Done():
		// If we got an error, and the context has been canceled,

feat(internal/gensupport): wrap retry failures with context and prev error #1684

feat(internal/gensupport): wrap retry failures with context and prev error #1684

Conversation

matheuscscp commented Sep 2, 2022 • edited by quartzmo

google-cla bot commented Sep 2, 2022

codyoss commented Sep 2, 2022

matheuscscp commented Sep 2, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

matheuscscp Sep 7, 2022 • edited

Choose a reason for hiding this comment

tritone left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tritone left a comment

Choose a reason for hiding this comment

codyoss left a comment

Choose a reason for hiding this comment

matheuscscp commented Sep 2, 2022 •

edited by quartzmo

matheuscscp Sep 7, 2022 •

edited