Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

internal/gensupport: SendRequestWithRetry sometimes spuriously makes requests when its context was canceled #1358

Closed
lhchavez opened this issue Dec 21, 2021 · 0 comments · Fixed by #1359
Assignees
Labels
priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.

Comments

@lhchavez
Copy link
Contributor

lhchavez commented Dec 21, 2021

Environment details

  • Programming language: Go (v1.7+)
  • OS: Any
  • Language runtime version: v1.7+
  • Package version: v0.61.0

Steps to reproduce

  1. Call SendRequestWithRetry with a canceled context
  2. The underlying network call sometimes still happens. This is due to Go legally being allowed to visit either of the branches in the following snippet of code if both are satisfied:
    select {
    case <-ctx.Done():
    // If we got an error, and the context has been canceled,
    // the context's error is probably more useful.
    if err == nil {
    err = ctx.Err()
    }
    return resp, err
    case <-time.After(pause):
    }

Checking the context being done afterwards makes the issue go away.

Motivation

Some Cloud APIs expect callers to cancel the context if they want to cancel the operation, like
storage.(*ObjectHandle).NewWriter. The spuriousness of SendRequestWithRetry sometimes causes spooky things to happen, like uploading a 0-byte object to GCS, where no object was desired or expected!

@lhchavez lhchavez added priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. labels Dec 21, 2021
lhchavez added a commit to lhchavez/google-api-go-client that referenced this issue Dec 21, 2021
This change makes it deterministic to call SendRequestWithRetry with a
canceled context. This is going to be useful because some Cloud APIs
rely on the context being canceled to prevent some operations from
proceeding, like
[`storage.(*ObjectHandle).NewWriter`](https://pkg.go.dev/cloud.google.com/go/storage#ObjectHandle.NewWriter).

Fixes: googleapis#1358
codyoss pushed a commit that referenced this issue Dec 21, 2021
…d contexts twice (#1359)

This change makes it deterministic to call SendRequestWithRetry with a
canceled context. This is going to be useful because some Cloud APIs
rely on the context being canceled to prevent some operations from
proceeding, like
[`storage.(*ObjectHandle).NewWriter`](https://pkg.go.dev/cloud.google.com/go/storage#ObjectHandle.NewWriter).

Fixes: #1358
tritone added a commit to tritone/google-api-go-client that referenced this issue Dec 22, 2021
The select statement is non-deterministic, so currently, if the
pause is completed and ALSO the context has been canceled or
timeout elapsed, a request may still occur. This PR prevents that
circumstance from occurring.

Also removed something in a test that seems to be a workaround for
the race condition.

Inspired by googleapis#1359 & googleapis#1358.
tritone added a commit that referenced this issue Jan 6, 2022
The select statement is non-deterministic, so currently, if the
pause is completed and ALSO the context has been canceled or
timeout elapsed, a request may still occur. This PR prevents that
circumstance from occurring.

Also removed something in a test that seems to be a workaround for
the race condition.

Inspired by #1359 & #1358.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Projects
None yet
3 participants