spanner: refactor retrying

[jeanbza]

Spanner should:

• Use github.com/googleapis/gax-go/v2 for retrying, replacing all the custom logic in runRetryable and runRetryableNoWrap.
• Retry on codes.Aborted only, removing entirely our transport is closing, streaming terminated by RST_STREAM, and unexpected EOF retries.
  • These should happen approximately never, and when they do happen are probably indicative of some critical failure that we should not retry.
  • These may have been returned by grpc-go when it was younger and buggier. It seems as though a lot of the retry logic in spanner was implemented pre- grpc-go 1.9. (grpc-go is currently at 1.20.1 by comparison)
• Leave almost all retrying to gapic layer (including UNAVAILABLE and UNKNOWN). The only retrying that should happen in the manual client is retries around transactions in which the retry block (runInTransaction) encompasses all transactional RPCs, and ABORTED causes the entire transaction to be rolled back and retried.
  • One likely exception to this is StreamingRead, which seems to (correctly) have retry logic around resumableStreamDecoder. This retry logic should probably be replaced with something like a gax.OnCodes(ABORTED, UNAVAILABLE, UNKNOWN) rather than doing retry counting and and so on.
  • It's not 100% clear to me whether UNKNOWN and UNAVAILABLE should cause the entire transaction to be rolled back, but from the reading of ABORTED documentation and spanner.proto I believe these are safe to retry per-RPC, whereas ABORTED should retry the entire transaction.
• Remove the assignment of gRPC codes to errors, as seen for example here and here. The only acceptable place to generate gRPC errors is gRPC.
  • This might need to be a separate errors epic.

[jeanbza]

cc @broady @olavloite

[jeanbza]

Made some progress on this in https://code-review.googlesource.com/c/gocloud/+/40790, but not sure if I have the time to continue. Thought I'd record my thoughts in the meantime if someone else has more time.

[olavloite]

Reopening as there is still some work to do on this.

[olavloite]

Custom retry loops have been replaced with standard GAX retryers where applicable. All other retries are handled by GAPIC. As noted above: Removing/refactoring the assignment of gRPC codes to errors should be handled in a separate issue.

[apstndb]

Retry on codes.Aborted only, removing entirely our transport is closing, streaming terminated by RST_STREAM, and unexpected EOF retries.

It seems that the retry on RST_STREAM has been restored. Now, is it considered safe to retry?
#6699