Data race in http2Client

[knz]

What version of gRPC are you using?

1.39
(the bug was not present in 1.38)

What version of Go are you using (go version)?

1.16.5

What operating system (Linux, Windows, …) and version?

linux

What did you do?

1. Upgraded from gRPC 1.38 to 1.39
2. Ran unit tests in CockroachDB with the go data race detector enabled.

What did you expect to see?

No races (There were no races when using gRPC 1.38)

What did you see instead?

WARNING: DATA RACE
Read at 0x00c0035a4ba0 by goroutine 827:
  google.golang.org/grpc/internal/transport.(*Stream).RecvCompress()
      /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:338 +0xf0a
  google.golang.org/grpc.(*csAttempt).recvMsg()
      /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:943 +0xeca
  google.golang.org/grpc.(*clientStream).RecvMsg.func1()
      /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:808 +0x64
  google.golang.org/grpc.(*clientStream).withRetry()
      /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:666 +0xe6
  google.golang.org/grpc.(*clientStream).RecvMsg()
      /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:807 +0x164
  google.golang.org/grpc.invoke()
      /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/call.go:73 +0x1a2
  github.com/cockroachdb/cockroach/pkg/util/tracing.ClientInterceptor.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/tracing/grpc_interceptor.go:259 +0x150
  google.golang.org/grpc.(*ClientConn).Invoke()
      /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/call.go:35 +0x1a6
  github.com/cockroachdb/cockroach/pkg/server/serverpb.(*statusClient).ListLocalSessions()
      /go/src/github.com/cockroachdb/cockroach/pkg/server/serverpb/status.pb.go:5008 +0x10a
  github.com/cockroachdb/cockroach/pkg/server.(*statusServer).listSessionsHelper.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/server/status.go:2178 +0xd0
  github.com/cockroachdb/cockroach/pkg/server.(*rpcNodePaginator).queryNode()
      /go/src/github.com/cockroachdb/cockroach/pkg/server/pagination.go:381 +0x1ea
  github.com/cockroachdb/cockroach/pkg/server.(*statusServer).paginatedIterateNodes.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/server/status.go:2155 +0x65
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
      /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:442 +0x161

Previous write at 0x00c0035a4ba0 by goroutine 777:
  google.golang.org/grpc/internal/transport.(*http2Client).operateHeaders()
      /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1295 +0x2309
  google.golang.org/grpc/internal/transport.(*http2Client).reader()
      /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1484 +0xdb1

[knz]

NB: a similar bug had existed previously in gRPC 1.9.1 (issue #1803), was a regression from v1.7, and was fixed in 1.9.2

[menghanl]

Thanks for the report. This is a regression caused by #3313. Working on a fix.
Fix is in #4627 (for v1.39.x).

[menghanl]

https://github.com/grpc/grpc-go/releases/tag/v1.39.1 published with this fixed

[knz]

Cc @adityamaru you can proceed to attempt the upgrade to 1.39.1