Multiple GOAWAY frames on graceful shutdown leads to crash when Swift clients are connected

[karsten42]

What version of gRPC are you using?

google.golang.org/grpc v1.50.1

What version of Go are you using (go version)?

go 1.19

What operating system (Linux, Windows, …) and version?

Linux

What did you do?

NOTE: Copied from an issue for the swift client since we don't know what is the correct behaviour: grpc/grpc-swift#1520

We use a server implemented with grpc-go This server may gracefully shutdown the connection before shutting down itself. In the repository of grpc-go it is documented that the server sends two GOAWAY frames to circumvent a http2 race condition:

'''
// A client can receive multiple GoAways from the server (see
// #1387). The idea is that the first
// GoAway will be sent with an ID of MaxInt32 and the second GoAway will be
// sent after an RTT delay with the ID of the last stream the server will
// process.
'''

The grpc-swift-client responds to both GOAWAY frames with GOAWAY frames which leads the grpc-go-server to crash. E.g. the client implementation of grpc-go checks for already received GOAWAY frames and their stream ids which prevents two following GOAWAY frames by the client (Implementation).

Since the behavior of the grpc-swift-client causes other servers to crash we expect grpc-swift to change the behavior.

To reproduce
Steps to reproduce the bug you've found:

1. Initiate connection
2. Make a bidirectional RPC to the server (grpc-go)
3. Invoke gracefullShutdown on the server
4. Clients sends two GOAWAY frames
5. Server crashes due to close of closed channel

We could reproduce the behaviour utilising the client from RouteGuide-Example and implementing a go-server using the example proto file.

What did you expect to see?

The server doesn't crash.

What did you see instead?

The server crashes due to closing a channel twice.

[zasweq]

Working on reproducing this now. Do you happen to have a stack trace of the crash?

[karsten42]

Yes, it's exactly the stack trace you find in the Swift issue.

[dfawley]

Our side fixed by #5854