GOAWAY message leads to other Clients/Channels being unresponsive

[konnextv]

Bug Report

Version

• tonic@0.11.0
• tonic-build@0.10.2

Platform

Linux nomad-client-1 5.15.0-102-generic #112-Ubuntu SMP Tue Mar 5 16:49:56 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux

Description

In a scenario where a gRPC Server (in my case hosted with ASP.NET Core) is redeployed and the new address is updated I tried

• switching the Client completely (creating a new client with ::connect and using that on the next request)
• using balance_channel and updating the only endpoint with Change::Insert and MyDataClient::new(Channel)

What I observe is the following:

• the connection to the new address succeeds
• before the next request is made by the application, the old connection receives a GOAWAY
• the "new" connection becomes unresponsive

I suspect the old Client's Transport influences the new Client with its attached Channel. Maybe this is expected behaviour and you can enlighten me. Maybe we are on to something, I can provide more logs if needed - I tried to summarize them above.