GRPC hang after timeout

[biran0083]

Bug Report

Version

0.10.2

Platform

Linux bir 6.2.0-1018-aws #18~22.04.1-Ubuntu SMP Wed Jan 10 22:54:16 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Description

Both server and client are using tonic 0.10.2.
Both run in k8s pods.
When server restarts, client starts getting

status: Unavailable, message: "error trying to connect: dns error: failed to lookup address information: Name or service not known", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Tue, 13 Feb 2024 16:39:13 GMT", "content-length": "0"} }�

Client retries the request, and RPC hangs for 1 min+. Then we killed the client process.
This is not expected because we set both timeout and connection_timeout to 10 sec when creating client channel.