grpcurl not able to connect to secondary IPs in FQDN if primary is not reachable

[deep27ak]

Hi,

We have an FQDN which resolves to two IPs for High Availability. The client needs to check the reachable IP and attempt to connect with the same.

In our script we first use nc command to make sure at least 1 IP in the FQDN is reachable

$ nc some.fqdn.svc.cluster.local 7055 -v -z -w2
Ncat: Version 7.70 ( https://nmap.org/ncat )
Ncat: Connection to 192.106.167.129 failed: Connection timed out.
Ncat: Trying next address...
Ncat: Connected to 192.106.226.1:7055.
Ncat: 0 bytes sent, 0 bytes received in 2.30 seconds.

and then execute grpcurl with the FQDN. But it fails to establish connection sporadically which I am assuming is because it tries to connect to an IP which is not reachable and when we execute grpcurl again then it works.

I see there are two observations (not sure if these are issues or expected behavior):

1. -v or -vv is not printing the IP to which grpcurl is connecting to when using FQDN
2. Automatic lookup and attempting connection of all available IPs inside FQDN is not happenening

When using -v or -vv we get output similar to below (which doesn't contain IP used)

$ ./grpcurl -vv -cacert /certs/cacert.pem -cert /certs/cert.pem -key /certs/key.pem -proto secretstore.proto -d '{"vnfid": "DEMO", "relver":"1","mulappid": "secrets","netype":"ABC"}' [some.fqdn.svc.cluster.local]:7055 <API>

Resolved method descriptor:
rpc Watch ( <API> ) returns ( stream <API> );

Request metadata to send:
(empty)

Response headers received:
content-type: application/grpc
date: Tue, 04 Apr 2023 07:07:00 GMT
server: envoy-ingress-ss
x-envoy-upstream-service-time: 99

Estimated response size: 2698 bytes

Response contents:
...

and the connection fails with this error message (when I assume the connection is attempted using non-reachable IP)
Failed to dial target host "some.fqdn.svc.cluster.local:7055": context deadline exceeded

Platform Details

$ ./grpcurl -version
grpcurl v1.8.7

$ cat /etc/os-release 
NAME="Rocky Linux"
VERSION="8.7 (Green Obsidian)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="8.7"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Rocky Linux 8.7 (Green Obsidian)"
ANSI_COLOR="0;32"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:rocky:rocky:8:GA"
HOME_URL="https://rockylinux.org/"
BUG_REPORT_URL="https://bugs.rockylinux.org/"
ROCKY_SUPPORT_PRODUCT="Rocky-Linux-8"
ROCKY_SUPPORT_PRODUCT_VERSION="8.7"
REDHAT_SUPPORT_PRODUCT="Rocky Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.7"