No healthy upstream with service mesh and namespace

[sudheerit]

Problem description

Node js grpc call is returning no healthy upstream when the services are configured behind service mesh with namespace.

Reproduction steps

The gRPC services are developed in GoLang, and deployed in Kubernetes cluster in multiple namespaces. These services are accessed through service mesh, redirecting to the respective namespace based on URL.

Example: service-mesh-host/namespace

Node js is failing to connect with no healthy upstream issue. Postman and other tools were able to connect without any issue.

Node js is working when single namespace is there, we could able to route without namespace in URL and hardcoded namespace inside service mesh route.

Environment

• OS name, version and architecture: [macOS Monterey 64 bit]
• Node version [v20.3.1]
• Node installation method [nvm]
• Package name and version [e.g. gRPC@1.8.16]

[murgatroid99]

Are you using @grpc/grpc-js-xds? If so, it would be helpful to know the following:

• What version of @grpc/grpc-js are you using?
• What version of @grpc/grpc-js-xds are you using?
• Do you have environment variables set to enable or disable any features in @grpc/grpc-js-xds?
• What is the exact text of the error you are getting?
• Can you share the output of running the client with the environment variables GRPC_TRACE=all and GRPC_VERBOSITY=DEBUG?

[sunny3774]

@murgatroid99 I work with @sudheerit . Let me answer your questions.
-We are not using @grpc/grpc-js-xds. Are we supposed to use it if we are trying to connect to services under a service mesh? (in our case we were able to connect to the istio endpoint when namepsace was not involved with out using this package)

• @grpc/grpo-is version: 1.8.21
• NA since we aren't using @grpc/grpc-js-xds
  -Error:

When I include the namespace along with the hostname for istio: hostname/namespace

XYZ Client: Request Error: Error: 14 UNAVAILABLE: Name resolution failed for target dns:hostname/namespace

Note:
When we remove the namespace configuration on our istio layer, the client connects and a successful grpc call is made. The issue happens only after the namespace is configured in our istio layer.

Output of the client:

D 2023-12-12T19:14:43.376Z | v1.9.11 1943 | resolving_load_balancer | dns:hostname/namespace IDLE -> IDLE D 2023-12-12T19:14:43.378Z | V1.9.11 1943 | connectivity_state | (1) dns:hostname/namespace IDLE -> IDLE D 2023-12-12T19:14:43.378Z | v1.9.11 1943 | dns_resolver | Resolver constructed for target dns:hostname/namespace D 2023-12-12T19:14:43.380Z | v1.9.11 1943 | channel | (1) dns:hostname/namespace Channel constructed with options {
"grpc.ssl_target_name_override*: "hostname"
}
D 2023-12-12719:14:43.416Z |v1.9.11 1943 | channel_stacktrace | (2) Channel constructed at new InternalChannel (/location/Project/node_modules/@grpc/grpc-js/build/src/internal-channel.js:237:23)
at new Channelimplementation (location/Project/node_modules/@grpc/grpc-js/build/src/channel.js:35:32)
at new Client (/location/Project/node_modules/@grpc/grpc-js/build/src/client.js:65:36)
at new ServiceClientimpl (location/Project/node modules/@grpc/grpc-js/build/src/make-client.js:58:5)
at new TestSrv (location/Project/src/grpc-sry/test-srv.js:23:26)
at Object. <anonymous> (location/Project/src/grpc-srv/test-srv.js:36:24)
at Module._compile (node:internal/modules/cjs/loader: 1267:14)
at Module._extensions.js (node:internal/modules/cis/loader:1321:10)
at Module.load (node:internal/modules/cis/loader: 1125:32) at Module. load (node:internal/modules/cis/loader:965:12)
D 2023-12-12719:14:43.4292 | v1.9.11 1943 | resolving load balancer | dns:hostname IDLE -> IDLE
D 2023-12-12T 19:14/43.429Z | v1.9.11 1943 | connectivity_state | (3) dns:hostnamo IDLE •> IDLE D 2023-12-12719:14:43.429Z | v1.9.11 1943 | dns_resolver | Resolver constructed for target dns:hostname D 2023-12-12T19:14.43.429Z | v1.9.11 1943 | channel | (3) dns:hostname Channel constructed with options {
grpc.ssl_target_name_override: "hostname"
}
D 2023-12-12T 19:14.43.801Z | index | Loading @grpc/grpo-js version 1.8.21 storage dependency
D 2023-12-12719:14:45.548Z | index | Loading @grpc/grpc-js version 1.8.21 dJobName Automation 4933-env
D 2023-12-12T19:14:46.241Z | v1.9.11 1943 | channel | (2) dns:hostname/namespace createResolvingCall [O] methods="/test.service.test.v1.testService/service
D 2023-12-12719:14:46.241Z | v1.9.11 1943 | resolving_call | (0] Created
D 2023-12-12T19:14:46.242Z| v1.9.11 1943 | resolving_call | (0] Deadline: Infinity
D 2023-12-12T19:14:46.242Z. | v1.9.11 1943 | resolving_call | (0] start called
D 2023-12-12T19:14:46.243Z | v1.9.11 1943 | dns_resolver | Looking up DNS hostname hostname/namespace
D 2023-12-12T 19:14:46.243Z | v1.9.11 1943 | resolving_load_balancer | dns:hostname/namespace IDLE •> CONNECTING D 2023-12-12T 19:14/46.243Z | v1.9.11 1943 | connectivity_state | (2) dns:hostname/namespace IDLE -> CONNECTING D 2023-12-12T19:14:46.244Z | v1.9.11 1943 | channel | (2) dns:hostname/namespace callRefTimer.ref|configSelectionQueue.length=1pickQueue.length=0
D 2023-12-12T19:14:46.244Z | v1.9.11 1943 | resolving_call | (0] startRead called
D 2023-12-12719:1446.249Z | v1.9.11 1943 | resolving call I (O] write called with message of length 787
D 2023-12-12T19:14:46.249Z | v1.9.11 1943 | resolving_call | (0] halfClose called
D 2023-12-12719:14:46.255Z | v1.9.11 1943 | dns_resolver | Resolution error for target dns:hostname/namespace: getaddrinfo ENOTFOUND hostname/namespace
D 2023-12-12T19:14:46.256Z | v1.9.11 1943 | resolving_load_balancer | dnshostname/namespace CONNECTING -> TRANSIENT_FAILURE
D 2023-12-12T19:14:46.256Z | v1.9.11 1943 | channel | (2) dns:hostname/namespace calRefTimer.unref | config SelectionQueue lengths 1 pickQueue.length 0
D 2023-12-12719:14:46.256Z | V1.9.11 1943 | connectivity_state | (2) dns:hostname/namespace CONNECTING -> TRANSIENT_FAILURE D 2023-12-12T19:14:46.256Z. | v1.9.11 1943 | channel | (2) dns:hostname/namespace Name resolution failed with calls queued for config selection D 2023-12-12719:14:46.257Z | v1.9.11 1943 | resolving_call | (0] ended with status: code=14 details 'Name resolution failed for target dns:hostname/namespace test Client: Request Error: Error: 14 UNAVAILABLE: Name resolution failed for target dns-hostname/namespace

[murgatroid99]

You are not required to use @grpc/grpc-js-xds, but I need to know whether or not you are using it to determine where to look for the problem. In this case, "no healthy upstream" seemed like the kind of error that might come from that library, but the actual error message is "Name resolution failed".

I don't think this is a problem in grpc-js itself. It just looks up whatever DNS name you give it, so you have to look elsewhere to determine why that DNS lookup is failing in your environment. As far as I know, a valid DNS name cannot contain a / character, so that's probably part of the problem.

[sunny3774]

@murgatroid99 The issue here seems to be that like the grpc-js library doesn’t have an option to take namespace as an option while making the call. The above stack trace is when i provided the DNS with hostname/namespace instead of just hostname. When I provide just the hostname while creating the client, it results in the below error like you suggested

XYZ Client: Request Error: Error: 14 UNAVAILABLE: no healthy upstream

Can you please let us know how how do we make changes in the client to make a successful grpc when a namespace is involved.

[murgatroid99]

A namespace is an Istio concept. I suggest referring to Istio's documentation for information about how to use namespaces. I think that error also comes from Istio.

[sunny3774]

@murgatroid99 I am confused here on why this is an istio concept. If i am hitting the same service from another client like postman, it worked as expected. It should work with a nodejs grpc client as well right?
All i wanted to know is if there is a way i can configure an option where the call happens
For https://hostname/namespace/protopath/service
(This works in postman client which suggests that there is nothing wrong with the istio configuration )

https://hostname/protopath/service
NOTE: the above worked when we did not configure namespace in istio as well as in the node grpc client.
The issue happens only when namespace is enabled in istio but there is no way i can pass the namespace value while constructing the url

[murgatroid99]

In gRPC, when you construct a client, you provide it with only the hostname. It resolves that hostname to an IP address using DNS (or another name resolution protocol), and then makes requests to that server with the path /fullyQualifiedServiceName/MethodName. That is all part of the definition of the gRPC over HTTP2 protocol.

In the URL https://hostname/namespace/protopath/service, only hostname is the hostname, and everything after it (/namespace/protopath/service) is the path. This URL structure with the namespace is not compatible with the gRPC protocol as implemented by the gRPC libraries. Including part of that path (/namespace) where gRPC only expects a hostname will not make it work, it will just make gRPC unable to resolve the hostname.

Based on the information I have here, I would conclude that Istio namespaces are not compatible with gRPC at all. However, I am also aware that Istio is generally compatible with gRPC. So, I would recommend checking the Istio documentation about namespaces and/or gRPC to see whether they specify how to make them compatible. For example, they might instruct you to specify the namespace in a specific gRPC metadata key.