Jaeger Query on AWS ECS Fargate

[evilr00t]

What happened?

Hi!

I'm deploying Jaeger Query on ECS Fargate and I have an issue seeing my services as GRPC Query service has an issue with connectivity.

• 127.0.0.1 - can listen, can't connect

{
    "level": "info",
    "ts": 1707149120.5698245,
    "caller": "app/server.go:299",
    "msg": "Starting GRPC server",
    "port": 4444,
    "addr": "localhost:4444"
}
{
    "level": "info",
    "ts": 1707149121.5698493,
    "caller": "grpc@v1.58.2/clientconn.go:1340",
    "msg": "[core][Channel #2 SubChannel #3] Subchannel Connectivity change to IDLE, last error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:4444: connect: connection refused\"",
    "system": "grpc",
    "grpc_log": true
}

• localhost - can listen, can't connect (not recommended on ECS to use localhost, 127.0.0.1 is advised)

• service-discovery DNS (IP of Container) - can't listen...

{
    "level": "fatal",
    "ts": 1707148669.5664368,
    "caller": "./main.go:127",
    "msg": "Could not start servers",
    "error": "listen tcp 10.10.76.5:4444: bind: cannot assign requested address",
    "stacktrace": "main.main.func1\n\t./main.go:127\ngithub.com/spf13/cobra.(*Command).execute\n\tgithub.com/spf13/cobra@v1.7.0/command.go:940\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tgithub.com/spf13/cobra@v1.7.0/command.go:1068\ngithub.com/spf13/cobra.(*Command).Execute\n\tgithub.com/spf13/cobra@v1.7.0/command.go:992\nmain.main\n\t./main.go:159\nruntime.main\n\truntime/proc.go:267"
}

I'm using CLI Flag: --query.grpc-server.host-port but when I specify for example 0.0.0.0:16685 - then Jaeger Query does not know where to connect.

{
    "level": "info",
    "ts": 1707147647.5808122,
    "caller": "app/server.go:299",
    "msg": "Starting GRPC server",
    "port": 16685,
    "addr": "0.0.0.0:16685"
}
{
    "level": "info",
    "ts": 1707147648.5809,
    "caller": "grpc@v1.58.2/clientconn.go:1340",
    "msg": "[core][Channel #2 SubChannel #3] Subchannel Connectivity change to IDLE, last error: connection error: desc = \"transport: Error while dialing: dial tcp 0.0.0.0:16685: connect: connection refused\"",
    "system": "grpc",
    "grpc_log": true
}

I'm wondering, would it make sense to have 2 CLI Flags?

• one for server
• second for client?

In that case I could use 0.0.0.0 for server and then use DNS Host for client which would point to that container?

Or maybe I'm missing something or doing something wrong here?

Steps to reproduce

Try to deploy Jaeger Query to ECS Fargate using AWSVPC Networking.

Expected behavior

GRPC Query Service could connect to itself internally.

Relevant log output

No response

Screenshot

No response

Additional context

No response

Jaeger backend version

1.53.0

SDK

No response

Pipeline

No response

Stogage backend

AWS OpenSearch

Operating system

Linux

Deployment model

AWS ECS Fargate

Deployment configs

No response

[yurishkuro]

• Internal connections should be via 127.0.0.1
• External ingress should be managed through the environment (no specific advice here as I don't know Fargate). For example, when running in Docker container you just map the external port to internal and it normally works.

[evilr00t]

1. Indeed, altho it does not work for GPRC Query service.
2. Exposing UI to ALB / Load Balancer is done using port mapping but again, this does not solve the GRPC issue.

I still can't figure out why 127.0.0.1 / localhost is refused on Fargate - this is working fine when I host on EC2 or ECS on EC2. Debug log mode didn't reveal too much as well...

[yurishkuro]

What gRPC self-loop are you referring to? Prior to v1.53 the query service had an HTTP v3 API implemented by looping back into gRPC service. This has been removed in v1.53, the looping is now internal in the code, not via a network hop.

[evilr00t]

After removing extra environment variables jaeger query works again. Tested with 1.50.0 and 1.53.0. Will close this discussion.