Custom ports not used for ingress in ACA

[amolenk]

I've got an AppHost project with custom ports for Kafka and Postgres:

var kafka = builder.AddKafka("kafka", port: 6000);
var postgres = builder.AddPostgres("postgres", port: 6001);

var rideService = builder.AddProject<Projects.Application_RideService>("ride-service")
	.WithReference(postgres)
	.WithReference(kafka);

This works well locally, but when I deploy to Azure Container Apps using azd the custom ports aren't taken into consideration.

This is the ingress that's configured for the kafka container:

"ingress": {
        "allowInsecure": false,
        "clientCertificateMode": null,
        "corsPolicy": null,
        "customDomains": null,
        "exposedPort": 0,
        "external": false,
        "fqdn": "kafka.internal.__________________.swedencentral.azurecontainerapps.io",
        "ipSecurityRestrictions": null,
        "stickySessions": null,
        "targetPort": 9092,
        "traffic": [
          {
            "latestRevision": true,
            "weight": 100
          }
        ],
        "transport": "Tcp"
      },

Meanwhile, the connection string secrets of the ride-service are set as follows:

connectionstrings-kafka = kafka:6000

The result is that the ride-service cannot reach the Kafka cluster.

[davidfowl]

Can you say what version of azd you are using and can you also run

dotnet run --publisher manifest --output-path manifest.json

in the apphost directory and paste the output here?

[davidfowl]

The ingress is correct, that's the container port i.e.g the one you are bound to in the container itself. The "port" is what other containers use to talk to you, not what you are bound to.

[amolenk]

azd version 1.9.2 (commit c58b02f71710960aba28f81f3698e64cfdda9f96)

Manifest looks pretty good:

{
  "resources": {
    "application-insights": {
      "type": "azure.bicep.v0",
      "connectionString": "{application-insights.outputs.appInsightsConnectionString}",
      "path": "application-insights.module.bicep",
      "params": {
        "logAnalyticsWorkspaceId": ""
      }
    },
    "kafka": {
      "type": "container.v0",
      "connectionString": "{kafka.bindings.tcp.host}:{kafka.bindings.tcp.port}",
      "image": "docker.io/confluentinc/confluent-local:7.6.1",
      "env": {
        "KAFKA_ADVERTISED_LISTENERS": "PLAINTEXT://localhost:29092,PLAINTEXT_HOST://localhost:9092"
      },
      "bindings": {
        "tcp": {
          "scheme": "tcp",
          "protocol": "tcp",
          "transport": "tcp",
          "port": 6000,
          "targetPort": 9092
        }
      }
    },
    "postgres": {
      "type": "container.v0",
      "connectionString": "Host={postgres.bindings.tcp.host};Port={postgres.bindings.tcp.port};Username=postgres;Password={postgres-password.value}",
      "image": "docker.io/library/postgres:16.2",
      "env": {
        "POSTGRES_HOST_AUTH_METHOD": "scram-sha-256",
        "POSTGRES_INITDB_ARGS": "--auth-host=scram-sha-256 --auth-local=scram-sha-256",
        "POSTGRES_USER": "postgres",
        "POSTGRES_PASSWORD": "{postgres-password.value}"
      },
      "bindings": {
        "tcp": {
          "scheme": "tcp",
          "protocol": "tcp",
          "transport": "tcp",
          "port": 6001,
          "targetPort": 5432
        }
      }
    },
    "ride-service": {
      "type": "project.v0",
      "path": "../Application.RideService/Application.RideService.csproj",
      "env": {
        "OTEL_DOTNET_EXPERIMENTAL_OTLP_EMIT_EXCEPTION_LOG_ATTRIBUTES": "true",
        "OTEL_DOTNET_EXPERIMENTAL_OTLP_EMIT_EVENT_LOG_ATTRIBUTES": "true",
        "OTEL_DOTNET_EXPERIMENTAL_OTLP_RETRY": "in_memory",
        "ASPNETCORE_FORWARDEDHEADERS_ENABLED": "true",
        "APPLICATIONINSIGHTS_CONNECTION_STRING": "{application-insights.connectionString}",
        "ConnectionStrings__postgres": "{postgres.connectionString}",
        "ConnectionStrings__kafka": "{kafka.connectionString}"
      },
      "bindings": {
        "http": {
          "scheme": "http",
          "protocol": "tcp",
          "transport": "http"
        },
        "https": {
          "scheme": "https",
          "protocol": "tcp",
          "transport": "http"
        }
      }
    },
    "postgres-password": {
      "type": "parameter.v0",
      "value": "{postgres-password.inputs.value}",
      "inputs": {
        "value": {
          "type": "string",
          "secret": true,
          "default": {
            "generate": {
              "minLength": 22
            }
          }
        }
      }
    }
  }
}

Looks like the problem is with the exposedPort property in the ingress. When I change it to 6000 (0 after deployment) I can connect to the kafka container from the ride-service.

That immediately brings up another issue though. The KAFKA_ADVERTISED_LISTENERS environment variable is set to PLAINTEXT://localhost:29092,PLAINTEXT_HOST://localhost:9092. This is fine when running locally, but when deployed to ACA, the value should be changed to include the ACA service name and ports. Otherwise, the ride-service will try to connect to localhost:9092 and obviously fail.

I guess I can override the env var in publish mode as a workaround?

[davidfowl]

Looks like the problem is with the exposedPort property in the ingress. When I change it to 6000 (0 after deployment) I can connect to the kafka container from the ride-service.

@vhvb1989 Looks like this is a problem.

@amolenk can you file this one on https://github.com/Azure/azure-dev/issues.

Also as a future debugging tip, run azd infra synth and look at the bicep (yeah I know), it's a good way to understand the translation.

That immediately brings up another issue though. The KAFKA_ADVERTISED_LISTENERS environment variable is set to PLAINTEXT://localhost:29092,PLAINTEXT_HOST://localhost:9092. This is fine when running locally, but when deployed to ACA, the value should be changed to include the ACA service name and ports. Otherwise, the ride-service will try to connect to localhost:9092 and obviously fail.

This was recently fixed

aspire/src/Aspire.Hosting.Kafka/KafkaBuilderExtensions.cs

Lines 123 to 126 in 90cd4b5

	var advertisedListeners = context.ExecutionContext.IsRunMode
	? ReferenceExpression.Create($"PLAINTEXT://localhost:29092,PLAINTEXT_HOST://localhost:{primaryEndpoint.Property(EndpointProperty.Port)},PLAINTEXT_INTERNAL://{internalEndpoint.ContainerHost}:{internalEndpoint.Property(EndpointProperty.Port)}")
	: ReferenceExpression.Create(
	$"PLAINTEXT://{primaryEndpoint.Property(EndpointProperty.Host)}:29092,PLAINTEXT_HOST://{primaryEndpoint.Property(EndpointProperty.Host)}:{primaryEndpoint.Property(EndpointProperty.Port)},PLAINTEXT_INTERNAL://{internalEndpoint.Property(EndpointProperty.Host)}:{internalEndpoint.Property(EndpointProperty.Port)}");

(along with some other issues)

Yes, you should be able to copy this logic and override it in the kafka container

[amolenk]

@davidfowl I'd expected the KAFKA_ADVERTISED_LISTENERS fix to be part of the 8.0.1 release, but it looks like it's not. Any idea when the fix will be released?

[davidfowl]

The release after that. 8.0.1 doesn't have any of the features in main. We're still working out the scope and date for that release. For now, just copy the code above into your project as a workaround.