Need multiple ports open on service/sidecar for communication between containers.

[al-dpopowich]

I am migrating an existing compose-esc (docker-compose integration
with ecs fargate) to copilot. I am having trouble figuring out how to
configure a service/sidecar with multiple ports open on the private network for intra-container communication.

My existing architecture has the following defined docker-compose
services:

• NGINX - Listens on ports 80/443, open from the Internet. These are
  the only public ports exposed in the application. Its main
  job is to serve our SPA's statically built assets (html +
  js/css bundles) and reverse proxy requests on certain paths
  to internal services.
• API - Provides our REST API. Listens on private net on port 8080.
  Incoming requests on /api are reverse-proxied to here.
• AUTH - Provides authentication services (keycloak). Listens on
  private net on port 8080. Incoming requests on /auth are
  reverse-proxied to here.
• DB - Postgres, used by both API and AUTH, listens on port 5432 on
  the private net.
• DEVSMTP - Test mail service (mailpit). Allows us to test mail
  delivered by AUTH and API services in our test
  environments. (Our services are configured with a real SMTP
  server in production.) It listens on TWO ports: 1025 for
  SMTP (which both the API and AUTH services are configured to
  use) and 8025, a web application allowing a tester to
  see/read the delivered mail.

          80/443
Internet -------->  NGINX-SVC
                      |
                /auth | /api
   -- AUTH-SVC <------|-----> API-SVC --
  |     (8080)        |         (8080)   |
  |                   | /mail            |    DB (5432)
  |                   v (8025)           |
  |               DEVSMTP-SVC            |
  |                   ^                  |
  |                   | (1025)           |
   --------------------------------------

I cannot figure out how to configure the DEVSMTP service:

1. ...as a service exposing two internal ports to the private network
   (continung to reverse-proxy w/ nginx). (WHY are we only allowed to
   configure one port for a service/sidecar?!?!?) -or-
2. ...as a sidecar to the NGINX service with 1025 open to
   the private network and 8025 configured as an
   additional_rule.

I cannot figure out if there is a way to do 1 above without yaml
patch (the whole point of a tool like copilot is to AVOID using
cloudformation templates, right?).

For 2, my last attempt (as a sidecar, see the relevant yaml below)
keeps failing because of a healthcheck failure on 1025.

http:
   # Requests to this path will be forwarded to your service.
   # To match all requests you can use the "/" path.
   path: '/'
   # You can specify a custom health check path. The default is "/".
   healthcheck:
      path: '/health-check'
      port: 80
   additional_rules:
   -  path: 'mail'
      tartget_port: 8025
      target_container: devsmtp

network:
  connect: true

sidecars:
   devsmtp:
      image: axllent/mailpit
      port: 1025
      network:
         connect: true
      secrets:
         MP_UI_AUTH: arn:aws:secretsmanager:us-east-2:990838414927:secret:staging/mailauth-HnnGfz
      variables:
         MP_WEBROOT: /mail

I get this error:

• (service onboard-staging-webapp-Service-lVlIBc1pUjjz) (port 1025) is
  unhealthy in (target-group
  arn:aws:elasticloadbalancing:us-east-2:990838414927:targetgroup/onboar-Targe-POTKBIR2SDLP/813af8176602329b)
  due to (reason Health checks failed).

I know I could get this to work if I could just run mailpit with two exposed ports to the private network. The api and auth could connect to port 1025 (like they do now to postgres:5432) and I could configure nginx to reverse-proxy /mail, like I do now.

Help!

And thanks!!

[Lou1415926]

@al-dpopowich Thanks for the write-up and all the details (and the ASCII diagram)!

Your second attempt involves setting "devsmtp" as a sidecar to the nginx service, a public-facing Load-Balanced Web Service. I'd like to note that the manifest that you presented exposes the "/mail" path to the public and routes those public traffic to "devsmtp"'s port 8025. This doesn't seem like what you want since "devsmtp" is meant to be hidden in the private network.

Then here are some ways to achieve your goal ⬇️

As a backend service, with a YAML patch; use Service Discovery

You can deploy "devsmtp" as a Backend Service. Set image.port to 1025 (or 8025), and create a YAML patch that adds an additional PortMappings that exposes 8025 (or 1025) as well, on the main container. Your other services should be able to send traffic to "devsmtp" over Service Discovery on both port. The endpoint would look something like devsmtp.test.myapp.local:8025 and devsmtp.test.myapp.local:1025.

I know you've indicated that you preferred not to use YAML patch -

I cannot figure out if there is a way to do 1 above without yaml patch (the whole point of a tool like copilot is to AVOID using cloudformation templates, right?).

In addition to avoiding CFN as much as possible, Copilot also helps by providing service patterns, operational commands that let you monitor the applications, test your applications locally (local run), etc. Client-side tools such as Copilot abstracts away a lot of the headache to set up infra, provides knobs and switches that let you attach additional components... but it nevertheless has to expose some lower-level layers (in this case, the CFN template) for users to break the ceiling instead of waiting for Copilot's support. This is a whole different discussion but I'm happy to chat more!

Having to use YAML patch here is indeed due to Copilot's lack of native support for exposing multiple ports per container. If image.port and network.connect, or an imaginary field like network.discoverycan accepts a list of ports, then you wouldn't have needed to use yaml patch.

As a backend service, use internal ALB

Set up the http field for the backend service, with something like this:

http:
  path: '/'
  target_port: 1025
  additional_rule:
    - path: '/mail'
      target_port: 8025

This under the hood sets up an internal application load balancer (accessible only within the VPC). The other services would send the traffic to the internal ALB URL that's generated (and provided to you at the end of the deployment), over port 80 and the respective path. Requests to :80/ will route to the main container's 1025 port, and requests to :80/mail will go to the 8025 port. Extra cost are involved with internal ALB, but you don't need to maintain a YAML patch, and you get the load-balancing.

Side note

I did find the health check failure mentioned in your post unexpected. The application load balancer isn't expected to send health check to port 1025 because 1025 is not set to connect to the alb 🤔 . The alb should be sending health check to just port 8025 (because it's in additional_rule) and port 80 (because it's in http.healthcheck). I need to investigate on this, but this isn't directed related to your question, since you probably don't want "devsmtp" as a sidecar to the LBWS anyway.

[al-dpopowich]

Thanks for the response. I'll give the YAML patch a try.

(And yes, the failed healthcheck was a head-banger!)

It's rather surprising that multi-port support isn't part of the manifest yaml definition considering multiple EXPOSE directives have been part of the Dockerfile spec since forever.

What's the best way to ask for a new feature, e.g., image.port can be a single Integer or array of Integer, so all services can be referenced internally by their service name via Service Connect? Open an Issue?

[Lou1415926]

Yes opening an issue is the best way to request for a feature!