Required variables are not required except the last one in a string

[rimelek]

Description

About two months ago I discovered a bug in Docker Compose when I wanted to use required environment variables. Everything works as expected when I have only one variable, but when I have multiple variables in a string, only the last one is validated. It is more complex, because it is actually validated and the error message is just overridden by the next non-empty variable.

I also fixed it in "compose-spec/compose-go": compose-spec/compose-go#242

However it was not merged and now the latest development version of "compose-spec/compose-go" is not compatible with Docker Compose. I don't exactly know the relation between the compose and compose-go developer teams, so I don't know what the next step could be, but I would like Docker Compose v2 to work properly so I offer any help I can give you. In the end, people will see and remember bugs in Docker Compose and not in its modules.

I create this issue because even if my pull request were merged, considering the current branches and the process of the development, you could not apply that in Docker Compose until you make Docker Compose compatible with the new changes in compose-go in which ExtraHosts is not []string anymore but type HostsList map[string]string . You might have a better idea how it could be fixed in Compose v2 as well without waiting for an other minor version or you can assure us that a new minor version with that fix could be released soon.

If I can see that bugfixes like these can happen, I am happy to continue to work on other bugs as well if it helps.

Steps to reproduce the issue:

Use a compose file like this:

services:
  bash:
    image: bash:5.1
    command:
      - bash
      - -c
      - "echo VARIABLES: ${COMPOSE_TEST_A:?A is empty} ${COMPOSE_TEST_B:?B is empty} ${COMPOSE_TEST_C:?C is empty}"

with this .env file:

COMPOSE_TEST_A=A
COMPOSE_TEST_B=B
COMPOSE_TEST_C=C

Run

docker compose up
COMPOSE_TEST_B="" docker compose up
COMPOSE_TEST_C="" docker compose up

Describe the results you received:

The output of the above example is:

[+] Running 1/0
 ⠿ Container compose-test-bash-1  Recreated                                                                                                                                                                                        0.0s
Attaching to compose-test-bash-1
compose-test-bash-1  | VARIABLES: A B C
compose-test-bash-1 exited with code 0

[+] Running 1/0
 ⠿ Container compose-test-bash-1  Created                                                                                                                                                                                          0.0s
Attaching to compose-test-bash-1
compose-test-bash-1  | VARIABLES: A C
compose-test-bash-1 exited with code 0

invalid interpolation format for services.bash.command.[]: "required variable COMPOSE_TEST_C is missing a value: C is empty". You may need to escape any $ with another $

Describe the results you expected:

Any empty but required variable should result an error message started with the first error:

[+] Running 1/0
 ⠿ Container compose-test-bash-1  Recreated                                                                                                                                                                                        0.0s
Attaching to compose-test-bash-1
compose-test-bash-1  | VARIABLES: A B C
compose-test-bash-1 exited with code 0

invalid interpolation format for services.bash.command.[]: "required variable COMPOSE_TEST_B is missing a value: B is empty". You may need to escape any $ with another $

invalid interpolation format for services.bash.command.[]: "required variable COMPOSE_TEST_C is missing a value: C is empty". You may need to escape any $ with another $

Output of docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., v0.8.2)
  compose: Docker Compose (Docker Inc., v2.5.0)
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc., 0.6.0)
  scan: Docker Scan (Docker Inc., v0.17.0)

Server:
 Containers: 3
  Running: 1
  Paused: 0
  Stopped: 2
 Images: 65
 Server Version: 20.10.14
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: active
  NodeID: z6hojlb8sew26nf77ip1xwzzh
  Is Manager: true
  ClusterID: ibg3se3bghfvjz73apiicsu6l
  Managers: 1
  Nodes: 1
  Default Address Pool: 10.0.0.0/8
  SubnetSize: 24
  Data Path Port: 4789
  Orchestration:
   Task History Retention Limit: 3
  Raft:
   Snapshot Interval: 10000
   Number of Old Snapshots to Retain: 0
   Heartbeat Tick: 1
   Election Tick: 10
  Dispatcher:
   Heartbeat Period: 5 seconds
  CA Configuration:
   Expiry Duration: 3 months
   Force Rotate: 0
  Autolock Managers: false
  Root Rotation In Progress: false
  Node Address: 192.168.65.3
  Manager Addresses:
   192.168.65.3:2377
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3df54a852345ae127d1fa3092b95168e4a88e2f8
 runc version: v1.0.3-0-gf46b6ba
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.10.104-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 4
 Total Memory: 7.75GiB
 Name: docker-desktop
 ID: PH52:XTVE:4E2N:YEUX:LQ6J:6TQ2:Q7LQ:2IMB:TS4F:2OSW:YGK6:QKNT
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal,192.168.65.4,192.168.65.4
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5000
  127.0.0.0/8
 Live Restore Enabled: false

Additional environment details:

The environment is not relevant here since I already found the cause which was error messages overriding each other in a callback function, however I tried on MacOs and Linux mainly.