docker compose v2 does not override if new value is empty or null

[osauldmy]

Description

I'd like to rewrite the value of environmental variable using multiple compose files mechanism, which worked perfectly in 1.29, but seems like a bug in v2.

Steps to reproduce the issue:

Simple two files example. Compose files are intentionally minified to focus on the issue.

$ cat docker-compose.yml
---
version: '2.4'
services:
  test:
    image: alpine
    environment:
      FOO_ENDPOINT: http://localhost:8000

$ cat docker-compose.override.yml
---
version: '2.4'
services:
  test:
    environment:
      FOO_ENDPOINT: null

---

v1 overrides the env value, but v2 not.

$ docker-compose -f docker-compose.yml -f docker-compose.override.yml config
services:
  test:
    environment:
      FOO_ENDPOINT: null
    image: alpine
version: '2.4'

$ docker compose -f docker-compose.yml -f docker-compose.override.yml config
services:
  test:
    environment:
      FOO_ENDPOINT: http://localhost:8000
    image: alpine
    networks:
      default: null
networks:
  default:
    name: docker_compose_bug_default

Describe the results you received:

FOO_ENDPOINT value is taken from docker-compose.yml

Describe the results you expected:

FOO_ENDPOINT would be null (a value from override compose file)

Additional information you deem important (e.g. issue happens only occasionally):

—

Output of docker compose version:

Docker Compose version 2.0.0

Output of docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Build with BuildKit (Docker Inc., v0.6.1-docker)
  compose: Docker Compose (Docker Inc., 2.0.0)

Server:
 Containers: 16
  Running: 0
  Paused: 0
  Stopped: 16
 Images: 399
 Server Version: 20.10.8
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: false
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 72cec4be58a9eb6b2910f5d10f1c01ca47d231c0.m
 runc version: v1.0.2-0-g52b36a2d
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.14.6-arch1-1
 Operating System: Arch Linux
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 15.32GiB
 Name: dell
 ID: 7VRY:2F5T:HXPQ:5XDY:Q2KT:TMVX:BL4E:6DQN:CVJM:XOYI:7M5C:OVTW
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional environment details:

—

[osauldmy]

Possible labels: [v2] [bug]

[osauldmy]

Interesting, in 2.2.2 it seems like the example I mentioned above is fixed.

$ docker compose version
Docker Compose version 2.2.2
$ docker compose -f docker-compose.yml -f docker-compose.override.yml config

services:
  test:
    environment:
      FOO_ENDPOINT: null
    image: alpine
    networks:
      default: null
networks:
  default:
    name: docker_compose_bug_default

[ndeloof]

While this makes perfect sense this behavior unfortunately was undocumented.
I've created compose-spec/compose-spec#284 so we can get this a canonical approch

[ndeloof]

This has just been fixed in compose specification