Fix lookup precedence

[ulyssessouza]

No description provided.

[nicksieger]

LGTM

[milas]

This seems to be changing the precedence to make env file higher than OS/shell, which I didn't think we wanted

[milas]

This seems to be making env file HIGHER precedence than shell - isn't that the opposite of what we want?

[ulyssessouza]

No. That's exactly the intention.
If the users want to choose the OS one they can use something like this in the .env file:

MYVAR=${MYVAR:-DefaultValue}

This would use the OS variable when present and use DefaultValue when unset or empty.

That's why I had to fix the default values first.

[milas]

Ahhh okay I just did some sleuthing, and v1 does behave in the same way that you are making the changes here: env has higher precedence than shell, so interpolation/default syntax is required for any values you want to "fall through" to shell.

HOWEVER, my confusion came from the fact that the Compose docs, even historically going back to v1, actually incorrectly state the opposite behavior 🙈 (e.g. here's the page in 2019 -> https://github.com/docker/docker.github.io/blob/f2093d4e2a5f950d1fd8e6dc61441a1d90cecdb4/compose/environment-variables.md?plain=1#L115-L117). NOTE: the docs are now technically correct, as current v2 behavior matches what they say rather than what v1 actually does 🙃

So, I think for v1 compat, this makes sense! But we'll definitely need to take care to find any remaining errant references to the order being the other way around, and given that v2 is now flip-flopping its behavior, should make sure we call that out explicitly in release notes, along with the "workaround" (use MYVAR=${MYVAR:-DefaultValue} syntax in .env file), and explanation that it should now behave the same between v1+v2 😅

[glours]

LGTM
Waiting for @milas approval before merging

[milas]

@ulyssessouza @glours I found a case where (after these changes) v1 and v2 behave slightly differently: the in-file interpolation in v1 DID take shell as higher precedence than env.

Arguably, that's a bug/inconsistency in v1, but noting here so that we can mention it in docs/release notes somehow and to make sure there's not some reason that behavior was desirable???

$ cat .env
VAR=dotenv

$ cat docker-compose.yaml
services:
  echo:
    image: alpine
    command: [
      '/bin/ash',
      '-c',
      'echo "Compose config: ${VAR} | Container: $${VAR}"'
    ]
    env_file: .env

Version	Command	Result
v1	docker-compose up	✅ Compose config: dotenv | Container: dotenv
v1	VAR=shell docker-compose up	❌ Compose config: shell | Container: dotenv
v2 (with fixes)	docker compose up	✅ Compose config: dotenv | Container: dotenv
v2 (with fixes)	VAR=shell docker compose up	✅ Compose config: dotenv | Container: dotenv

[ulyssessouza]

@milas Hmmm! It's because in V1 the precedence of .env and the OS Env are switched in this case. So the OS is not overwritten by the .env.

If you change the name of the env file (also in docker-compose.yml) you should see the same results.

[milas]

in V1 the precedence of .env and the OS Env are switched in this case

Yeah, I'd argue this is a bug / inconsistency / undesirable behavior from v1, so I don't think it makes sense to carry over.

I think the behavior of v2 (with your changes from this PR + others) is now good/correct! 💪

[glours]

🎉