Environment variables priority #9636

ulyssessouza · 2022-07-08T12:13:44Z

What I did
This PR is meant to define the precedence of the environment variables evaluation.

For the environment variables to be available in the container:

Command Line (docker compose run --env <KEY[=VAL]> https://docs.docker.com/engine/reference/commandline/compose_run/#options)
Compose File (service::environment section: https://docs.docker.com/compose/compose-file/#environment)
Compose File (service::env_file section file: https://docs.docker.com/compose/compose-file/#env_file)
Container Image ENV directive (https://docs.docker.com/engine/reference/builder/#env)

For the environment variables available for docker compose's runtime:

Command Line parent command option docker compose --env-file <FILE> (or .env in the root of the project by default)
OS Environment Variables

Obs: Note that the variables available for runtime will not be available in the container unless it's imported by one of the methods on the list above

Related issue
Resolves #9521
Resolves #9638
Resolves #9608
Resolves #9578
Resolves #9468
Resolves #9683

Depends on compose-spec/compose-go#280

Documentation update PR

(not mandatory) A picture of a cute animal, if possible in relation with what you did

ulyssessouza · 2022-07-12T18:45:30Z

go.mod

@@ -151,3 +151,5 @@ replace (
 	k8s.io/apimachinery => k8s.io/apimachinery v0.22.4
 	k8s.io/client-go => k8s.io/client-go v0.22.4
 )
+
+replace github.com/compose-spec/compose-go => github.com/compose-spec/compose-go v1.2.9-0.20220712021117-dcfe0889b601 // TODO Remove and bump require section when PR is merged


Note that this points to compose-spec/compose-go#280
This PR shouldn't be merged before this dependency is merged and updated here.

milas

Wahoo! Looks good for approval once the compose-go side of the house is merged + tagged 🥳

(Sorry for nits - as you're well aware, the env code is pretty complex, so I was trying to identify places I got confused and had to dig in more)

pkg/compose/run.go

pkg/e2e/compose_environment_test.go

cmd/compose/compose.go

nicksieger

Overall this looks good, just have some questions about the tests and what the new priority order is.

nicksieger · 2022-07-26T15:06:44Z

pkg/e2e/compose_environment_test.go

+	t.Run("compose file priority", func(t *testing.T) {
+		cmd := c.NewDockerComposeCmd(t, "-f", "./fixtures/environment/env-priority/compose-with-env.yaml",
+			"--env-file", "./fixtures/environment/env-priority/.env.override",
+			"run", "--rm", "-e", "WHEREAMI", "env-compose-priority")


So there's a difference between -e WHEREAMI with the var defined, and -e WHEREAMI=shell? In this case shouldn't 3. be the expected result?

In the first, the variable value will be inherited from the environment.
The second one will have the explicit value passed (shell in this case).

nicksieger · 2022-07-26T15:07:55Z

pkg/e2e/compose_environment_test.go

 			"--rm", "-e", "WHEREAMI", "env-compose-priority")
 		cmd.Env = append(cmd.Env, "WHEREAMI=shell")
 		res := icmd.RunCmd(cmd)
-		assert.Equal(t, strings.TrimSpace(res.Stdout()), "Compose File")
+		assert.Equal(t, strings.TrimSpace(res.Stdout()), "override")


Update to 3. Environment file <-- Result expected above?

Also, these tests so far seem to indicate that 1. Compose file is not the highest priority. Am I reading this right?

I just refactored the comments to fix this and reflect the order of this change.
BTW, the new doc on priority is in the oven docker/docs#15161

Signed-off-by: Ulysses Souza <ulyssessouza@gmail.com>

glours

LGTM

K0Te · 2022-08-10T05:48:07Z

Hi,

Sorry if this is not the correct place to ask a question, I've checked this PR source code and documentation update and am still not sure what the expected behavior is.
I've used .env file to override COMPOSE_FILE for local development: COMPOSE_FILE=docker/docker-compose.yml:docker/docker-compose.override.yml. This allowed to have all docker stuff in a separate folder and provide some secrets in docker-compose.override.yml.
And on CI I've just used COMPOSE_FILE=docker/docker-compose.yml as an env variable in GH Actions.

This worked fine as of docker compose 2.7.0, but with docker compose 2.9.0 this workflow doesn't work anymore. Value from .env started to take precedence over the environment variable.

Is this a new expected behaviour or a bug?

ulyssessouza requested a review from glours July 8, 2022 12:13

ulyssessouza force-pushed the dotenvfile-priority branch 3 times, most recently from 8fc5249 to e931d20 Compare July 12, 2022 17:24

ulyssessouza mentioned this pull request Jul 12, 2022

env_file variable values overwritten by .env values since v2.6.0 #9521

Closed

ulyssessouza marked this pull request as ready for review July 12, 2022 18:44

ulyssessouza commented Jul 12, 2022

View reviewed changes

ulyssessouza requested review from nicksieger, milas, ndeloof and laurazard July 12, 2022 18:46

ulyssessouza mentioned this pull request Jul 12, 2022

Import dotenv file to os environment #9512

Merged

ulyssessouza force-pushed the dotenvfile-priority branch from e931d20 to a5fbd64 Compare July 13, 2022 19:34

milas reviewed Jul 14, 2022

View reviewed changes

pkg/compose/run.go Outdated Show resolved Hide resolved

pkg/compose/run.go Outdated Show resolved Hide resolved

pkg/e2e/compose_environment_test.go Outdated Show resolved Hide resolved

cmd/compose/compose.go Outdated Show resolved Hide resolved

ulyssessouza force-pushed the dotenvfile-priority branch 3 times, most recently from 8d19ecf to 4870c50 Compare July 14, 2022 23:53

nicksieger reviewed Jul 26, 2022

View reviewed changes

Fix .env and --envfile priorities

bac398a

Signed-off-by: Ulysses Souza <ulyssessouza@gmail.com>

ulyssessouza force-pushed the dotenvfile-priority branch from 4870c50 to 69ad343 Compare July 28, 2022 10:17

Fix environment variables priority between environment and .env

e9c8cfc

Signed-off-by: Ulysses Souza <ulyssessouza@gmail.com>

ulyssessouza force-pushed the dotenvfile-priority branch from 69ad343 to e9c8cfc Compare July 28, 2022 10:57

nicksieger approved these changes Jul 28, 2022

View reviewed changes

glours approved these changes Jul 29, 2022

View reviewed changes

glours merged commit 6d613c8 into docker:v2 Jul 29, 2022

miketheman mentioned this pull request Jul 30, 2022

[docker-up] Update docker compose to v2.7.0 gitpod-io/gitpod#11604

Merged

1 task

ikedam mentioned this pull request Jul 30, 2022

Resolve environment variables case-insensitively on Windows #9438

Merged

Jason-Crowe mentioned this pull request Aug 1, 2022

docker-compose overrides values from .env with host value if present in host's environment variables #9578

Closed

curry684 mentioned this pull request Aug 8, 2022

Compose V1 doesn't behave as V2 when referencing environment variables across multiple .env files #9725

Closed

milas mentioned this pull request Aug 26, 2022

docker-compose.yml env_file option not process properly #9767

Closed

divbzero mentioned this pull request Sep 28, 2022

make errors after upgrade to Docker Compose 2.6.0 pypi/warehouse#11674

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Environment variables priority #9636

Environment variables priority #9636

ulyssessouza commented Jul 8, 2022 •

edited

ulyssessouza Jul 12, 2022 •

edited

milas left a comment

nicksieger left a comment

nicksieger Jul 26, 2022

ulyssessouza Jul 28, 2022

nicksieger Jul 26, 2022

nicksieger Jul 26, 2022

ulyssessouza Jul 28, 2022 •

edited

glours left a comment

K0Te commented Aug 10, 2022

Environment variables priority #9636

Environment variables priority #9636

Conversation

ulyssessouza commented Jul 8, 2022 • edited

ulyssessouza Jul 12, 2022 • edited

Choose a reason for hiding this comment

milas left a comment

Choose a reason for hiding this comment

nicksieger left a comment

Choose a reason for hiding this comment

nicksieger Jul 26, 2022

Choose a reason for hiding this comment

ulyssessouza Jul 28, 2022

Choose a reason for hiding this comment

nicksieger Jul 26, 2022

Choose a reason for hiding this comment

nicksieger Jul 26, 2022

Choose a reason for hiding this comment

ulyssessouza Jul 28, 2022 • edited

Choose a reason for hiding this comment

glours left a comment

Choose a reason for hiding this comment

K0Te commented Aug 10, 2022

ulyssessouza commented Jul 8, 2022 •

edited

ulyssessouza Jul 12, 2022 •

edited

ulyssessouza Jul 28, 2022 •

edited