v2.7 expands .env file values

[antonagestam]

In prior versions, this .env file would result in the variable BASIC_AUTH_CLIENTS being set to the value {"a_client":"md5$ldjDcN3a5x7x$46b3330841bbe4974ef43cb8b97034e"}:

BASIC_AUTH_CLIENTS={"a_client":"md5$ldjDcN3a5x7x$46b3330841bbe4974ef43cb8b97034e"}

In version 2.7 this results in the following error message:

WARN[0000] The "ldjDcN3a5x7x" variable is not set. Defaulting to a blank string. 
Invalid template: "{\"a_client\":\"md5$ldjDcN3a5x7x$46b3330841bbe4974ef43cb8b97034e\"}"

.env files used to be "dumb" and not allow any expansions, which is a rigid but stable behavior. This change is problematic because it isn't backwards compatible, and will require synchronized upgrades across environments.

Output of docker compose version:

Docker Compose version v2.7.0

[ulyssessouza]

In Docker Compose V2 dollar signs have to be escaped by doubling it ($$).
Another solution would be surrounding the whole string with single quotes (''), like in:

BASIC_AUTH_CLIENTS ='{"a_client":"md5$ldjDcN3a5x7x$46b3330841bbe4974ef43cb8b97034e"}'

This indicates that the string mustn't be evaluated.

[antonagestam]

@ulyssessouza What are you referring to by Docker Compose V2, spec or implementation? Your statement at least seems to contradict this document, is that not the correct spec to look at?

The value of VAL is used as a raw string and not modified at all. If the value is surrounded by quotes (as is often the case for shell variables), the quotes MUST be included in the value passed to containers created by the Compose implementation.

If you are referring to the version of the implementation, I don't think that's true either for versions prior to 2.6 or perhaps 2.7?

[ulyssessouza]

I mean the implementation. So this git repo.
Thanks for the reference. I'll propose a change in the spec for that.

I don't remember exactly when the behaviour was introduced, but it was with godotenv. So I think that was since the beginning of version 2.

[laurentlbm]

This behaviour was not present for env_file before 2.7, I assume it was added because of this "fix": Fix environment variable expansion

It has caused me problems, as it seems to have for many others, and I find it a bit frustrating. The specification explicitly says that values are used "as a raw string and not modified at all." I seriously think this change has a broader impact than you realize: docker-compose >= 2.7 now doesn't implement the specification correctly and environment files aren't backward-compatible anymore... It's not just about expansion, a value surrounded by quotes isn't treated the same anymore! Many users might be affected by this change and not even realize it (yet).

The change in behaviour could be reverted, but that would affect all those who had to fix their environment files and those who depend on this feature. If it's not reverted and the specification is updated, it needs to be clear that the behaviour has changed and that not all implementations work the same.

Either way, I think this change needs to be communicated quickly and broadly so that people can react and fix their code.

[antonagestam]

@laurentlbm That's what I suspected too, but it was a little too cumbersome to downgrade or build previous versions of docker-compose for me locally to prove it.

[kiere]

I downloaded the version of Docker Desktop for Mac that I had before (4.9, I believe) and reinstalled. That solved the issue without making any changes, so it's definitely new.

[nephthys]

I downloaded the version of Docker Desktop for Mac that I had before (4.9, I believe) and reinstalled. That solved the issue without making any changes, so it's definitely new.

Same problem on the latest version of Docker Desktop for Mac (4.11.1 with Apple chip, Docker Compose version v2.7.0).

If I roll back to Docker Desktop 4.9.1 (Docker Compose version v2.6.0), it works again.

[superbiche]

Same here.
Docker 20.10.17
Docker Compose 2.9.0

Even worse, envvars like APP_NAME="Vacation [Local]" are also expanded. We use these variables in docker-compose.yml so that Traefik picks the services, resulting in all our development apps being broken.

I don't want to seem rude but I find it really hard to understand how such a behavior can be introduced without giving sufficient information to users.

[dimadeush]

The same issue now. F.E. have a php symfony framework with .env file and config params inside. There I have row:
CORS_ALLOW_ORIGIN=^https?://localhost(:[0-9]+)?$
when I'm trying to build using docker-compose I can see error: Invalid template: "^https?://localhost(:[0-9]+)?$"
docker-compose v2.10.2, docker 2.10.17

[antonagestam]

@ulyssessouza It would be nice with an acknowledgement that the reason for closing this issue was based on an invalid assumption. Given your comments it really seems that this change of behavior in v2.7 wasn't intentionally introduced.

I think this change warrants a decent migration path, eg a switch that enables or disables the old behavior. But regardless of whether you intend to accept or reject this issue, acknowledging it exists seems appropriate.

[milas]

Hey everyone - I wanted to provide an update here with a little more context.

We are aware that Compose behavior is divergent from the specification. However, upon further review, it's clear that the specification is actually incorrect/in error. The behavior it describes does not accurately reflect Compose v1 or (any version of) v2. That's an unfortunate oversight and one we're actively working to correct!

We place a very high emphasis on compatibility, and the changes around environment variable expansion have all been done with this in mind. Similarly, it's why we're correcting the spec in this instance instead of dramatically changing Compose behavior. (For example, quotes are not literally passed through as part of values in either v1 or v2! And interpolation has been a feature in both v1 and all versions of v2!)

For the specific issue in this thread, I want to re-iterate the suggestion to single-quote the value, which will suppress interpolation in v2 and is backwards-compatible with v1. (Note: v1 ALWAYS performs interpolation! However, it does not support "unbraced" interpolation, e.g. $foo, so won't attempt substitution for the specific example represented by this issue.)

See #9879 for more examples and detail on the syntax and keep an eye out for bigger docs updates soon.

[antonagestam]

@milas So you're basically not acknowledging that there WAS a "dramatic change in Compose behavior" in version 2.7!?

[antonagestam]

A breaking change between v1 and v2 is one thing, but this issue is really about a breaking change between versions 2.6 and 2.7.

[milas]

@antonagestam Compose v1's parameter expansion logic has numerous issues and pitfalls. At the same time, the original expansion logic in Compose v2 frankly had its own set of issues. And the spec was unfortunately not of use here.

I apologize for any disruption this has caused, but I hope you understand we're working to balance defining a sane, maintainable contract while also correcting long-standing issues and ambiguities.