allow removing something in docker-compose.override.yml

[stefanfoulis]

My usecase is that the base docker-compose.yml maps a certain port in the ports section. In my docker-compose.override.yml I'd like to change that port since I already have a different service running there on my host.
As far as I understand the current implementation it is only possible to add stuff, override extends the other file.
As far as I can tell there is no way to remove an entry. And I can't think of an obvious syntax for it.

[aanand]

Correct. Best practice is to not include anything in docker-compose.yml that you might not want to have in another environment.

[nkovacs]

My problem with your suggestion is that docker-compose.override.yml might not exist (it's an optional file the developer can create locally). I only want to use docker-compose.override.yml to locally customize ports, like this:

docker-compose.yml:

version: '2'
services:
    web:
        ports:
            - 80
            - 443
        ...

docker-compose.override.yml:

version: '2'
services:
    web:
        ports:
            - 32080:80
            - 32443:443

This isn't a big deal because this will only result in 80 being exposed twice, once as a random port and once as 32080. I think it would make sense in this case to only expose it as 32080 automatically (without having to specify anything in the override to remove the original config value)

[tomekit]

Facing the same problem. I've got one specific host which shouldn't start the one of the docker containers for given service. Yeah, maybe my docker-composer.yml approach is not the best, but I can see the need of removing something.

[paunin]

Did anyone get script to remove ports from docker-compose file ? :)

[paunin]

Alright, there is no solution so far, so I got a simple port-remover
https://github.com/paunin/docker-compose-v2-ports-remover

In general I dont think docker-compose should allow to do it, as then logic of merging many files is gonna be complex...

[bluemoehre]

This was already solved here #3939

[jedie]

It's also not possible to removed a service, isn't it?

[OnkelTem]

Having an "override" thing that cannot do half of the job... well, it's something.

[mqu]

I'd like to be able to override some services and disable some others.

[sauron918]

Same issue, I'd like to have possibility to remove port, dns, etc in docker-compose.override.yml but not override or extend with another one..

[keithy]

Anyone notice that the only thing that you can't do with docker-compose.overide.yml, is to actually override anything!

[gnumoksha]

I would like to remove some services because I rarely use them on my machine, so I ended up with the following docker-compose.override.yml:

version: '3.5'

services:
  elasticsearch:
    image: hello-world

  selenium:
    image: hello-world

[darakian]

To add two more cents to the pile. It can happen that you pull a docker-compose.yml from another group and want to not run some of what they have turned on. Please add some syntax for removing.

[speller]

Three years passed and still nothing?

[immanuelfodor]

Stumbled upon this issue when trying to override the dns settings of a random Github project without modifying their docker-compose.yml. The new dns servers are appended to the original list instead of overriding the original ones - as I supposed it will work based on the file name. This behavior is misleading IMHO.

[Somebi]

Same problem here... It seems it's not overriding some values correctly. Can't remove either.
For example i'm overriding with network_mode: host and need to remove links, because otherwise it's throwing error:

ERROR: for processor Cannot create container for service processor: conflicting options: host type networking can't be used with links. This would result in undefined behavior

Setting links: [] doesn't help, links: null or links: ~ throws format error.

[m-hu]

If the above saying is true, it means that the inherent parser is not able to just load several dictionaries and then apply the overriding of keys following some arbitrary order, which is not complicated in nature.

The truth is that the current design has made a key dependent behaviour. ref https://docs.docker.com/compose/extends/

My feeling is that it would be better to go for a more intuitive behaviour for developers, like that in multiple inheritance cases. The current use case consists of combining multiple yaml files in the same command line, which looks just like a one layer inheritance. The better would even be able to support multiple layer multi-inheritance using some kind of inclusion key word. Also, to specify the case when child will either merge or simply replace, some key words like "super" could hence be interesting to have.

"Composition" could be a workaround to achieve the same thing as "inheritance". In the case of docker compose file one may do something similar. The key word "extend" becomes helpful in this case, which may , yet, result in a more verbose compose file. Also, one would need to very carefully craft each file so that things are always extended/override but not removed.

[thomaspeitz]

#!/usr/bin/env ruby
require 'yaml'

data = YAML.load_file "docker-compose.yml"

majorVersion = data['version'][0].to_i
serviceName = ARGV[0]
image = ARGV[1]

if majorVersion == 3
  data['services'][serviceName]['image'] = image
  data['services'][serviceName].delete('build')
else
  puts 'Version #{majorVersion} not supported yet. Please ask your cloud-ops team!'
  exit 1
end

File.open('docker-compose.yml', 'w') { |f| YAML.dump(data, f) }

Needed a small script which removes build and adds image to a docker-compose service. Maybe this helps other building CI scripts as well e.g. removing a service.

From my point of view the remove feature should be added to docker-compose overwrite as it decreases user experience of docker-compose a lot.

[keithy]

Hi Thomas,

I wrote a simple script, docker-compose-use, as the basis of my AngelBox toolkit. This makes docker-compose files "composable". You will find this at http://github.com/keithy/sprankle-pod and http://github.com/keithy/angelbox

[gothrek22]

@thomaspeitz or just use yq. you can easily edit parts of yaml with it.

F.e. Adding image: yq w -i ./docker-compose.yml services.nginx.image nginx:latest
or removing ports: yq d -i ./docker-compose.yml services.db.ports

[sshishov]

It is ridiculous that the team of "professionals" of docker-compose cannot solve this easy issue for years. And even more funny that override file cannot even properly override anything.

[mlensment]

Any progress on this?

[ndeloof]

@gnat there's some special handling for ports and volume: those are internally converted into maps, using volume target and port {target,ip,published,protocol} as key so that merging overrides does not produce duplicates. This is not exposed in the compose.yaml file format to maintain backward compatibility, but still you should get an equivalent behavior as maps.

I've been investigating options we have to support compose-spec/compose-spec#284 in compose-go. While not trivial, I expect we get some viable solution ASAP

[gorbak25]

Also came across this .-.

[debbyglance]

Just chiming in to say "me too"

[ndeloof]

compose-spec/compose-spec#340 has been approved and merged
support in compose-go (compose-spec/compose-go#380) will be available in next release

[J7mbo]

Damn, I remember reading this in 2016 and 7 years later it's still nice to know that it got in.

[default-value-serhiia]

Yeah! It finally happened! Thank you!

[hubertnnn]

I feel this is still not enough.
Spec mentions the ability to remove elements, not overwrite them.
So based on new spec:

# docker-compose.yml
services:
  foo:
    ports:
      - 80:80
      - 443:443

docker-compose.override.yml
services:
  foo:
    ports: !reset

Will remove those ports.
But there is still no way to replace ports with custom ones, unless this is going to work (which I doubt since its not a valid yaml):

docker-compose.override.yml
services:
  foo:
    ports: !reset
      - 8080:80
      - 8081:443

[ndeloof]

@hubertnnn yes indeed, !reset only can be used reset value to type's default (empty)
to replace port, the simplest option for now is to rely on variables:

services:
  foo:
    ports:
      - ${HTTP_PORT:-80}:80
      - ${HTTPS_PORT:-443}:443

so you can run compose with a local .env file to set those variable to 8080,8081

[kj]

I think this is where I was confused too. What if you want to remove a port (or volume, or similar) rather than just replace it? I'm struggling to see how you can actually replace the whole value of ports.

[dokgu]

So how are we supposed to override ports using docker-compose.override.yml?

I have a docker-compose.yml file with a predefined port mappings - those ports are already in use so I need to override them. I don't want to make any modifications to the docker-compose.yml file because my changes will get overwritten with updates.

[olitomlinson]

yeah, I'm struggling removing all ports from a Service in the override. (I'm not intending to replace, just remove all ports)

In the override, if I do this :

version: '3.4'
services:
  workflow-app:
    ports: !reset 

I get the error : services.workflow-app.ports must be a list

So I try this :

version: '3.4'
services:
  workflow-app:
    ports: !reset 
      - ""

I get the error : error decoding 'ports': No port specified: <empty>

---

The only work around I've found is to create an entry for some non-existant ports, which is just a mess, like this :

version: '3.4'
services:
  workflow-app:
    ports: !reset 
    - "9999:9999"

What am I missing here? @ndeloof any ideas? TIA

[ndeloof]

services:
  workflow-app:
    ports: !reset [] // empty list won't trigger a parsing error

(and please, don't set a version: attribute, this is an obsolete practice!)

[olitomlinson]

@ndeloof awesome thanks so much! Thank you for the guidance around the obsolete practice. Much appreciated.

[kj]

@ndeloof I just want to check if I understand correctly.

ports: !reset []

Here, the value [] is ignored. It's just the default value for ports. So whatever you pass to ports is ignored, meaning that you cannot override it with a new set of ports. For example:

# docker-compose.yml
services:
  foo:
    ports:
      - '1000:1000'
      - '2000:2000'

# docker-compose.override.yml
services:
  foo:
    ports: !reset
      - '1000:1000'

The intention being to remove port 2000, but keep port 1000. That isn't possible at all currently (or anything involving replacing the full list)?

[ndeloof]

The intention being to remove port 2000, but keep port 1000. That isn't possible at all currently (or anything involving replacing the full list)?

yes indeed, this isn't yet supported.
You will have to use 2 override files, first one to reset to empty list, second one to set port 1000
or maybe you can tweak the original compose file to offer more flexibility via variables ?

[kj]

Thanks for clarifying @ndeloof (and so quickly!). It's good at least to know for certain that I'm not missing something,

[douardda]

Hi @ndeloof

yes indeed, this isn't yet supported.

Does it mean you plan to support it in the future? (cannot find an issue for this but I may have missed it).

[ndeloof]

feel free to create an issue on github.com/compose-spec/compose-go for this purpose
I don't have any specific plan for this, but we are debating a major refactoring of compose-go library which could make this easier in the future

[quarky42]

My development system has https_proxy and http_proxy environment variables set in order for applications (python, npm) to download their dependencies.
Docker Desktop assumes that if I have a proxy defined on the system, come hell or high water it is going to force my containers to have that proxy in them. On Linux I can add the proxy to the /etc/systemd/system/docker.d/proxy.conf file and Docker will respect the fact that I don't want that environment variable stuffed into my containers. I haven't found a similar method for Windows yet. It knows about the proxy so I've got to deal with it showing up in the container.

I need the proxy to be used for build time, and I'm careful to use "ARG" in my Dockerfiles for building so that I can use the proxy during build time and exclude it for runtime, but docker compose has other plans.

Being able to:
environment: !reset
- VAR1: var1_value_here
and not get any other variables besides VAR1 in my container is exactly what I want. Others that want to inherit environment variables from the host can simply leave off the !reset.

I tried using - http_proxy: ~ but I can see from the discussion above that docker compose doesn't implement that yaml specification correctly. Explicitly null > implicitly pulled from the host environment.

Being able to explicitly tell docker to let me have 100% control of what environment variables get defined in the docker-compose.yml would cut back on a lot of overhead dealing with corporate environments.

If someone has created the issue over on the compose-spec/compose-go github, can you please link it here as I would like to add my support for this feature. I have been fighting environment variable proxy garbage off and on for a year and I'm sick of it. I need a solution, like the one proposed above, that works on all OSes regardless of the underlying config. At the docker-compose.yml and docker-compose.override.yml level I ought to be able to definitively nail down the "truth" of how a container is defined and have it be defined that way on any system I run it.

[ndeloof]

@quarky42 HTTP_PROXY (and related) build args are automatically set based on your client configuration (~/.docker/config, proxies section`), you don't need those to be declared in your compose file.

[quarky42]

@quarky42 HTTP_PROXY (and related) build args are automatically set based on your client configuration (~/.docker/config, proxies section`), you don't need those to be declared in your compose file.

I don't want them to be set. I don't have them set in ~/.docker/config because that will set them in the containers.... Which is what I don't want.

On Linux, the solution is to set the environment variables in /etc/systemd/system/dockerd/proxy.conf

This works. There is no equivalent for Windows. Having !reset at a block level would ensure that I can create a config that explicitly denies all environment variables. When working on a project shared by multiple developers, it is very important to be able to have a consistent container generation.

I want to EXCLUDE all (100%) environment variables at the docker-compose.yml level. Currently, proxy values are being carried into my containers. Having support for !reset on a block level would ensure that only explicitly set values were used.

[ndeloof]

Do you have a comparable issue having such variables set when you run a plain docker buildx build ... for your Dockerfile(s)?

[quarky42]

In short: no, we do not build 30 services across 3 nested projects each with about 10 subprojects each by first building each Dockerfile individually, nor are we interested in doing that.

When you have a team of devs using Windows, Linux, and Mac, it is critical to have control at the docker-compose.yml level of what makes it into and doesn't make it into the container.

On Linux and Mac, that control is there. On Windows Docker Desktop, I have not found a way to prevent it from embedding proxy info when using the latest version of Docker Desktop.

I can get close on Windows, but there is still a variable there that shouldn't be.

[ndeloof]

my question was not to use this on a daily basis, but to confirm you only get such a variable issue when building with compose, and a plain docker build get a "clean" environment

[quarky42]

Thank you for the clarification. I do appreciate it. I will have to build an image manually on the CLI as I don't have anything like that. It'll be Monday before I'm back in the office.

The remote servers have a couple manually built images but those are Linux and I know they are good. They weren't at first, but I figured out the use of build args plus the systemd level conf file and that fixed them. Also those get built and tested with a script, so not as much hands on as it could be.

[paolomainardi]

It works, thanks a lot!

I have just an issue with vscode extension:

It seems that the schema is missing something https://raw.githubusercontent.com/compose-spec/compose-spec/master/schema/compose-spec.json ?

[ndeloof]

@paolomainardi !reset is a yaml (custom) tag, it is not defined by the (json) schema

[paolomainardi]

thanks @ndeloof so the issue is with the vscode official docker extension

[zhfkt]

It is finally supported from Docker Compose version 2.24.4 and later

services:
  app:
    ports: !override
      - "8443:443" 

Pls check - https://docs.docker.com/compose/compose-file/13-merge/#replace-value