[BUG] Compose up often fails to join one random network on v2.24.6 #11533
Assignees
Labels
Comments
|
Hello @Millio345 |
Thank you - Latest version available for Arch seems to be 25.0.2 but I'll test and report back as soon as an update is available |
|
Reverting to 2.24.5 did not fix the problem for me. I did think it would since I have a Nextcloud instance which will not connect to the database because it is not seen on the network as it usually is. My config looks like this: And when I inspect the Notice the The |
|
@Millio345 @mariaa144 we bumped the version of compose-go yesterday into the main branch of Compose with a fix about networks order & priority, can you check if the latest binary of Compose potentially fixes your issue? |
|
Thanks @glours. Probably I'll be waiting for a new release on Arch. |
|
I encountered this bug as well and I can confirm that the linked artifact of version c5a88de works again. I have restarted my service over 20 times and cannot reproduce the issue anymore. |
|
I rebooted today after updating a few packages and the kernel on Arch Linux and the networks started fine, along with Nextcloud. I have no explanation as to what is different. Perhaps it is a race condition issue. 🤷🏽♀️ |
It has been my experience. |
(specific to Arch Linux:) If you'd like to "try the artifact" you can quite simply do it with a PKGBUILD: # Maintainer: Andrew Crerar <crerar@archlinux.org>
# Maintainer: Morten Linderud <foxboron@archlinux.org>
# Contributor: Felix Yan <felixonmars@archlinux.org>
# Contributor: Iwan Timmer <irtimmer@gmail.com>
# Contributor: Vincent Demeester <vincent@sbr.io>
# Contributor: Josh VanderLinden <arch@cloudlery.com>
# Contributor: Anton Hvornum <torxed@archlinux.org>
pkgname=docker-compose-git
pkgver=2.24.6
pkgrel=1
pkgdesc="Fast, isolated development environments using Docker"
arch=('x86_64')
url="https://www.docker.com/"
license=("Apache-2.0")
makedepends=('go')
checkdepends=('docker')
commit='c5a88deeba6351c77b40301bdc3af9bafe5920ed' # Fixes network issues in v2.24.6-1
source=("$pkgname-$pkgver::git+https://github.com/docker/compose.git#commit=${commit}")
b2sums=('SKIP')
conflicts=("docker-compose")
provides=("docker-compose-git docker-compose")
build() {
ls -l
cd "$pkgname-$pkgver"
export CGO_CPPFLAGS="${CPPFLAGS}"
export CGO_CFLAGS="${CFLAGS}"
export CGO_CXXFLAGS="${CXXFLAGS}"
export CGO_LDFLAGS="${LDFLAGS}"
export GOFLAGS="-buildmode=pie -trimpath -ldflags=-linkmode=external -ldflags=-X=github.com/docker/compose/v2/internal.Version=${pkgver} -mod=readonly -modcacherw"
CGO_ENABLED=0 go build -trimpath -tags "e2e,kube" -o compose ./cmd
}
check(){
cd "$pkgname-$pkgver"
rm pkg/compose/ps_test.go
CGO_ENABLED=0 go test -tags "e2e,kube" -v $(go list -tags "e2e,kube" ./... | grep -vE 'e2e')
}
package() {
cd "$pkgname-$pkgver"
install -Dm644 LICENSE "$pkgdir"/usr/share/licenses/$pkgname/LICENSE
install -Dm755 compose "$pkgdir"/usr/lib/docker/cli-plugins/docker-compose
install -d "$pkgdir/usr/bin"
ln -sf /usr/lib/docker/cli-plugins/docker-compose "$pkgdir/usr/bin/docker-compose"
}It's +2 line modification of the official PKGBUILD, that just replaces the source and targets the commit they're mentioning. Run As for:
Yes, restarting fixes the race condition temporarily. But I tend to hit this after a full days worth of restarting services. It feels like it's the same amount of restarts as there are available IP addresses in the configured space until the issue triggers heh. But regarding c5a88de - with it I don't appear to hit the issue either. |
|
Had same issue on our project had to downgrade to 2.24.5 to solve my issues. |
|
Same issue here with 2.24.6. This is exactly the same like happened few months before (one network failed to attach randomly if multiple network defined). This was an issue once in 2.20.x, but it was fixed and closed! Downgraded to 2.24.5 and its ok again. |
|
I'm experiencing the same problem when I upgraded from 2.24.1 to 2.24.6. My Docker installation was using |
|
|
I also came across this same issue with an update to compose 2.24.6. I wonder if this is a bug that has been re-introduced in this version, as there was something similar in a previous version. |
I think it was version 2.19.1 that had the dns issue |
|
I also see the same issue when docker compose is upgraded from v2.24.5 to v2.24.6. I need to revert back to v2.24.5. |
|
Same for me also, seems similar to #10805 . Reverted to v2.24.5 and all good Ubuntu 22.04.4 LTS |
|
The problem first appeared a few days ago when I updated the compose plugin on my Debian machines. Docker Engine was not updated on that day (excerpt from Since then, I'm also getting the errors described here in #11563. |
|
I also had the same issue. Downgrading to 2.24.5 fixed the issue. |
|
btw, I find strange that stable versions of compose are released that depend on release candidates of other software. This could be particularly dangerous in some cases. Is there a particular reason why it is done like this? |
|
@marco-calautti they did mention to me here #11544 (comment) that they consider it prod-ready despite the RC tags, but I agree that these bugs and subtle breaking changes indicate the opposite. |
Are the issues due to the RC of the other software - or a internal bug of compose? |
|
Updating compose-go from rc7 to rc8 fixed the issue. So it looks like it's the RC's fault. |
@glours this fixed the issue for me. Every other restart of my containers had this issue. Now I have restarted four times (compose up/down) and all containers have every time been connected to all the networks they are supposed to be connected with. I was on Docker Compose v2.24.6 and Docker Engine - Community v25.0.3 Instructions for installing the fix on Ubuntu as an example, in case someone needs them: |
|
We just released Compose v2.24.7 which should resolve this. Thanks for your patience and the detailed reports! A bit of clarity on These projects/repositories are split to enable other tools to use the compose-go library without needing to depend on the full Compose app (this repo), which has a substantially larger dependency tree. API compatibility within Go modules is a challenging topic with very specific SemVer expectations from the Go tooling1. Recently, we made some more substantial changes to the API: adding These types of breaking changes are easy for us to coordinate & handle as the maintainers of both packages, but we couldn't practically version it as So, what you're seeing with the "pre-release" tags on compose-go is an artifact of that. Once we tag a "stable" v2 release, that prevents us from practically making further changes to the library's public API without another major release. tl;dr The RC status is an indication that the That said: regardless of version numbers, the stability & quality of Docker Compose is paramount, and we clearly had a miss here. We hugely appreciate the constructive & civil discussion on this issue; we've already covered some gaps in test coverage from these reports and identified additional areas, such as multi-network, where we need to improve E2E testing. Footnotes
|
|
Can we downgrade the Debian version (with apt pinning) to get docker compose plugin to work, until the newer version exists? If yes, which version is guaranteed to work? Edit: I see in the main post it says "Reverting to 2.24.5 fixes this problem". I'll give that a shot. |
Hello! |
Since today, the new v2.24.7 is in apt, at least for Ubuntu 20.04. The new version is working, I tested compose up/down five times, with 100% success rate. |
|
Hello, can I ask about the status of rolling out v2.24.7 in official MacOS packages? |
@rmlearney-digicatapult I managed to force an update of docker compose on mac with the following - had this is my history from the last time this issue occurred a few months back. |
|
Thanks @seanrabey - good workaround Just seems to be taking longer than usual to roll out an update with the fix |
|
Hello, any info about when v2.24.7 will be available via Docker Desktop for Windows ? |
|
@InTheNooB Compose |
2 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
and others
Description
This may be closely related to #11510 but it seemed different enough to open another issue.
Whenever I run docker-compose up -d and one or multiple containers in the compose has multiple networks, it often randomly fails to join exactly one network.
I have tried with internal, external and a mix of internal and external networks with anything between 3 and 10 networks and the result is always the same: About 60% of the time it fails to join exactly one network, the exact network is different. Tried on 2 different installations and both has the same result.
I see nothing relevant in the container logs, journalctl or docker event.
Reverting to 2.24.5 fixes this problem.
Steps To Reproduce
Doing the above 10 times, the container joined all 3 networks 4 times and then only 2 networks 6 times (the missing network seemed random, either test1, test2 or test3)
A quick way to test step 2 and 3:
docker stop whoami && docker rm whoami && docker compose up -d && docker container inspect -f '{{ len .NetworkSettings.Networks }}' whoami && docker container inspect -f '{{ range $key, $value := .NetworkSettings.Networks }}{{$key}} {{end}}' whoamiCompose Version
Docker Environment
Anything else?
No response
The text was updated successfully, but these errors were encountered: