Use health status when looking up reaper (ryuk) container #2508
+15
−0
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
✅ Deploy Preview for testcontainers-go ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
This was referenced Apr 22, 2024
|
Hey team, just bumping this PR. Would love to hear feedback and if there's anything I can do to help move it forward 🙏 Thanks for all your efforts on Testcontainers! Great project. |
Hi @emetsger I cannot reproduce this yet: I've run run it even with count=10, with no errors. Is there anything you think I can do to reproduce it? |
|
Thank you for trying to reproduce!
I’m using Rancher, so I wonder if that may be contributing to the
discrepancy.
What container engine/version are you on? I’ll be back at my computer
later this weekend and I can do some more testing on my side. Thank you!
…On Fri, Apr 26, 2024 at 11:35 AM Manuel de la Peña ***@***.***> wrote:
Clone this test case repo
<https://github.com/emetsger/testcontainers-ryuk-testcase> and run go
test -count=1 -v ./... and observe the errors.
Hi @emetsger <https://github.com/emetsger> I cannot reproduce this yet:
go test -count=1 -v ./...
=== RUN TestA
2024/04/26 17:27:38 github.com/testcontainers/testcontainers-go - Connected to docker:
Server Version: 78+testcontainerscloud (via Testcontainers Desktop 1.10.1)
API Version: 1.43
Operating System: Ubuntu 22.04.4 LTS
Total Memory: 15779 MB
Resolved Docker Host: tcp://127.0.0.1:54509
Resolved Docker Socket Path: /var/run/docker.sock
Test SessionID: a077cc8ef4acf0841cd4b227aeac15cf8611190b47530d735cb90d3993d5d084
Test ProcessID: b81dbac4-46a1-4e91-a577-a7e47d57518e
2024/04/26 17:27:38 🔥 Reaper obtained from Docker for this test session 494993ccefba558a0d64b9ddce7d8db30444e35a49bf12d9cbd8ef205c23e05f
2024/04/26 17:27:38 🐳 Creating container for image docker.io/postgres:16-alpine
2024/04/26 <http://docker.io/postgres:16-alpine2024/04/26> 17:27:38 ✅ Container created: 0b7ae9cbdeb1
2024/04/26 17:27:38 🐳 Starting container: 0b7ae9cbdeb1
2024/04/26 17:27:39 ✅ Container started: 0b7ae9cbdeb1
2024/04/26 17:27:39 🚧 Waiting for container id 0b7ae9cbdeb1 image: docker.io/postgres:16-alpine. Waiting for: &{timeout:<nil> deadline:0x140003c6e88 Strategies:[0x140003e2c30]}
2024/04/26 17:27:41 🔔 Container is ready: 0b7ae9cbdeb1
2024/04/26 17:27:41 🐳 Terminating container: 0b7ae9cbdeb1
2024/04/26 17:27:41 🚫 Container terminated: 0b7ae9cbdeb1
--- PASS: TestA (3.65s)
PASS
ok tc-debug/a 4.012s
=== RUN TestB
2024/04/26 17:27:37 github.com/testcontainers/testcontainers-go - Connected to docker:
Server Version: 78+testcontainerscloud (via Testcontainers Desktop 1.10.1)
API Version: 1.43
Operating System: Ubuntu 22.04.4 LTS
Total Memory: 15779 MB
Resolved Docker Host: tcp://127.0.0.1:54509
Resolved Docker Socket Path: /var/run/docker.sock
Test SessionID: a077cc8ef4acf0841cd4b227aeac15cf8611190b47530d735cb90d3993d5d084
Test ProcessID: 2a7fd6eb-d055-4e31-aaa6-00cf7cd47542
2024/04/26 17:27:37 🐳 Creating container for image testcontainers/ryuk:0.7.0
2024/04/26 17:27:38 ✅ Container created: 494993ccefba
2024/04/26 17:27:38 🐳 Starting container: 494993ccefba
2024/04/26 17:27:38 ✅ Container started: 494993ccefba
2024/04/26 17:27:38 🚧 Waiting for container id 494993ccefba image: testcontainers/ryuk:0.7.0. Waiting for: &{Port:8080/tcp timeout:<nil> PollInterval:100ms}
2024/04/26 17:27:38 🔔 Container is ready: 494993ccefba
2024/04/26 17:27:38 🐳 Creating container for image docker.io/postgres:16-alpine
2024/04/26 <http://docker.io/postgres:16-alpine2024/04/26> 17:27:38 ✅ Container created: 8386f359812f
2024/04/26 17:27:38 🐳 Starting container: 8386f359812f
2024/04/26 17:27:39 ✅ Container started: 8386f359812f
2024/04/26 17:27:39 🚧 Waiting for container id 8386f359812f image: docker.io/postgres:16-alpine. Waiting for: &{timeout:<nil> deadline:0x1400037ef28 Strategies:[0x1400039ebd0]}
2024/04/26 17:27:41 🔔 Container is ready: 8386f359812f
2024/04/26 17:27:41 🐳 Terminating container: 8386f359812f
2024/04/26 17:27:41 🚫 Container terminated: 8386f359812f
--- PASS: TestB (3.92s)
PASS
ok tc-debug/b 4.104s
=== RUN TestC
2024/04/26 17:27:38 github.com/testcontainers/testcontainers-go - Connected to docker:
Server Version: 78+testcontainerscloud (via Testcontainers Desktop 1.10.1)
API Version: 1.43
Operating System: Ubuntu 22.04.4 LTS
Total Memory: 15779 MB
Resolved Docker Host: tcp://127.0.0.1:54509
Resolved Docker Socket Path: /var/run/docker.sock
Test SessionID: a077cc8ef4acf0841cd4b227aeac15cf8611190b47530d735cb90d3993d5d084
Test ProcessID: 0b1569fa-f57c-46af-afe3-b26dc818fbc8
2024/04/26 17:27:38 🔥 Reaper obtained from Docker for this test session 494993ccefba558a0d64b9ddce7d8db30444e35a49bf12d9cbd8ef205c23e05f
2024/04/26 17:27:38 🐳 Creating container for image docker.io/postgres:16-alpine
2024/04/26 <http://docker.io/postgres:16-alpine2024/04/26> 17:27:38 ✅ Container created: b2b8530b534d
2024/04/26 17:27:38 🐳 Starting container: b2b8530b534d
2024/04/26 17:27:38 ✅ Container started: b2b8530b534d
2024/04/26 17:27:38 🚧 Waiting for container id b2b8530b534d image: docker.io/postgres:16-alpine. Waiting for: &{timeout:<nil> deadline:0x1400037ef28 Strategies:[0x1400039ebd0]}
2024/04/26 17:27:40 🔔 Container is ready: b2b8530b534d
2024/04/26 17:27:40 🐳 Terminating container: b2b8530b534d
2024/04/26 17:27:41 🚫 Container terminated: b2b8530b534d
--- PASS: TestC (3.28s)
PASS
ok tc-debug/c 3.756s
=== RUN TestD
2024/04/26 17:27:38 github.com/testcontainers/testcontainers-go - Connected to docker:
Server Version: 78+testcontainerscloud (via Testcontainers Desktop 1.10.1)
API Version: 1.43
Operating System: Ubuntu 22.04.4 LTS
Total Memory: 15779 MB
Resolved Docker Host: tcp://127.0.0.1:54509
Resolved Docker Socket Path: /var/run/docker.sock
Test SessionID: a077cc8ef4acf0841cd4b227aeac15cf8611190b47530d735cb90d3993d5d084
Test ProcessID: bb20c031-95bb-4791-af3b-e4b1ecf5d818
2024/04/26 17:27:38 🔥 Reaper obtained from Docker for this test session 494993ccefba558a0d64b9ddce7d8db30444e35a49bf12d9cbd8ef205c23e05f
2024/04/26 17:27:38 🐳 Creating container for image docker.io/postgres:16-alpine
2024/04/26 <http://docker.io/postgres:16-alpine2024/04/26> 17:27:38 ✅ Container created: 2f51f8a61406
2024/04/26 17:27:38 🐳 Starting container: 2f51f8a61406
2024/04/26 17:27:39 ✅ Container started: 2f51f8a61406
2024/04/26 17:27:39 🚧 Waiting for container id 2f51f8a61406 image: docker.io/postgres:16-alpine. Waiting for: &{timeout:<nil> deadline:0x140003f8e58 Strategies:[0x14000458bd0]}
2024/04/26 17:27:41 🔔 Container is ready: 2f51f8a61406
2024/04/26 17:27:41 🐳 Terminating container: 2f51f8a61406
2024/04/26 17:27:41 🚫 Container terminated: 2f51f8a61406
--- PASS: TestD (3.32s)
PASS
ok tc-debug/d 3.933s
I've run run it even with count=10, with no errors. Is there anything you
think I can do to reproduce it?
—
Reply to this email directly, view it on GitHub
<#2508 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AABD4GRM6OFYD2Q3WX5QWJDY7JX5DAVCNFSM6AAAAABGR5VJXSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZZGYZDQOJRHA>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
I'm using both Docker for Mac and Testcontainers Cloud. |
|
I am experiencing a similar issue in (Jenkins) CI, although it's flaky - only on some runs there is a connection refused error as follows. Note that I am unable to reproduce this on my machine (with different container engine & version). |
|
Do you know what docker engine/version is running in your CI?
…On Thu, May 2, 2024 at 6:00 AM Simon Bos ***@***.***> wrote:
I am experiencing a similar issue in (Jenkins) CI, although it's flaky -
only on some runs there is a connection refused error as follows.
[2024-05-02T09:39:21.954Z] 2024/05/02 09:39:12 github.com/testcontainers/testcontainers-go - Connected to docker:
[2024-05-02T09:39:21.954Z] Server Version: 20.10.23
[2024-05-02T09:39:21.954Z] API Version: 1.41
[2024-05-02T09:39:21.954Z] Operating System: Rocky Linux 9.2 (Blue Onyx)
[2024-05-02T09:39:21.954Z] Total Memory: 15982 MB
[2024-05-02T09:39:21.954Z] Resolved Docker Host: unix:///var/run/docker.sock
[2024-05-02T09:39:21.954Z] Resolved Docker Socket Path: /var/run/docker.sock
[2024-05-02T09:39:21.954Z] Test SessionID: 9644673c1b6ea45d4962460bb5e7a2a9ba32c1f293bf4166d0e43de70c021a86
[2024-05-02T09:39:21.954Z] Test ProcessID: 05a16b84-4e96-4b61-ac3d-29f63de4d258
[2024-05-02T09:39:21.954Z] 2024/05/02 09:39:12 🐳 Creating container for image testcontainers/ryuk:0.7.0
[2024-05-02T09:39:21.954Z] 2024/05/02 09:39:13 🔥 Reaper obtained from Docker for this test session 1179229b008b48672081fd3372999389793505d2b013d054c454a6838a6b5544
[2024-05-02T09:39:21.954Z] <redacted>: dial tcp <redacted>:35412: connect: connection refused: Connecting to Ryuk on <redacted>:35412 failed: connecting to reaper failed: failed to create container
Note that I am unable to reproduce this on my machine (with different
container engine & version).
—
Reply to this email directly, view it on GitHub
<#2508 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AABD4GQZUZCQO7VRD4VO4U3ZAIFEHAVCNFSM6AAAAABGR5VJXSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOJQGA3TCMRYGE>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
It's running "Docker Engine - Community" version 20.10.23. There is some extra info in the logs above. |
|
@simonbos what docker engine/version are you running on your local machine? |
(link to gist) |
|
@mdelapenya I ran the test case against a variety of container managers/docker engines (see table above) I was able to successfully run my test case against all the container managers except for Rancher.
@mdelapenya can you install Rancher 1.13.1 (latest at the time of this writing) and attempt to reproduce on your side? |
|
Wow! @emetsger thank you so much for dedicating that amount of time to reproduce the potential bug in so many container environments. Very proud of you 🙇 I'll install rancher today and try to reproduce it. Will ping you back here |
Bablzz
reviewed
May 6, 2024
docker.go
Outdated
| @@ -49,6 +49,11 @@ const ( | |||
| packagePath = "github.com/testcontainers/testcontainers-go" | |||
|
|
|||
| logStoppedForOutOfSyncMessage = "Stopping log consumer: Headers out of sync" | |||
|
|
|||
| healthStatusNone = "" // default status for a container with no healthcheck | |||
There was a problem hiding this comment.
hello @emetsger ! up to you, can use here status from docker/types?
who knows maybe docker change name status in the future:)
There was a problem hiding this comment.
👍
docker/types defines types.NoHealthcheck = "none". I did not see a constant for a health check value equal to the zero-value string.
So, to be safe, I check for a zero-length string before checking the health status value:
// if a health status is present on the container, and the container is not healthy, error
if r.healthStatus != "" {
if r.healthStatus != types.Healthy && r.healthStatus != types.NoHealthcheck {
return fmt.Errorf("container %s is not healthy, wanted status=%s, got status=%s", resp[0].ID[:8], types.Healthy, r.healthStatus)
}
}
It's the least I can do: trying to pay it forward! |
|
Thanks for the effort @emetsger !
Docker Desktop for Mac v4.29.0. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Updates
lookUpReaperContainerto consider the health of the ryuk container.lookUpReaperContainerwill require a healthy container before returning.lookUpReaperContainerwill return the container.Why is it important?
The ryuk container can be exposed by the Docker API before it is ready to accept connections, resulting in messages like:
Tests will fail when they would otherwise pass.
This is especially the case in Go, where test binaries are run in parallel, one for each package. If multiple packages use
testcontainers, they can obtain a reaper instance from the Docker API before the container is ready to serve connections, leading to dialing errors likeconnect: connection refused.The current workaround is to run test binaries serially, e.g.,
go test -p 1 ./..., which is not optimal, even for modest codebases.Related issues
See complementary PR, which adds a
HEALTHCHECKto the ryuk container: testcontainers/moby-ryuk#128How to test this PR
To recreate the initial problem and run the workaround
Clone this test case repo and run
go test -count=1 -v ./...and observe the errors.Run
go test -count=1 -p 1 -v ./...and tests should pass.To test the fix
Clone moby-ryuk, apply pull/128, then build a new ryuk container, and tag it:
docker build -f linux/Dockerfile -t local/ryuk .docker tag local/ryuk testcontainers/ryuk:0.7.0Check out this branch/PR locally.
Clone the test case repo and update
go.modto point to this branch via a replace, e.g.:replace github.com/testcontainers/testcontainers-go => /Users/esm/workspaces/testcontainers-goRunning
go test -count=1 -v ./...should pass.