New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Bug]: TestGenericReusableContainerInSubprocess test is flaky #2348
Comments
Fix waitForPruneCondition timeout handling which was not resetting the connection timeout when new connections came in resulting in the reaper shutting down incorrectly. This was causing random failures as detailed by: testcontainers/testcontainers-go#2348 Don't log EOF errors. Add buffer to connection channels so we don't block the accepting goroutine.
Fix waitForPruneCondition timeout handling which was not resetting the connection timeout when new connections came in resulting in the reaper shutting down incorrectly. Don't log EOF errors. Add buffer to connection channels so we don't block the accepting goroutine. Fixes testcontainers/testcontainers-go#2348
PR is now open against moby-ryuk which fixes the underlying issue which is that the reaper container doesn't reset it's waiting for connection timeout, which means it will randomly shutdown while in use if its connection count ever reaches zero. Once that's merged and a new release made, we'll need to PR to bump the container image version. |
Testing locally I can now run 100 iterations without fail, where as previously 10 was usually enough to trigger an issue. go test -run=TestGenericReusableContainerInSubprocess -count=100 -v -failfast |
Super deep investigation, thanks for doing this! |
No problem @mdelapenya just been tripping over issues trying to get a clean test run, so now have a few PR's in flight. Looks like someone authorised the test run which has now passed, don't suppose you have the power to do a merge and release so I can get the image PR done do you as this is going to be causing all sorts of issues so would be good to get it in? |
Should we re-open until we have a release and PR to bump the container version? |
Reopening until the next release of Ryuk, which will solve this issue |
@mdelapenya not sure if you intended to close this with the PR against a fork of moby-ryuk? |
Oh my! I pushed to my origin right after updating ryuk and it seems it triggered the auto-close. Reopening 🤦 |
Testcontainers version
fe0d3a8
Using the latest Testcontainers version?
Yes
Host OS
Linux
Host arch
amd64
Go version
1.22
Docker version
Client: Cloud integration: v1.0.35+desktop.11 Version: 25.0.3 API version: 1.44 Go version: go1.21.6 Git commit: 4debf41 Built: Tue Feb 6 21:13:00 2024 OS/Arch: linux/amd64 Context: default Server: Docker Desktop Engine: Version: 25.0.3 API version: 1.44 (minimum version 1.24) Go version: go1.21.6 Git commit: f417435 Built: Tue Feb 6 21:14:25 2024 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.6.28 GitCommit: ae07eda36dd25f8a1b98dfbf587313b99c0190bb runc: Version: 1.1.12 GitCommit: v1.1.12-0-g51d5e94 docker-init: Version: 0.19.0 GitCommit: de40ad0
Docker info
What happened?
Running TestGenericReusableContainerInSubprocess randomly fails.
go test -run=TestGenericReusableContainerInSubprocess -count=10 -v -failfast
Relevant log output
Additional information
This is a bug in the reaper, PR incoming.
The text was updated successfully, but these errors were encountered: