Testcontainers do NOT work on kubernetes self hosted runners, but they DO work on VM self hosted runners and locally

[robgutsopedra]

I'm running the tests on exactly the same scenario (same image, resources, code...), with a difference:
On VM-self-hosted-runners and/or locally, everything works as expected
On Kubernetes-self-hosted-runners(https://github.com/actions/actions-runner-controller/tree/master), it doesn't

After some debugging, I assume the problem is here:

import { GenericContainer } from 'testcontainers';
import { RedisHelper } from './utils/redis.helper';

let container;

export let host: string;
export let port: string;
export let dbNumber: string;
export let redisHelper: RedisHelper;

beforeAll(async () => {
    console.log('Starting Redis container'); ##THIS IS PROPERLY LOGGED
    container = await new GenericContainer('redis').withExposedPorts(6379).start(); ##FAILS HERE
    console.log('Redis container started'); ##THIS IS NOT
    console.log(container.getHost());
    console.log(container.getMappedPort(6379).toString());

    redisHelper = new RedisHelper();
    await redisHelper.connectToRedis(container.getHost(), container.getMappedPort(6379).toString(), '0');
    process.env.REDIS_HOST = container.getHost();
    process.env.REDIS_PORT = container.getMappedPort(6379).toString();
});

Expected Behaviour
Containers should be able to communicate properly

Actual Behaviour
Connection is never established

Testcontainer Logs
2023-09-20T14:26:44.277Z testcontainers [DEBUG] [e15b38c0264b] Demuxing stream... 2023-09-20T14:26:44.278Z testcontainers [DEBUG] [e15b38c0264b] Demuxed stream 2023-09-20T14:26:44.278Z testcontainers [DEBUG] [e15b38c0264b] Fetched container logs 2023-09-20T14:26:44.281Z testcontainers [DEBUG] [e15b38c0264b] Log wait strategy complete 2023-09-20T14:26:44.282Z testcontainers [INFO] [e15b38c0264b] Container is ready 2023-09-20T14:26:44.283Z testcontainers [DEBUG] [e15b38c0264b] Connecting to Reaper (attempt 1) on "localhost:32780"... 2023-09-20T14:26:44.284Z testcontainers [ERROR] Reaper e15b38c0264b7d35def0ce57224ce2fd291d11d351768bc8468fe96cb89f0a15 socket error: Error: connect ECONNREFUSED 127.0.0.1:32780 2023-09-20T14:26:44.285Z testcontainers [ERROR] [e15b38c0264b] Connection to Reaper closed with error 2023-09-20T14:26:45.285Z testcontainers [DEBUG] [e15b38c0264b] Connecting to Reaper (attempt 2) on "localhost:32780"... 2023-09-20T14:26:45.286Z testcontainers [ERROR] Reaper e15b38c0264b7d35def0ce57224ce2fd291d11d351768bc8468fe96cb89f0a15 socket error: Error: connect ECONNREFUSED 127.0.0.1:32780 2023-09-20T14:26:45.286Z testcontainers [ERROR] [e15b38c0264b] Connection to Reaper closed with error 2023-09-20T14:26:46.286Z testcontainers [DEBUG] [e15b38c0264b] Connecting to Reaper (attempt 3) on "localhost:32780"... 2023-09-20T14:26:46.287Z testcontainers [ERROR] Reaper e15b38c0264b7d35def0ce57224ce2fd291d11d351768bc8468fe96cb89f0a15 socket error: Error: connect ECONNREFUSED 127.0.0.1:32780 2023-09-20T14:26:46.287Z testcontainers [ERROR] [e15b38c0264b] Connection to Reaper closed with error 2023-09-20T14:26:47.287Z testcontainers [DEBUG] [e15b38c0264b] Connecting to Reaper (attempt 4) on "localhost:32780"... 2023-09-20T14:26:47.288Z testcontainers [ERROR] Reaper e15b38c0264b7d35def0ce57224ce2fd291d11d351768bc8468fe96cb89f0a15 socket error: Error: connect ECONNREFUSED 127.0.0.1:32780 2023-09-20T14:26:47.288Z testcontainers [ERROR] [e15b38c0264b] Connection to Reaper closed with error 2023-09-20T14:26:48.289Z testcontainers [DEBUG] [e15b38c0264b] Connecting to Reaper (attempt 5) on "localhost:32780"... 2023-09-20T14:26:48.290Z testcontainers [ERROR] Reaper e15b38c0264b7d35def0ce57224ce2fd291d11d351768bc8468fe96cb89f0a15 socket error: Error: connect ECONNREFUSED 127.0.0.1:32780 2023-09-20T14:26:48.290Z testcontainers [ERROR] [e15b38c0264b] Connection to Reaper closed with error 2023-09-20T14:26:48.290Z testcontainers [ERROR] [e15b38c0264b] Failed to connect to Reaper 2023-09-20T14:26:48.290Z testcontainers [DEBUG] Releasing lock file "/tmp/testcontainers-node.lock"... 2023-09-20T14:26:48.290Z testcontainers [DEBUG] Released lock file "/tmp/testcontainers-node.lock"

Steps to Reproduce

1. Connect to the runner Pod with kubectl exec -it [POD] /bin/bash
2. git clone {repo with tests}
3. curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.3/install.sh | bash + refresh session (in case your image does not include node/npm/etc)
4. cd repo + nvm install 18
5. npm install; npm run test — And it FAILS - logs above.

Environment Information

• Operating System: Ubuntu 22
• Docker Version: Docker version 24.0.5, build ced0996
• Node version: v18.18.0
• Testcontainers version: ^10.2.1

[strahinjatasic]

Any solutions on this one?

[cristianrgreco]

How was Docker installed on the pods? Could you try these environment variables:

TESTCONTAINERS_RYUK_PRIVILEGED=false

If that doesn't work try

TESTCONTAINERS_RYUK_DISABLED=true

Also please share all of the logs, not just the error.

[ekaitzht]

This env variable fix to me this issue.

TESTCONTAINERS_RYUK_DISABLED=true

PS: I am using Colima instead Docker stop for MacosX