[Bug]: Could not find Docker environment on macOS 12.6 with Docker For Mac

[martin-tarjanyi]

Module

Core

Testcontainers version

1.17.5

Using the latest Testcontainers version?

Yes

Host OS

macOS 12.6

Host Arch

Apple M1 Pro

Docker version

Client:
 Cloud integration: v1.0.29
 Version:           20.10.20
 API version:       1.41
 Go version:        go1.18.7
 Git commit:        9fdeb9c
 Built:             Tue Oct 18 18:20:35 2022
 OS/Arch:           darwin/arm64
 Context:           desktop-linux
 Experimental:      true

Server: Docker Desktop 4.13.0 (89412)
 Engine:
  Version:          20.10.20
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.7
  Git commit:       03df974
  Built:            Tue Oct 18 18:18:16 2022
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.8
  GitCommit:        9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

What happened?

I'm setting up a new Apple Macbook for development. I've installed the latest version of Docker for Mac. After that I've tried to run a maven build of a Java application using testcontainers. The build is failing because testcontainers cannot find the docker environment. It's likely a user error from my side but I couldn't figure out what is missing exactly. I'm able to interact with docker from command line.

Any help is appreciated. Thank you.

Relevant log output

13:50:04.396 [main] DEBUG org.testcontainers.utility.TestcontainersConfiguration - Testcontainers configuration overrides will be loaded from file:/Users/Martin_Tarjanyi/.testcontainers.properties
13:50:04.404 [main] WARN  org.testcontainers.utility.TestcontainersConfiguration - Attempted to read Testcontainers configuration file at file:/Users/Martin_Tarjanyi/.testcontainers.properties but the file was not found. Exception message: FileNotFoundException: /Users/Martin_Tarjanyi/.testcontainers.properties (No such file or directory)
13:50:04.412 [main] INFO  org.testcontainers.utility.ImageNameSubstitutor - Image name substitution will be performed by: DefaultImageNameSubstitutor (composite of 'ConfigurationFileImageNameSubstitutor' and 'PrefixingImageNameSubstitutor')
13:50:04.440 [main] DEBUG org.testcontainers.dockerclient.DockerClientProviderStrategy - Trying out strategy: UnixSocketClientProviderStrategy
13:50:04.441 [main] DEBUG org.testcontainers.dockerclient.DockerClientProviderStrategy - UnixSocketClientProviderStrategy: failed with exception InvalidConfigurationException (Could not find unix domain socket). Root cause NoSuchFileException (/var/run/docker.sock)
13:50:04.442 [main] INFO  org.testcontainers.dockerclient.DockerMachineClientProviderStrategy - docker-machine executable was not found on PATH ([/Users/Martin_Tarjanyi/.jenv/shims, /Users/Martin_Tarjanyi/.jenv/bin, /opt/homebrew/bin, /opt/homebrew/sbin, /usr/local/bin, /usr/bin, /bin, /usr/sbin, /sbin])
13:50:04.443 [main] ERROR org.testcontainers.dockerclient.DockerClientProviderStrategy - Could not find a valid Docker environment. Please check configuration. Attempted configurations were:
13:50:04.443 [main] ERROR org.testcontainers.dockerclient.DockerClientProviderStrategy -     UnixSocketClientProviderStrategy: failed with exception InvalidConfigurationException (Could not find unix domain socket). Root cause NoSuchFileException (/var/run/docker.sock)
13:50:04.443 [main] ERROR org.testcontainers.dockerclient.DockerClientProviderStrategy - As no valid configuration was found, execution cannot continue

java.lang.IllegalStateException: Could not find a valid Docker environment. Please see logs and check configuration

	at org.testcontainers.dockerclient.DockerClientProviderStrategy.lambda$getFirstValidStrategy$4(DockerClientProviderStrategy.java:157)
	at java.base/java.util.Optional.orElseThrow(Optional.java:403)
	at org.testcontainers.dockerclient.DockerClientProviderStrategy.getFirstValidStrategy(DockerClientProviderStrategy.java:149)
	at org.testcontainers.DockerClientFactory.getOrInitializeStrategy(DockerClientFactory.java:147)
	at org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:188)
	at org.testcontainers.DockerClientFactory$1.getDockerClient(DockerClientFactory.java:102)
	at com.github.dockerjava.api.DockerClientDelegate.authConfig(DockerClientDelegate.java:108)
	at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:325)
	at org.testcontainers.repro.ReproExampleTest.demonstration(ReproExampleTest.java:28)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
	at com.intellij.rt.junit.IdeaTestRunner$Repeater$1.execute(IdeaTestRunner.java:38)
	at com.intellij.rt.execution.junit.TestsRepeater.repeat(TestsRepeater.java:11)
	at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:35)
	at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:235)
	at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:54)

Additional Information

No response

[kiview]

There should be more relevant log output. Can you please run with this example project and share the logs? You can edit your original question accordingly, and comment on this answer once done.

[martin-tarjanyi]

@kiview Thanks for your help. I added the full logs in the original question.

In the meantime I found a possible culprit for the issue. In the release notes of Docker Desktop I see the following:

By default Docker will not create the /var/run/docker.sock symlink on the host and use the docker-desktop CLI context instead.

This might explain the issue I'm facing. What do you think?

[martin-tarjanyi]

I can confirm that downgrading to 4.12.0 solved the issue.

I guess on the long term there needs to be some fix on Docker Desktop or testcontainers side (actually IntelliJ Docker plugin was also affected).

[JakubZiolkowski]

Same. Thanks for saving my time :)

[eddumelendez]

Interesting... it works fine for me. I haven't do anything especial.

[martin-tarjanyi]

Did you update your existing Docker Desktop app or did you do a fresh installation? Maybe in case of update it doesn't delete the symlink. I did a fresh install when encountered the issue.

[eddumelendez]

it was an update. now that it works for you with 4.12, can you update, please?

[martin-tarjanyi]

It's still working after the update. So it looks like only fresh installs are affected.

[abialas]

I downgraded to version 4.12 (first I completely removed 4.13 previously installed). However, now when I run in terminal
docker ps I get:

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

When I try to run test with testcontainer I get:

ERROR org.testcontainers.dockerclient.DockerClientProviderStrategy - Could not find a valid Docker environment. Please check configuration. Attempted configurations were:
UnixSocketClientProviderStrategy: failed with exception InvalidConfigurationException (Could not find unix domain socket). Root cause NoSuchFileException (/var/run/docker.sock)As no valid configuration was found, execution cannot continue.

Seems for version 4.12 it does not create docker.sock for me as well.

[abialas]

I had to use Uninstall from Docker Desktop and then install it once again. I was following tutorials how to completely remove but seems I had to use Docker Desktop uninstall.

[jordi9]

I can confirm with a fresh install of Docker 4.13 the same issue happened to me.

The way I solved was by setting DOCKER_HOST to the address shown in docker context ls, eg: export DOCKER_HOST=unix:///Users/<username>/.docker/run/docker.sock

[eddumelendez]

From Docker Desktop Release Notes:

By default Docker will not create the /var/run/docker.sock symlink on the host and use the docker-desktop CLI context instead.

We have updated documentation to create the symlink manually but will work on a fix for the next release.

[boroda4436]

Colima causing some conflicts on my mac. So I removed Colima from my mac. Could you please tell the ETA for this bug fix?

[vaibhav-shimpi-maersk]

@boroda4436 - I am using colima for this test container on my mac by setting up the DockerEndPoint configuration to colima but still getting the error -
Docker.DotNet.DockerContainerNotFoundException
Docker API responded with status code=NotFound, response={"message":"No such container: 1d046098fd289c31fb130b0c18be1dc17922ed7d78a3135ca956d0c7ff05e9bf"}
Not sure whether the issue has been resolved or not yet for you?

[haydin505]

Following option fixed the issue:

[eddumelendez]

Additional update from Docker Desktop Release Notes:

Added back the /var/run/docker.sock symlink on Mac by default, to increase compatibility with tooling like tilt and docker-py. Fixes docker/for-mac#6529.

[parawanderer]

@eddumelendez I am (was) currently having this same issue but on Linux (Ubuntu 22.04) using Docker Desktop 4.17.0 and testcontainers version 1.17.2.

I see that on the documentation page you do not officially support Docker Desktop for Linux (different from Docker Engine). However following this thread and recalling that the Docker Desktop team had attempted to make the experience on all "Docker Desktop" versions similar across operating systems, I figured the solution for me might end up being similar to Mac users.

It turns out that the solution I found for my problem was similar: Running

sudo ln -s $HOME/.docker/desktop/docker.sock /var/run/docker.sock

Fixed the issue for me. Note the altered path (The one for Mac Docker Desktop is $HOME/.docker/run/docker.sock whereas the one for Linux Docker Desktop appears to be $HOME/.docker/desktop/docker.sock)

Could this be added to the testcontainers docs maybe?

I'm not sure if this issue should be passed to the Docker (Desktop) team to be added in the same way they did for Mac.

Edit: I have the same issue on version 1.18.0 (latest at the time of writing) too

[eddumelendez]

There is an issue #6426 about Docker Desktop for Linux

[lusorio]

That solved the issue in my machine as well (MacBook pro M1) running Docker 4.19.0 (106363) and testcontainers 1.17.6

[shakthizen]

Thanks

[julianSelser]

looks like we could add a strategy to extract the docker sock from docker context, I dont mind submitting a PR

[daggerok]

Hello, please let us know how we can workaround this? I'm using latest versions of Docker and test containers, and cannot run my tests which are works before because of this error:

Could not find a valid Docker environment. Please check configuration. Attempted configurations were: 
UnixSocketClientProviderStrategy: failed with exception InvalidConfigurationException (Could not find unix domain socket).
Root cause NoSuchFileException (/var/run/docker.sock)As no valid configuration was found, execution cannot continue.
See https://www.testcontainers.org/on_failure.html for more details. 

[eddumelendez]

os? I guess mac? Go to Settings > Advanced make sure Enable default Docker socket (Requires password) is checked.

[ph4r5h4d]

I'm using Quarkus 3.0.2 with test containers 1.17.6 which works fine, but upgrading to 1.18.1 now I'm seeing the following error while trying to run Integration tests
MacOs Ventura 13.3.1 (a)

INFO  [org.tes.uti.ImageNameSubstitutor] (main) [{}] Image name substitution will be performed by: DefaultImageNameSubstitutor (composite of 'ConfigurationFileImageNameSubstitutor' and 'PrefixingImageNameSubstitutor')
2023-05-16 10:57:38.765+02:00 [] [] [] INFO  [org.tes.doc.DockerClientProviderStrategy] (pool-6-thread-1) [{}] Loaded org.testcontainers.dockerclient.UnixSocketClientProviderStrategy from ~/.testcontainers.properties, will try it first
2023-05-16 10:57:38.830+02:00 [] [] [] INFO  [org.tes.doc.DockerMachineClientProviderStrategy] (pool-6-thread-1) [{}] docker-machine executable was not found on PATH ([MY PATHs])
2023-05-16 10:57:38.830+02:00 [] [] [] ERROR [org.tes.doc.DockerClientProviderStrategy] (pool-6-thread-1) [{}] Could not find a valid Docker environment. Please check configuration. Attempted configurations were:
	UnixSocketClientProviderStrategy: failed with exception NoClassDefFoundError (com/github/dockerjava/api/command/LoadImageAsyncCmd). Root cause ClassNotFoundException (com.github.dockerjava.api.command.LoadImageAsyncCmd)As no valid configuration was found, execution cannot continue.
See https://www.testcontainers.org/on_failure.html for more details.

[eddumelendez]

@ph4r5h4d that's a totally different issue. Make sure when upgrading to testcontainers-java 1.18.1 use docker-java 3.3.0 please.

[ph4r5h4d]

Just to be clear, I'm not importing docker-java directly in my pom.xml. This is a dependency for testcontainer,right? in that case should I explicitly define the version for docker-java or am I missing something here?

[grv95]

@eddumelendez
I have the docker socket enabled still unable to connect to Ryuk.
Mostly noticing this issue on Mac M1 machines, it runs fine for intel chips.

java.lang.IllegalStateException: Could not connect to Ryuk at 172.17.0.1:51501 at org.testcontainers.utility.RyukResourceReaper.maybeStart(RyukResourceReaper.java:132) at org.testcontainers.utility.RyukResourceReaper.init(RyukResourceReaper.java:42) at org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:209) at org.testcontainers.DockerClientFactory$1.getDockerClient(DockerClientFactory.java:89) at com.github.dockerjava.api.DockerClientDelegate.authConfig(DockerClientDelegate.java:108) at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:321)

[grv95]

Yes, it's enabled.

[eddumelendez]

Testcontainers for Java 1.18.2 is out. Please, give it a try. It contains some improvements to auto-detect the docker.sock

[Notyohclone]

I am having the same issue on Mac M1
testcontainers: 1.18.2
docker desktop 4.20.1
I am also trying to run testcontainers/ryuk:0.5.1 from IntelliJ

[Notyohclone]

Hey @eddumelendez any ideas on how to resolve this?

[kappaj2]

Just upgraded on normal Intel mac - same story.
Fix : sudo ln -s $HOME/.docker/run/docker.sock /var/run/docker.sock