Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug: Healthcheck not working for docker in docker. #856

Open
zbindenren opened this issue Mar 29, 2023 · 7 comments
Open

Bug: Healthcheck not working for docker in docker. #856

zbindenren opened this issue Mar 29, 2023 · 7 comments

Comments

@zbindenren
Copy link
Contributor

zbindenren commented Mar 29, 2023

Describe the bug
Our go tests run in docker containers via gitlab runners. In order that this works the docker socket is mapped inside the container. Everythink works (container starts), but I get a timeout because the health check tries to connect via localhost.

"L":"INFO","T":"2023-03-29T13:38:24.964Z","M":"connecting to docker engine","id":"8e1ded32-14cc-4ee0-b1e0-c681beaf7f8d"}
181{"L":"INFO","T":"2023-03-29T13:38:24.964Z","M":"connected to docker engine","id":"8e1ded32-14cc-4ee0-b1e0-c681beaf7f8d"}
182{"L":"INFO","T":"2023-03-29T13:38:24.964Z","M":"starting container","id":"8e1ded32-14cc-4ee0-b1e0-c681beaf7f8d"}
183{"L":"INFO","T":"2023-03-29T13:38:24.964Z","M":"pulling image","id":"8e1ded32-14cc-4ee0-b1e0-c681beaf7f8d"}
184{"L":"INFO","T":"2023-03-29T13:38:29.730Z","M":"image pulled","id":"8e1ded32-14cc-4ee0-b1e0-c681beaf7f8d"}
185{"L":"INFO","T":"2023-03-29T13:38:30.387Z","M":"waiting for container network","id":"8e1ded32-14cc-4ee0-b1e0-c681beaf7f8d","container":"e77f31af6f467f38d6f4fbfbfe7226911b6d6a924fd02c00d8567695a924ef00"}
186{"L":"INFO","T":"2023-03-29T13:38:30.638Z","M":"waiting for port allocation","id":"8e1ded32-14cc-4ee0-b1e0-c681beaf7f8d","container":"e77f31af6f467f38d6f4fbfbfe7226911b6d6a924fd02c00d8567695a924ef00"}
187{"L":"INFO","T":"2023-03-29T13:38:30.639Z","M":"container started","id":"8e1ded32-14cc-4ee0-b1e0-c681beaf7f8d","container":{"id":"e77f31af6f467f38d6f4fbfbfe7226911b6d6a924fd02c00d8567695a924ef00","host":"127.0.0.1","ports":{"default":{"protocol":"tcp","port":32768,"host_port":0}}}}
188{"L":"INFO","T":"2023-03-29T13:38:30.639Z","M":"starting container logs forwarder","id":"8e1ded32-14cc-4ee0-b1e0-c681beaf7f8d"}
189{"L":"INFO","T":"2023-03-29T13:38:30.639Z","M":"container logs forwarder ready","id":"8e1ded32-14cc-4ee0-b1e0-c681beaf7f8d"}
190{"L":"INFO","T":"2023-03-29T13:38:30.639Z","M":"waiting for healthcheck to pass","id":"8e1ded32-14cc-4ee0-b1e0-c681beaf7f8d"}
191{"L":"INFO","T":"2023-03-29T13:38:34.519Z","M":"healthcheck failed: Get \"http://127.0.0.1:32768/v1/sys/health?drsecondarycode=299&performancestandbycode=299&sealedcode=299&standbycode=299&uninitcode=299\": dial tcp 127.0.0.1:32768: connect: connection refused","id":"8e1ded32-14cc-4ee0-b1e0-c681beaf7f8d"}
192{"L":"INFO","T":"2023-03-29T13:38:37.875Z","M":"healthcheck failed: Get \"http://127.0.0.1:32768/v1/sys/health?drsecondarycode=299&performancestandbycode=299&sealedcode=299&standbycode=299&uninitcode=299\": dial tcp 127.0.0.1:32768: connect: connection refused","id":"8e1ded32-14cc-4ee0-b1e0-c681beaf7f8d"}
193{"L":"INFO","T":"2023-03-29T13:38:42.169Z","M":"healthcheck failed: Get \"http://127.0.0.1:32768/v1/sys/health?drsecondarycode=299&performancestandbycode=299&sealedcode=299&standbycode=299&uninitcode=299\": dial tcp 127.0.0.1:32768: connect: connection refused","id":"8e1ded32-14cc-4ee0-b1e0-c681beaf7f8d"}
194{"L":"INFO","T":"2023-03-29T13:38:45.943Z","M":"healthcheck failed: Get \"http://127.0.0.1:32768/v1/sys/health?drsecondarycode=299&performancestandbycode=299&sealedcode=299&standbycode=299&uninitcode=299\": dial tcp 127.0.0.1:32768: connect: connection refused","id":"8e1ded32-14cc-4ee0-b1e0-c681beaf7f8d"}

It would be nice to have a possibility to set the Host to a something else than 127.0.0.1 (ip of the host where docker runs on in my case) for the health checks. Then I think everyhing would work.

Am I missing something?

@orlangure
Copy link
Owner

Hi @zbindenren and sorry for the delay. Gnomock supports DOCKER_HOST environment variable, but I'm not sure it works correctly in docker-in-docker configuration. Can you try that please?

@zbindenren
Copy link
Contributor Author

Hi @orlangure

If I set the DOCKER_HOST variable I get the following error:

can't start container: can't prepare container: can't pull image: can't pull image: Cannot connect to the Docker daemon at tcp://xxxxxxx:2375. Is the docker daemon running?

because we do not listen on a tcp port (only docker socket).

I even changed the health check to use the correct runner, but I still get a connection refused.

I wonder what this line says:

{"L":"INFO","T":"2023-04-11T06:41:03.798Z","M":"container started","id":"d9926324-0529-45b9-a96b-447451a32ca7","container":{"id":"cd9cf2ca96418cc762699d181f5509adeea3d2823d5ddfa0d6ee209b0e25bc9b","host":"127.0.0.1","ports":{"default":{"protocol":"tcp","port":32768,"host_port":0}}}}

Does "host":"127.0.0.1" mean, that it listens only on localhost? It is strange that all works with dockertest.

@orlangure
Copy link
Owner

Another thing to try is WithHostMounts to mount you docker socket into the container, so it can access external docker daemon thinking it's the local one. You might need to use this in combination with DOCKER_HOST that will affect the network address of the container.

@aurelius15
Copy link

Faced with the same issue

@orlangure
Copy link
Owner

Hey @aurelius15,
Have you tried any of the above suggestions?

@aurelius15
Copy link

Hi orlangure,

Yes, I tried mentioned suggestions, and they did not work. But after going through sources, I found a solution for my case.

I have added GNOMOCK_ENV variable to my ci configuration and also added a few lines of code.

integration-test:
 ....
  variables:
    GNOMOCK_ENV: "gnomockd"
// in your test
if isInDocker() {
                  // db-host = fmt.Sprintf("%s:%d", "172.17.0.1", dbContainer.DefaultPort())
                 // 172.17.0.1 is a value of property gateway in *gnomock.Container 
	} else {
		 // db-host = dbContainer.DefaultAddress()
}
...
func isInDocker() bool {
	env := os.Getenv("GNOMOCK_ENV")
	return env == "gnomockd"
}

The tricky part is that healthcheck finds the correct IP address if GNOMOCK_ENV is present but the rest of code no. I think it might be fixed in hostAddr via adding some parts from envAwareClone or just making the gateway property public.

@orlangure
Copy link
Owner

Have you tried setting docker host variable to your host address?

I'll need a reproducible example to help more, something I can run and debug.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants