Bug: in DinD, Since 4.0.0 postgres.get_connection_url() within another container gives broken connection url
#475
Comments
|
Hi! Thanks for reporting, you are welcome to add a test case for it in the postgres module. Cheers, |
|
How are you running that code in a container? I could have a look if I had a clean way to look at this in a debugger. Presumably this PR - #388 is the cause/regression. You might try setting the TC_HOST envar to your external host. |
|
Hello @totallyzen @bearrito thanks for the quick answers! On your questions:
{
"mounts": [
"source=/var/run/docker.sock,target=/var/run/docker.sock,type=bind"
],
}Instead of: {
"features": {
"ghcr.io/devcontainers/features/docker-in-docker:2": {
"version": "latest",
"dockerDashComposeVersion": "v2"
}
},
}I was able to reproduce with this change, your devcontainer and the test code piece above.
|
|
I can repro this. But it looks largely intentional based of the PR and some of the related TC projects. A maintainer could comment more. |
|
Heyo! It is somewhat intentional, but I wouldn't say that I personally have no interest in supporting The main matter is energy in reverse-engineering what the original code was doing.
Yeah the |
|
Hello @totallyzen no worries, breaking things along the way happens!
Otherwise I can design them out-of-CI first and document how to run that. Let me know what you think |
|
Hey! Thanks for clarifying, I wouldn't call myself a docker expert for sure. I did know about the two ways, but never bothered to understand the true difference. Life is short 😅 I'm actually keen on getting the socket mount version of the testing going anyway. I think it's valuable for catching future regressions and I see what you mean with test slowness and I have a few tricks up my sleeve for building the image once and then using it in a follow-up matrix (may not be super effective). Scope is definintely only I think it's important to boil down the issue to the host+port inference issue because I think this will be an issue with other containers that make use of the same inference. It's also worth noting there was another recent PR with #368 that should help. Correct me if I'm wrong, I'm still grasping networking issues on this matter |
|
I don't believe with Managed Runners you can run DIND in Github Actions? I know with self-hosted runners you can, but don't believe I've seen it with managed. Love to be proven wrong though. |
|
Hello, @totallyzen you're welcome :) So I just tested on
@bearrito I'd have to test with a |
|
sorry, reading this for the first time - not really a DinD expert, but bear with me, so to run dind, i think im going to: then i open a new terminal, and set up my python dev env? $ docker exec -it dind sh
$ apk add bash python3 && bash # install python, bash shell (and launch it)
$ python3 -m venv .pe && . .pe/bin/activate && pip install -U pip && pip install poetry && ln -s $(which poetry) /usr/bin && deactivate # install poetry in the poetry venv
$ apk add git && git clone https://github.com/testcontainers/testcontainers-python
$ cd testcontainers-python
$ poetry install --with devnow i am running testcontainers on dind? and i can write a test like: cat > core/tests/test_issue_475.py <<EOF
from testcontainers.postgres import PostgresContainer
def test_issue_475():
with PostgresContainer("postgres:alpine") as postgres:
print(postgres.get_connection_url())
EOFand it ( but it should print:
ifconfig output8a51aa4b9616:/testcontainers-python# ifconfig
docker0 Link encap:Ethernet HWaddr 02:42:5A:E1:AC:CD
inet addr:172.18.0.1 Bcast:172.18.255.255 Mask:255.255.0.0
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:21 errors:0 dropped:0 overruns:0 frame:0
TX packets:29 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1020 (1020.0 B) TX bytes:2122 (2.0 KiB)
eth0 Link encap:Ethernet HWaddr 02:42:AC:11:00:02
inet addr:172.17.0.2 Bcast:172.17.255.255 Mask:255.255.0.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:15635 errors:0 dropped:0 overruns:0 frame:0
TX packets:14403 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:209109459 (199.4 MiB) TX bytes:1290642 (1.2 MiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:32 errors:0 dropped:0 overruns:0 frame:0
TX packets:32 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2040 (1.9 KiB) TX bytes:2040 (1.9 KiB)the default route's gateway ip address is what is provided in the issue, but does that really make sense # route | grep ^default | awk '{ print $2 }'
172.17.0.1when i do that outside of docker, i get my router so i think what we actually want is the ip address of the interface which is our outbound route: # # inside dind container:
# apk add iproute2 # because busybox doesn't support "brief"
# route | grep ^default | awk '{ print $NF }' | xargs ip -brief addr show | awk '{ print $3 }' | cut -d'/' -f1
172.17.0.2
# outside docker:
$ route | grep ^default | awk '{ print $NF }' | xargs ip -brief addr show | awk '{ print $3 }' | cut -d'/' -f1
192.168.<omitted>.<omitted>and i thought that this brings back the terribleness of the ipv6 localhost address that macs report, but maybe not: $ # on macos:
$ route -n get default | sed -n '/interface:/{s/.*: //;p;}' | xargs ipconfig getifaddr
192.168.<omitted>.<omitted> |
|
interestingly enough, this one seems to give me the 172 address inside docker and a localhost address in linux, the LAN address in macos(??? - i mean 192.168), and then on windows, it gives two fe80's, an ipv6, and the LAN address also: python -c 'import socket; print(set([a[4][0] for a in socket.getaddrinfo(socket.gethostname(), None)]))'anyways, since all the other discussions were about docker abstraction layer, i figured maybe we can get somewhere with just basic networking command |
|
what im really hoping to understand is folks' dind setup - are the commands i sent realistic? i think the answer is no, somehow there are multiple containers involved. taking the "two scenarios" from above - docker engine in container, docker engine on host:
are you supposed to interact with the dind container from outside of it? e.g. if you use devcontainers: |
|
Hey @alexanderankin , Thanks for the digging! I think there is still a slight misanderstanding.
N.B:
My issue happens in the setup 2., and AFAIK not with setup 1. Hope it's clearer 🙏🏻 |
|
@alexanderankin If you are having a problem getting this setup, you can just use this projects devcontainers to get what you need. You can follow @Tanguy-LeFloch comment here - #475 (comment) Then tweak our file here - https://github.com/testcontainers/testcontainers-python/blob/main/.devcontainer/devcontainer.json#L6 @Tanguy-LeFloch Have you tried again since this commit - b10d916 |
|
@Tanguy-LeFloch message received - will run through scenario 2 asap/schedule permitting. @bearrito I (me) do not plan on using devcontainers until some poor shmuck (also potentially me 😄) figures out how to fix all the issues with it. jokes aside, i am extremely conservative (bordering on luddite) with the kind of software i introduce into my development stack for related reasons.
very good point, i will try before and after when i do |
alexanderankin
changed the title
postgres.get_connection_url() within another container gives broken connection urlpostgres.get_connection_url() within another container gives broken connection url
|
@bearrito @alexanderankin yes, unfortunately I can confirm that the problem in setup 2 ( |
|
As a workaround I am currently using: class FixedMySqlContainer(MySqlContainer):
def get_container_host_ip(self) -> str:
if inside_container() and Path("/run/docker.sock").exists():
return self.get_docker_client().gateway_ip(self._container.id)
return super().get_container_host_ip()which works well in the gitlab ci with a privileged runner. I couldn't reproduce it locally, as all connection to containers started inside my container fail, I don't know why... |
|
Your workaround works in my setup as well @CarliJoy thanks! I just needed to either disable I'm not sure what's the state of the tests to be sure that this works as well in the dind (#1 here #475 (comment)) setup. |
|
I have found that this part of code is commented in def get_container_host_ip(self) -> str:
# infer from docker host
host = self.get_docker_client().host()
if not host:
return "localhost"
# see https://github.com/testcontainers/testcontainers-python/issues/415
if host == "localnpipe" and system() == "Windows":
return "localhost"
# # check testcontainers itself runs inside docker container
# if inside_container() and not os.getenv("DOCKER_HOST") and not host.startswith("http://"):
# # If newly spawned container's gateway IP address from the docker
# # "bridge" network is equal to detected host address, we should use
# # container IP address, otherwise fall back to detected host
# # address. Even it's inside container, we need to double check,
# # because docker host might be set to docker:dind, usually in CI/CD environment
# gateway_ip = self.get_docker_client().gateway_ip(self._container.id)
# if gateway_ip == host:
# return self.get_docker_client().bridge_ip(self._container.id)
# return gateway_ip
return hostIt seems similar to the hack @CarliJoy suggest , btw it also works for me, thanks for sharing it |
Describe the bug
Before
4.0.0, after creating aPostgresContainerwithin another container, but with the same external daemon,postgres.get_connection_url()would return somethinglike
postgresql+psycopg2://test:test@172.17.0.1:32770/testwhich is the correct connection url.However, since
4.0.0, the exact same container'spostgres.get_connection_url()returnspostgresql+psycopg2://test:test@localhost:5432/test.So it changed:
Considering that the newly formatted connection url doesn't work to create an engine, I wouldn't expect this behavior.
To Reproduce
Runtime environment
Operating system:
Linux d388e0f17614 5.10.179-168.710.amzn2.x86_64 #1 SMP Mon May 22 23:10:22 UTC 2023 x86_64 GNU/LinuxPython version:
Python 3.11.5docker info
pip freeze
The text was updated successfully, but these errors were encountered: