TCP File Provider is not updating ip's resolved from DNS

[Enquier]

Bug

What did you do?

We setup (in docker swarm) a akka based system that has to cluster using independently defined seeds and hosts. These systems have a webserver and a TCP application level connection. When one of those nodes restarts, Traefik's TCP router appears to be caching the IP of that node (not resolving the DNS name on the fly) and is still pointing to the static IP. Since this is a swarm system there is no way for us to know the IP name a-priori

What did you expect to see?

Traefik addressing the node by DNS (or polling periodically) and not caching the IP

What did you see instead?

Traefik caches the IP and doesn't resolve or poll the dns name

Output of traefik version: (What version of Traefik are you using?)

Version:      2.2.10
Codename:     chevrotin
Go version:   go1.14.8
Built:        2020-09-04T15:53:27Z
OS/Arch:      linux/amd64

What is your environment & configuration (arguments, toml, provider, platform, ...)?

 
dynamic-service.yml
http:                                                                       
  routers:                                                                 
    https:                                                     
      entryPoints: ["https"]                                                
      service: https@file                                      
      tls: {}                                                              
      rule: Host(`hostname.com`) && PathPrefix(`/api`)
  services:
    https:
       loadBalancer:
          servers:
          - url: https://seed-1:8111/
          - url: https://seed-2:8111/
          - url: https://node-1:8111/
tcp:
  routers:
    tcp:
      entryPoints: ["tcp_endpoint"]
      service: tcp@file
      rule: HostSNI(`*`)
      tls:
        passthrough: true
  services:
    tcp:
      loadBalancer:
        servers:
        - address: seed-1:10002
        - address: seed-2:10002
        - address: node-1:10002

If applicable, please paste the log output in DEBUG level (--log.level=DEBUG switch)

Traefik logs:
time="2020-09-02T15:58:19Z" level=error msg="Error while connection to backend: dial tcp 10.0.3.245:10002: connect: no route to host",
time="2020-09-02T16:00:48Z" level=error msg="Error while connection to backend: dial tcp 10.0.3.249:10002: connect: no route to host",
time="2020-09-02T16:01:47Z" level=error msg="Error while connection to backend: dial tcp 10.0.3.251:10002: connect: no route to host",
time="2020-09-02T16:05:41Z" level=error msg="Error while connection to backend: dial tcp 10.0.3.245:10002: connect: no route to host",
time="2020-09-02T16:08:44Z" level=error msg="Error while connection to backend: dial tcp 10.0.3.249:10002: connect: no route to host",
time="2020-09-02T16:19:44Z" level=error msg="Error while connection to backend: dial tcp 10.0.3.251:10002: connect: no route to host"

Expected IP addresses:
Node-1: 10.0.3.66
Seed-1: 10.0.3.59
Seed-2: 10.0.3.64

[rtribotte]

Hello @Enquier,

Thanks for your interest in Traefik!

It's indeed the behavior for TCP routers, the address is resolved when the received configuration is applied, and this is why your facing this issue.

In your context, a workaround could be for instance to reload your configuration file at a fixed interval, using touch command or something else.

[Enquier]

@rtribotte We have developed a workaround which does the following:

1. Rewrite traefik entrypoint to enable crond
2. Add /1min crontab
3. Docker Config Mount a script to cron /1min to "poll" dynamic conf folder (via touch)

There are two problems with this approach

1. Traefik detects that the file is the same (reports: detected same configuration, skipping) and it is not clear if at that time it is re-resolving the dns or just ignoring the poll completely. So we plan to further develop a workaround where our services publish a new configuration via EFS with it's IP at startup.
2. The second problem, therefore is a feature request for polling (fixed via the workaround you mentioned and that we implemented) because EFS doesn't publish the "change" operation (that traefik appears to be listening to) to all hosts that mount it (just the performing host) so Traefik doesn't know to poll the file (this was inspired by Enable polling for mounted file system watching #7305 ).

[traefiker]

Closed by #7370.