Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add systemd socket activation support #9891

Open
lalbers opened this issue May 10, 2023 · 22 comments · May be fixed by #10399
Open

Add systemd socket activation support #9891

lalbers opened this issue May 10, 2023 · 22 comments · May be fixed by #10399
Assignees
Labels
area/server kind/enhancement a new or improved feature.

Comments

@lalbers
Copy link

lalbers commented May 10, 2023

Description

The systemd project supports socket-activation, for services allowing systemd to listen on the socket initially. When the first connection comes in, systemd starts the service and passes in any listening sockets as file descriptors.

This technique is also useful for containers, for example when running non-root podman it uses netavark for the network stack. This has the disadvantage that container processes are unable to see the real source ip of incoming connections. One solution can be to use podman socket_activation. Currently socket_activation doesn't seem to be supported for traefik.

What I have tried

systemd-socket-activate -l 80 -l 443 podman run --rm docker.io/library/traefik:latest

Here i would expect traefik to use this sockets when configured to use port 80, 443. Using the same pattern for httpd just works.

@nmengin
Copy link
Contributor

nmengin commented May 11, 2023

Hey @lalbers!

Thanks for your suggestion.
We are interested in this issue but are unsure about the use case and the traction it will receive, so we are going to leave the status as "kind/proposal" to give the community time to let us know that they would like this.
We will reevaluate as people respond.

@nmengin nmengin added kind/proposal a proposal that needs to be discussed. area/server and removed status/0-needs-triage labels May 11, 2023
@thyeestes
Copy link

I love to see this feature to be added to Traefik

@yznerf
Copy link

yznerf commented Oct 5, 2023

I have a use case for this, too. I am trying to run traefik in a rootless podman stack with crowdsec. With the rootlesskit network mode the real IP address of clients is not preserved. With slirp4netns:porthandler=slirp4netns it is, but there is no name resolution, a performance hit and network separation is not possible (no user generated networks with this mode). Socket activation would be a nice feature to work arround the limitation.
That being said, the issue is known and since the new network mode "pasta" is able to preserve the real IP address while using rootlesskit to keep performance up, this might be resolved when pasta becomes the default network mode (I think it's on the roadmap for podman 6).

@Visne
Copy link

Visne commented Nov 1, 2023

Pasta is indeed a lot faster than slirp4netns, but from my experience slightly unstable (but that might improve in the future) and still slower than "native" performance (which socket activation should have). Since Traefik is a reverse proxy and users will most likely be routing almost all of their traffic through it I think it would still be great to have this feature.

@fsdrw08
Copy link

fsdrw08 commented Nov 5, 2023

also, pasta network mode cannot make traefik docker provider discover the container IP

@fsdrw08
Copy link

fsdrw08 commented Jan 26, 2024

any updates? socket activation is really a good need for rootless podman.

@juliens juliens linked a pull request Jan 31, 2024 that will close this issue
2 tasks
@nmengin nmengin added kind/enhancement a new or improved feature. and removed kind/proposal a proposal that needs to be discussed. labels Jan 31, 2024
@cmonty14
Copy link

cmonty14 commented Feb 9, 2024

I have a use case for this, too. I am trying to run traefik in a rootless podman stack with crowdsec. With the rootlesskit network mode the real IP address of clients is not preserved. With slirp4netns:porthandler=slirp4netns it is, but there is no name resolution, a performance hit and network separation is not possible (no user generated networks with this mode). Socket activation would be a nice feature to work arround the limitation. That being said, the issue is known and since the new network mode "pasta" is able to preserve the real IP address while using rootlesskit to keep performance up, this might be resolved when pasta becomes the default network mode (I think it's on the roadmap for podman 6).

Hi Friedemann,
I'm planning the same deployment: traefik + Crowdsec

Could you please share your deployment / configuration / docker-compose-file that you have used?

@yznerf
Copy link

yznerf commented Feb 10, 2024

Hi Friedemann, I'm planning the same deployment: traefik + Crowdsec

Could you please share your deployment / configuration / docker-compose-file that you have used?

Hi Thomas,
atm I am not using crowdsec, because of the aforementioned limitations of traefik in podmans rootless mode. Because I'm runnding rootless podman, I am using tailscale to access (and to give access to familymembers), to keep my inf secure. While I could share my previous docker-compose, there is a lot to configure to use crowdsec. I am not ready to write that all up. Let me point you to a very good ressource:
Smarthome-Beginner: Crowdsec
Anand, the author of this blog does really good work and has multiple writeups on the configuration of a traefik/Crowdsec stack. I highly recommend it. Please also consider supporting him - at least deactivate adblock on the site - if you find it helpful.

@cmonty14
Copy link

atm I am not using crowdsec, because of the aforementioned limitations of traefik in podmans rootless mode

Hi Friedemann, I'm already aware of the referred deployment guides.
And actually I would ask you about the limitations with podman rootless mode that prevent you from going on?
The restriction with priviliged ports could be overcome with kernel parameter net.ipv4.ip_unprivileged_port_start=80.

@yznerf
Copy link

yznerf commented Feb 11, 2024

Thank you, Thomas, but I already knew that. You could also overcome it by port forwarding in the firewall. I do it the same way, as you, but that is not the issue.

When using traefik in rootless mode the real IP of the client is not preserved. Ergo there is no filtering based on the ip lists, provided by crowdsec, possible. Traefik only forwards the ip of the rootless network entrypoint (local ip) and does not see the real ip. Like I wrote: you can overcome that by using pasta or slirp4netns:porthandler=slirp4netns as network backend, but that has disadvantages, too. There is no dns (or you have to configure it), so traefik can't do it's magic, unless you provide ip-adresses/dns entries for the containers. It is far more configuration to write (and for every new container), unlike traefik's automatic service discovery, where you just throw a label on the container and are done with it.
I appreciate your intention to help, but look where we are:
This is a feature request for socket activation. The implementation of this feature would allow to run traefik under rootless podman in a simple user-defined network, with the necessary preserved real client ip. So that traefik can write it into it's logs and crowdsec can read it to make the decision, if the client may have access.

EDIT: And since the pull-request of @juliens from last week seems to work, we might get that pretty soon.

@cmonty14
Copy link

Now I see... it's about socket activation.
I found this blog posting from 11/2023 and it looks like socket activation and traefik is working with rootless container.

@yznerf
Copy link

yznerf commented Feb 11, 2024

Different socket. That's the podman socket. This is about socket activation of containers.

@cmonty14
Copy link

Well, I don't know if this issue is still relevant.
My understanding of the blog is that traefik is working with rootless container.
However there's another issue:
[...] need to restart traefik whener a new container is started up or if an existing converter crashes and automatically restarts. This is because with Podman the Docker events doesn't seem to "refresh" / "reload" traefik like it's supposed to do using Docker (apparently).

A workaround for this issue is here.

@oleksandrborniak
Copy link

pasta network mode cannot make traefik docker provider discover the container IP

@fsdrw08 Can You provide an example of podman-compose file that allows traefik to work with other containers?

If I run the following config traefik docker provider can not discover the container IP:

version: "3.7"

services:
  traefik:
    image: "traefik"
    container_name: traefik
    network_mode: "pasta"
    hostname: traefic
    restart: always
    security_opt:
      - label=type:container_runtime_t

    command:
      - "--api=true"
      - "--api.insecure=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.web.http.redirections.entryPoint.to=websecure"
      - "--entrypoints.web.http.redirections.entrypoint.scheme=https"
      - "--entrypoints.websecure.address=:443"
      - "--certificatesresolvers.mytlschallenge.acme.tlschallenge=true"
      - "--certificatesresolvers.mytlschallenge.acme.email=${SSL_EMAIL}"
      - "--certificatesresolvers.mytlschallenge.acme.storage=/letsencrypt/acme.json"
      - "--log.level=ERROR"
      - "--log.filePath=/logs/traefik.log"
      - "--accesslog=true"
      - "--accesslog.filePath=/logs/access.log"
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - traefik_data:/letsencrypt
      - /run/user/1004/podman/podman.sock:/var/run/docker.sock:z
  n8n:
    image: docker.n8n.io/n8nio/n8n:latest
    container_name: n8n
    network_mode: "pasta"
    hostname: staging-automation.my-clouds.net
    restart: always
    ports:
      - "127.0.0.1:5678:5678"
    labels:
      - traefik.enable=true
      - traefik.http.routers.n8n.rule=Host(`${SUBDOMAIN}.${DOMAIN_NAME}`)
      - traefik.http.routers.n8n.tls=true
      - traefik.http.routers.n8n.entrypoints=web,websecure
      - traefik.http.routers.n8n.tls.certresolver=mytlschallenge
      - traefik.http.middlewares.n8n.headers.SSLRedirect=true
      - traefik.http.middlewares.n8n.headers.STSSeconds=315360000
      - traefik.http.middlewares.n8n.headers.browserXSSFilter=true
      - traefik.http.middlewares.n8n.headers.contentTypeNosniff=true
      - traefik.http.middlewares.n8n.headers.forceSTSHeader=true
      - traefik.http.middlewares.n8n.headers.SSLHost=${DOMAIN_NAME}
      - traefik.http.middlewares.n8n.headers.STSIncludeSubdomains=true
      - traefik.http.middlewares.n8n.headers.STSPreload=true
      - traefik.http.routers.n8n.middlewares=n8n@docker
      - traefik.http.middlewares.removeHostHeader.headers.customrequestheaders.Host=
    environment:
      - N8N_HOST=${SUBDOMAIN}.${DOMAIN_NAME}
      - N8N_PORT=5678
      - N8N_PROTOCOL=https
      - NODE_ENV=production
      - WEBHOOK_URL=https://${SUBDOMAIN}.${DOMAIN_NAME}/
      - GENERIC_TIMEZONE=${GENERIC_TIMEZONE}
    volumes:
      - n8n_data:/home/node/.n8n
      - /home/rootless/podman/data/n8n-local-files:/files

Error:

time="2024-04-25T09:46:15Z" level=error msg="service \"n8n-n8n\" error: unable to find the IP address for the container \"/n8n\": the server is ignored" providerName=docker container=n8n-n8n-35ea005810dbcff3dcc252500240f8781481875346251c316e7f7bf035d8bd63

@yznerf
Copy link

yznerf commented Apr 25, 2024

Thank you for your compose-file, Oleksandr.
With the option:
network_mode: "pasta"

podman does not provide name resolution in the network, so traefik can't find the container. The internal dns of podman does not work. One could provide dns externally or via host-file settings, but that's a hassle.
I tried the new podman version 5, in which pasta is the new default network driver. That works pretty well! But, atm you have to eather build it from source or intstall it via unstable packages. I would not consider it production ready, although Fedora 40 has the new version in it's stable repos.
I still think that socket-activation might be the way to go.

@oleksandrborniak
Copy link

Thank you, Friedemann
Yep, socket-activation is a better solution. I want to try "pasta" until socket-activation for Traefik is not ready.
I just updated yesterday from Fedora 39 to 40.
I wonder if trying Traefik in paste mode is possible without much effort.
Could not find ready-made examples on the internet.

@yznerf
Copy link

yznerf commented Apr 25, 2024

@oleksandrborniak If you are on Fedoara 40 check the Podman Version. If it is 5 and higher, pasta is the default network driver for all networks. That means you don't have to define the pasta network-mode via network_mode: "pasta". Every user-generated network and the default-network run pasta in rootless mode.
I've installed a development-package of podman on a rocky-linux machine
podman version 5.1.0-dev-45b809c06
and i do get pasta as a rootlessnetworkcmd.

$ podman info --format {{.Host.RootlessNetworkCmd}}
pasta

I've tried it out with the default network "podman" and a user-generated network without options. Works fine in testing. Internal DNS works, preservation of the Client-IP and all the shenanigans... traefik did not make any problems in a minimal setup.
I'll test it further when I'l find the time.
Good Luck and have fun!

Meanwhile I am hoping for a merge of the aforementioned pull request, that enables socket-activation for traefik in podman.

@oleksandrborniak
Copy link

It looks like I am doing something wrong.

rootless@aborniakFC:~/podman/n8n$ podman version
Client:       Podman Engine
Version:      5.0.1
API Version:  5.0.1
Go Version:   go1.22.1
Built:        Mon Apr  1 03:00:00 2024
OS/Arch:      linux/amd64
rootless@aborniakFC:~/podman/n8n$

rootless@aborniakFC:~/podman/n8n$ podman info | grep pasta
  pasta:
    executable: /usr/bin/pasta
      pasta 0^20240326.g4988e2b-1.fc40.x86_64-pasta
rootless@aborniakFC:~/podman/n8n$

Config file:

version: "3.7"

services:
  traefik:
    image: "traefik"
    container_name: traefik
    hostname: traefic
    network_mode: pasta
    restart: always
    security_opt:
      - label=type:container_runtime_t

    command:
      - "--api=true"
      - "--api.insecure=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--providers.docker.allowEmptyServices=true" #https://github.com/traefik/traefik/blob/master/docs/content/providers/docker.md #https://community.traefik.io/t/need-to-restart-traefik-whenever-a-converter-is-restarted-even-if-compose-yml-configuration-does-not-change/21166/12
      - "--entrypoints.web.address=:80"
      - "--entrypoints.web.http.redirections.entryPoint.to=websecure"
      - "--entrypoints.web.http.redirections.entrypoint.scheme=https"
      - "--entrypoints.websecure.address=:443"
      - "--certificatesresolvers.mytlschallenge.acme.tlschallenge=true"
      - "--certificatesresolvers.mytlschallenge.acme.email=${SSL_EMAIL}"
      - "--certificatesresolvers.mytlschallenge.acme.storage=/letsencrypt/acme.json"
      - "--log.level=ERROR"
      - "--log.filePath=/logs/traefik.log"
      - "--accesslog=true"
      - "--accesslog.filePath=/logs/access.log"
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - traefik_data:/letsencrypt
      - /run/user/1004/podman/podman.sock:/var/run/docker.sock:z
  n8n:
    image: docker.n8n.io/n8nio/n8n:latest
    container_name: n8n
    hostname: staging-automation.my-clouds.net
    network_mode: pasta
    restart: always
    ports:
      - "127.0.0.1:5678:5678"
    labels:
      - traefik.enable=true
      - traefik.http.routers.n8n.rule=Host(`${SUBDOMAIN}.${DOMAIN_NAME}`)
      - traefik.http.routers.n8n.tls=true
      - traefik.http.routers.n8n.entrypoints=web,websecure
      - traefik.http.routers.n8n.tls.certresolver=mytlschallenge
      - traefik.http.middlewares.n8n.headers.SSLRedirect=true
      - traefik.http.middlewares.n8n.headers.STSSeconds=315360000
      - traefik.http.middlewares.n8n.headers.browserXSSFilter=true
      - traefik.http.middlewares.n8n.headers.contentTypeNosniff=true
      - traefik.http.middlewares.n8n.headers.forceSTSHeader=true
      - traefik.http.middlewares.n8n.headers.SSLHost=${DOMAIN_NAME}
      - traefik.http.middlewares.n8n.headers.STSIncludeSubdomains=true
      - traefik.http.middlewares.n8n.headers.STSPreload=true
      - traefik.http.routers.n8n.middlewares=n8n@docker
      - traefik.http.middlewares.removeHostHeader.headers.customrequestheaders.Host=
    environment:
      - N8N_HOST=${SUBDOMAIN}.${DOMAIN_NAME}
      - N8N_PORT=5678
      - N8N_PROTOCOL=https
      - NODE_ENV=production
      - WEBHOOK_URL=https://${SUBDOMAIN}.${DOMAIN_NAME}/
      - GENERIC_TIMEZONE=${GENERIC_TIMEZONE}
    volumes:
      - n8n_data:/home/node/.n8n
      - /home/rootless/podman/data/n8n-local-files:/files


volumes:
  traefik_data:
    external: true
  n8n_data:
    external: true

Run containers: docker-compose up -d

rootless@aborniakFC:~/podman/n8n$ podman ps
CONTAINER ID  IMAGE                             COMMAND               CREATED         STATUS         PORTS                                     NAMES
f463adf41659  docker.io/library/traefik:latest  --api=true --api....  17 seconds ago  Up 17 seconds  0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp  traefik
1ad1aa539f4f  docker.n8n.io/n8nio/n8n:latest                          17 seconds ago  Up 17 seconds  127.0.0.1:5678->5678/tcp                  n8n
rootless@aborniakFC:~/podman/n8n$

Check networks:

rootless@aborniakFC:~/podman/n8n$ podman inspect traefik |  grep -i networkmode
               "NetworkMode": "pasta",
rootless@aborniakFC:~/podman/n8n$ podman inspect n8n |  grep -i networkmode
               "NetworkMode": "pasta",
rootless@aborniakFC:~/podman/n8n$

Result:
time="2024-04-26T07:18:39Z" level=error msg="service \"n8n-n8n\" error: unable to find the IP address for the container \"/n8n\": the server is ignored" providerName=docker container=n8n-n8n-1ad1aa539f4fee7d30103fdb2a82be38c1bc083894043752c1a18a142950c3c2

Removing "network_mode: pasta" from the config creates a container in a bridge network.

rootless@aborniakFC:~/podman/n8n$ podman inspect n8n |  grep -i networkmode
               "NetworkMode": "bridge",
rootless@aborniakFC:~/podman/n8n$

rootless@aborniakFC:~/podman/n8n$ podman network ls
NETWORK ID    NAME         DRIVER
edec7a84157e  n8n_default  bridge
2f259bab93aa  podman       bridge
rootless@aborniakFC:~/podman/n8n$ podman network inspect n8n_default
[
     {
          "name": "n8n_default",
          "id": "edec7a84157e5b4dfaa7e4087eb7e93bdd8b829c4766f85d83600a3405142a81",
          "driver": "bridge",
          "network_interface": "podman1",
          "created": "2024-04-26T10:25:11.262032044+03:00",
          "subnets": [
               {
                    "subnet": "10.89.0.0/24",
                    "gateway": "10.89.0.1"
               }
          ],
          "ipv6_enabled": false,
          "internal": false,
          "dns_enabled": true,
          "labels": {
               "com.docker.compose.network": "default",
               "com.docker.compose.project": "n8n",
               "com.docker.compose.version": "1.29.2"
          },
          "options": {
               "isolate": "true"
          },
          "ipam_options": {
               "driver": "host-local"
          },
          "containers": {
               "26e9d06bbad5e18cdaa5c80bd28064b4dac5ead3135575b9482a5c3559fe9ec4": {
                    "name": "n8n",
                    "interfaces": {
                         "eth0": {
                              "subnets": [
                                   {
                                        "ipnet": "10.89.0.3/24",
                                        "gateway": "10.89.0.1"
                                   }
                              ],
                              "mac_address": "fe:41:b8:7d:f6:63"
                         }
                    }
               },
               "bc80e0264605ef6b9dea5c3baca988cacfaab8a4f707770931a2710653926ea5": {
                    "name": "traefik",
                    "interfaces": {
                         "eth0": {
                              "subnets": [
                                   {
                                        "ipnet": "10.89.0.2/24",
                                        "gateway": "10.89.0.1"
                                   }
                              ],
                              "mac_address": "c6:9b:6b:69:5f:f0"
                         }
                    }
               }
          }
     }
]
rootless@aborniakFC:~/podman/n8n$

@yznerf
Copy link

yznerf commented Apr 26, 2024

Hm, try removing network-mode:pasta. Until podman 5 this option gave some default-options of slirpnetns to pasta and that includes no (or external) dns. Maybe that's the issue.

@oleksandrborniak
Copy link

I've already tried it and it looks like pasta is not used in this case.

Removing "network_mode: pasta" from the config creates a container in a bridge network.

This is when "network-mode:pasta" is removed from the config:

rootless@aborniakFC:~/podman/n8n$ podman inspect n8n |  grep -i networkmode
               "NetworkMode": "bridge",
rootless@aborniakFC:~/podman/n8n$

rootless@aborniakFC:~/podman/n8n$ podman network ls
NETWORK ID    NAME         DRIVER
edec7a84157e  n8n_default  bridge
2f259bab93aa  podman       bridge
rootless@aborniakFC:~/podman/n8n$

Might be the problem that I updated from Fedora 39 and it is not a new installation.
How did you install the development-package of Podman on a rocky-linux machine? I'll try to replicate your setup.

{{.Host.RootlessNetworkCmd}} is also unavailable in my setup.

root@aborniakFC:~/tmp# podman info --format {{.Host.RootlessNetworkCmd}}
Error: template: info:1:7: executing "info" at <.Host.RootlessNetworkCmd>: can't evaluate field RootlessNetworkCmd in type *define.HostInfo
root@aborniakFC:~/tmp#

@yznerf
Copy link

yznerf commented Apr 26, 2024

@oleksandrborniak
I am sorry, I can't provide my test config, because I ran it with podman run an burned the vm that I've tested it with.
I tried to write down a quick compose, but was not able to reproduce the results.
But I think I've found the problem. podman-compose seems to create the network in a fashion different from podman run.
I've verfied it with a whoami-container.

The command:
podman run -p 80:80 --name whoami traefik/whoami
get's me:

Hostname: 7c2bee0b7fad
IP: 127.0.0.1
IP: ::1
IP: 192.168.1.15
IP: fe80::a4ba:9fff:fe88:700a
RemoteAddr: 192.168.1.3:62301
GET / HTTP/1.1
Host: whoami.podman.local
User-Agent: REDACTED
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
Accept-Encoding: gzip, deflate
Accept-Language: de,en-US;q=0.7,en;q=0.3
Connection: keep-alive
Sec-Gpc: 1
Upgrade-Insecure-Requests: 1

192.168.1.15 is the podman-host
192.168.1.3 is the host from which I am connected
These are the real IPs from the machines. That's only possible, because of pasta.

While a little podman compose with the content:

services:
  whoami:
    image: docker.io/traefik/whoami
    containername: whoami
    ports:
      - 80:80

gets me:

Hostname: e32f4bec5ed4
IP: 127.0.0.1
IP: ::1
IP: 10.89.3.2
IP: fe80::50e3:deff:feae:d924
RemoteAddr: 10.89.3.2:36790
GET / HTTP/1.1
Host: whoami.podman.local
User-Agent: REDACTED
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
Accept-Encoding: gzip, deflate
Accept-Language: de,en-US;q=0.7,en;q=0.3
Connection: keep-alive
Sec-Gpc: 1
Upgrade-Insecure-Requests: 1

which contains only internal (podman-network) IP addresses. That's unfortunate. Apparently there are different options passed by the command podman-compose. I am not proficient enough - and unfortunatly to busy atm for a deep dive - to find the cause for that.

Regarding your other questions. I've installed the podman developer version from the copr:
https://copr.fedorainfracloud.org/coprs/rhcontainerbot/podman-next/

And "bridge" mode is okay. The podman default network is also a bridge network. I did not find another way to confirm that pasta is used as the network driver than podman info. The section of the full response looks like this on my test-machine:

  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20230818.g0af928e-4.el9.x86_64
    version: |
      pasta 0^20230818.g0af928e-4.el9.x86_64
      Copyright Red Hat
      GNU Affero GPL version 3 or later <https://www.gnu.org/licenses/agpl-3.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  rootlessNetworkCmd: pasta

Hope this helps. :)

@oleksandrborniak
Copy link

@yznerf
It is useful info.
Thank you for your assistance and have a nice weekend.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/server kind/enhancement a new or improved feature.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

10 participants