Network setup on bare-metal nodes with single public interface and private VLAN?

[petertiedemann]

I realize this is a bit of a newbie question, and not even entirely specific to Talos, but I will give it a try anyway :)

We are trialing self-hosted github action runners on bare metal Hetzner servers, and I wanted to try to use Talos to minimize the need for ops.

Each server has a single network interface (say eth0) with a (fixed) public IP4 address. I figured it would be a bad idea for nodes to communicate via nodes over public IP's so I setup a "vSwitch"/VLAN.

Lets say the public IP of each node N is 100.1.1.. and that I give each of them a VLAN IP of 192.168.1..

Now I am wondering:

1. I don't think Hetzner uses DHCP for bare-metal servers, so I assume I need to provide the public IP as a kernel command line? I would also expect that I do not need to specify the VLAN there? (because it can be configured later).
2. I want to be sure I don't accidentally have kubernetes "internal" services bind to the public interface. Does this look about right, or am I missing something? I am a bit confused about the kubelet node IP in particular, because I would not expect to specify a subnet, but rather a specific IP matching the VLAN IP?

 machine:
    network:
        interfaces:
            - interface: eth0 
              addresses:
                - 100.1.1.N
              routes:
              - network: 0.0.0.0/0 # The route's network (destination).
                gateway: <hetzner public IP gateway>
              mtu: 1500
              deviceSelector:
              - hardwareAddr: <server MAC> # Maybe use driver instead as it will only change with different server types
        vlans:
            vlan: eth0.4000 
                vlanId: 4000
                addresses:
                    - 192.168.1.N/16
                dhcp: false
                mtu: 1400 # dictated by Hetzner
   kubelet:
        nodeIP:
            # The `validSubnets` field configures the networks to pick kubelet node IP from.
            validSubnets:
              #  - ? What to put here ?
cluster:
  controlPlane:
     endpoint: https://100.1.1.1:6443 # Assuming first node is control plane
         # Provides cluster specific network configuration options.

  # These options I can leave to default?
  # network:
  #     dnsDomain: cluster.local
  #     # The pod subnet CIDR.
  #     podSubnets:
  #         - 10.244.0.0/16
  #     # The service subnet CIDR.
  #     serviceSubnets:
  #         - 10.96.0.0/12

Answers to the above as well as any other n00b tips would be greatly appreciated :) And before someone comments on needing proper ops for a production setup, I just want to clarify that we are talking test runners only here without access to any sensitive data or the ability to deploy to any customer systems :)

[smira]

I don't think Hetzner uses DHCP for bare-metal servers, so I assume I need to provide the public IP as a kernel command line? I would also expect that I do not need to specify the VLAN there? (because it can be configured later).

I don't know much about Hetzner, probably someone else might chime in. Talos has support for hcloud platform, but I believe this is not bare-metal (???) one.

I want to be sure I don't accidentally have kubernetes "internal" services bind to the public interface. Does this look about right, or am I missing something? I am a bit confused about the kubelet node IP in particular, because I would not expect to specify a subnet, but rather a specific IP matching the VLAN IP?

Talos at the moment doesnt support firewall, but that is being worked on (#4421), so you can't limit where it binds, but you can make sure inter-node communication goes over private VLAN/subnet. See https://www.talos.dev/v1.5/introduction/prodnotes/#multihoming.

Also CNI might need to be adjusted (depending on the CNI), for Flannel you might need to check #7754

[petertiedemann]

I don't know much about Hetzner, probably someone else might chime in. Talos has support for hcloud platform, but I believe this is not bare-metal (???) one.

@smira "hcloud" is indeed Hetzner's VM offering, and its a different (and probably easier :) setup than bare-metal.

Talos at the moment doesnt support firewall, but that is being worked on (#4421), so you can't limit where it binds, but you can make sure inter-node communication goes over private VLAN/subnet. See https://www.talos.dev/v1.5/introduction/prodnotes/#multihoming.

Then I hope the Talos endpoints are intended to be exposed on a public endpoint? :)

Thanks for pointing out that link I did not realize it was called "multihoming". Based on that I should also set

  etcd:
    advertisedSubnets:
      - 192.168.1.0/16

However I am still confused about kubelet.nodeIP. The comment in the generated config says:
# The validSubnets field configures the networks to pick kubelet node IP from.

However, the multihoming doc says:

"Stable IP addressing for kubelets (i.e., nodeIP) is not strictly necessary but highly recommended as it ensures that, e.g., kube-proxy and CNI routing take the desired routes. Analogously to etcd, for kubelets this is controlled via machine.kubelet.nodeIP.validSubnets"

I am having trouble consolidating those two descriptions :)

Also CNI might need to be adjusted (depending on the CNI), for Flannel you might need to check #7754

I played with k3s and microk8s previously and I indeed had a lot of trouble until I found out how to tell flannel which interface to use. I was considering using Cilium if nothing else because someone else has tried it on Hetzner bare-metal before: https://datavirke.dk/posts/bare-metal-kubernetes-part-1-talos-on-hetzner/

[smira]

Please, don't use hardwareAddr network selectors for VLANs, they match recursively VLAN as well.

In 1.7+ use physical: true, for < 1.7 add driver: or busAddr:

[smira]

I mean combine hardwareAddr with something else, so that it matches physical link only, but not the VLAN created

[lucawen]

This physical: true already exists? I didn't find it in the documentation.
Edit:
I find it at https://www.talos.dev/v1.7/talos-guides/network/device-selector/#configuring-network-device-using-device-selector

[lucawen]

@smira Worked! But im still receiving this error:

user: warning: [2024-04-26T15:05:54.430673057Z]: [talos] controller failed {"component": "controller-runtime", "controller": "network.RouteSpecController", "error": "1 error occurred:\n\t* error adding route:    
 netlink receive: invalid argument, message {Family:2 DstLength:0 SrcLength:0 Tos:0 Table:0 Protocol:4 Scope:0 Type:1 Flags:0 Attributes:{Dst:10.0.0.0 Src:<nil> Gateway:10.0.1.1 OutIface:9 Priority:1024 Table:    
 254 Mark:0 Pref:<nil> Expires:<nil> Metrics:<nil> Multipath:[]}}\n\n"}         

idk exactly what is wrong with the routes in the vlan configuration

[smira]

I can't guess exactly from an error message, but looks like invalid route to me. DstLength is 0, as if it's /0 route which makes zero sense.