nanak8s

Join the Japan7 cluster

Wireguard support is required on the node to join the private cluster innernet.

Steps

1. Setup innernet

Follow the install instructions, ask your invitation.toml and configure your peer with it.

Set the MTU to 1420 on the innernet interface to avoid any issue with QUIC handshakes.

2. Install K3s

Edit and put the following configuration in /etc/rancher/k3s/config.yaml:

token: <shared_secret>
flannel-iface: <innernet_interface>
kubelet-arg:
  - eviction-hard=memory.available<0%
  - eviction-soft=memory.available<100Mi,nodefs.available<5Gi,nodefs.inodesFree<5%,imagefs.available<5Gi
  - eviction-soft-grace-period=memory.available=5m,nodefs.available=5m,nodefs.inodesFree=5m,imagefs.available=5m
  - image-gc-high-threshold=5
  - image-gc-low-threshold=0

If running in server mode, also add the following keys:

secrets-encryption: true
disable:
  - local-storage

Then run the one-liner to install K3s:

curl -sfL https://get.k3s.io | K3S_URL=https://<existing_server_node_innernet_ip>:6443 sh -s - <node_type>

node_type is server or agent.

3. Longhorn requirements

Longhorn (block storage) requires some system packages. Please check their documentation and install them.

4. [Optional] Set up your node as an entrypoint to the cluster

Without more configuration, your node with only answer requests relayed by the other nodes in the cluster. You can set up your node to also answer requests received from the Internet.

Inbound ports

• 80/tcp (HTTP)
• 443/tcp (HTTPS)
• 777/udp (HTTP/3)
• 6443/tcp (K8s API)
• 8222/tcp (SFTPGo)
• 8999/tcp (Syncplay)

Traefik passthrough

K3s internal Traefik serves web apps on port 8443 (websecure). You may setup another Traefik outside the Kubernetes cluster with docker-compose to passthrough matching incoming requests on ports 80 and 443.

• compose.yaml

services:
  traefik:
    image: traefik:v3.0
    command:
      - --providers.docker=true
      - --providers.file.directory=/config
      - --entrypoints.web.address=:80
      - --entrypoints.web.http.redirections.entryPoint.to=websecure
      - --entrypoints.web.http.redirections.entryPoint.scheme=https
      - --entrypoints.websecure.address=:443
    ports:
      - "80:80/tcp"
      - "443:443/tcp"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./traefik:/config
    restart: unless-stopped
    extra_hosts:
      - "host.docker.internal:host-gateway"

• ./traefik/dynconfig.yml

tcp:
  routers:
    nanak8s:
      entryPoints:
        - "websecure"
      rule: "HostSNIRegexp(`^(.+\\.)?japan7\\.bde\\.enseeiht\\.fr$`)"
      service: "nanak8s-file"
      tls:
        passthrough: true
  services:
    nanak8s-file:
      loadBalancer:
        servers:
          - address: "host.docker.internal:8443"

Bootstrapping

Start a new cluster

Save the same configuration file as all above (server mode) and run the following one-liner:

curl -sfL https://get.k3s.io | sh -s - server --cluster-init

Launch Argo CD

export SOPS_AGE_KEY=<private_key>
helmfile apply -f apps/argo-cd/helmfile.yaml -n argocd --set notifications.enabled=false
kubectl apply -f apps/bootstrap.yaml -n argocd

This will start Argo CD in the cluster and configure it so it will automatically add and sync the other apps of this repository.