nixos/k3s: exec: "k3s-server": cannot run executable found relative to current directory with k3s_1_26

[bbigras]

Describe the bug

{
    package = pkgs.k3s_1_26; # err
    package = pkgs.k3s_1_25; # ok
}

Steps To Reproduce

Expected behavior

Screenshots

Additional context

mar 29 11:41:13 vps systemd[1]: Starting k3s service...
mar 29 11:41:13 vps k3s[76752]: time="2023-03-29T11:41:13-04:00" level=fatal msg="exec: \"k3s-server\": cannot run executable found relative to current directory"

Maybe related to the k3s changes referred in c7132173806

Notify maintainers

@asonix maybe

Metadata

• system: "x86_64-linux"
• host os: Linux 6.1.21, NixOS, 23.05 (Stoat), 23.05pre468444.4bb072f0a8b
• multi-user?: yes
• sandbox: yes
• version: nix-env (Nix) 2.13.3
• channels(root): "nixos, nixos-unstable"
• nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos

[asonix]

On first look - this doesn't seem related to the environmentFile change, but can you post more about your k3s configuration?

[asonix]

also pinging @superherointj

[bbigras]

my config:

{
  services.k3s = {
    enable = true;
    package = pkgs.k3s_1_25;
    role = "server";
    extraFlags = toString [
      "--kubelet-arg=eviction-hard=imagefs.available<1%,nodefs.available<1%"
      "--kubelet-arg=eviction-minimum-reclaim=imagefs.available=1%,nodefs.available=1%"
      "-disable=traefik"
    ];
  };

  # https://github.com/NixOS/nixpkgs/issues/103158
  systemd.services.k3s.after = [ "network-online.service" "firewall.service" ];
  systemd.services.k3s.serviceConfig.KillMode = lib.mkForce "control-group";
}

[superherointj]

I could not reproduce this issue.

• Needs more information.
• Simplify your configuration.
• Bisect.
• Run same configuration in a new host.
• Clean-up past state: (nevermind this)
  • backup first (will lose data).
  • As root:
    • systemctl stop k3s
    • umount kubelet
      • KUBELET_PATH=$(mount | grep kubelet | cut -d' ' -f3);
        ${KUBELET_PATH:+umount $KUBELET_PATH}; sleep 1
    • delete k3s data
      • rm -rf /etc/rancher/{k3s,node};
        rm -rf /var/lib/rancher/k3s/;
        rm -rf /var/lib/kubelet

[superherointj]

Btw, your patch is wrong. network-online.service => network-online.target
And that won't be necessary anymore: #223823

[bbigras]

I was able to reproduce using nixos/tests/k3s/single-node.nix.

"-disable=traefik" is wrong.

"--disable=traefik" or "--disable" "traefik" are OK.

so, sorry for the noise. and thank you very much for the quick replies.

and if you are on a bug killing spree, I also have this in my config:

{
  # https://github.com/NixOS/nixpkgs/issues/98766
  boot.kernelModules = [ "br_netfilter" "ip_conntrack" "ip_vs" "ip_vs_rr" "ip_vs_wrr" "ip_vs_sh" "overlay" ];
  networking.firewall.extraCommands = ''
    iptables -A INPUT -i cni+ -j ACCEPT
  '';
}