Usernetes: Kubernetes without the root privileges

Usernetes aims to provide a reference distribution of Kubernetes that can be installed under a user's $HOME and does not require the root privileges.

• Status
• Adoption
• How it works
• Requirements
  • Distribution-specific hint
    • Ubuntu
    • Debian GNU/Linux
    • Arch Linux
    • openSUSE
    • Fedora 31 and later
    • Fedora 30
    • RHEL/CentOS 8
    • RHEL/CentOS 7
• Restrictions
• Install from binary
• Install from source
• Quick start
  • Install
  • Use kubectl
  • Reset to factory defaults
  • Uninstall
• Run Usernetes in Docker
  • Single node
  • Multi node (Docker Compose)
• Advanced guide
  • Expose netns ports to the host
  • Routing ping packets
• License

Status

• master components (etcd, kube-apiserver, ...)
• kubelet
• CRI runtimes
  • CRI-O
  • containerd
• Cgroup
• Multi-node CNI
  • Flannel (VXLAN)

Currently, Usernetes uses our patched version of Kubernetes. See ./src/patches. We are proposing our patches to the Kubernetes upstream. See #42 for the current status.

Deployment shell scripts are in POC status. (It even lacks TLS setup - #76)

See Adoption for Usernetes-based Kubernetes distributions with TLS support.

Note

Usernetes no longer includes Docker (Moby) binaries since February 2020.

To install Rootless Docker, see https://get.docker.com/rootless .

See also https://docs.docker.com/engine/security/rootless/ for the further information.

Adoption

We encourage other Kubernetes distributions to adopt Usernetes patches and tools.

Currently, the following distributions adopt Usernetes:

• k3s
• Silverkube

How it works

Usernetes executes Kubernetes and CRI runtimes without the root privileges by using unprivileged user_namespaces(7), mount_namespaces(7), and network_namespaces(7).

To set up NAT across the host and the network namespace without the root privilege, Usernetes uses a usermode network stack (slirp4netns).

No SETUID/SETCAP binary is needed, except newuidmap(1) and newgidmap(1), which are used for setting up user_namespaces(7) with multiple sub-UIDs and sub-GIDs.

Requirements

• Kernel >= 4.18.

• Recent version of systemd. Known to work with systemd >= 242.

• mount.fuse3 binary. Provided by fuse3 package on most distros.

• iptables binary. Provided by iptables package on most distros.

• newuidmap and newgidmap binaries. Provided by uidmap package on most distros.

• /etc/subuid and /etc/subgid should contain more than 65536 sub-IDs. e.g. exampleuser:231072:65536. These files are automatically configured on most distros.

$ id -u
1001
$ whoami
exampleuser
$ grep "^$(whoami):" /etc/subuid
exampleuser:231072:65536
$ grep "^$(whoami):" /etc/subgid
exampleuser:231072:65536

Distribution-specific hint

Ubuntu

• No preparation is needed.

Debian GNU/Linux

• Add kernel.unprivileged_userns_clone=1 to /etc/sysctl.conf (or /etc/sysctl.d) and run sudo sysctl -p

Arch Linux

• Add kernel.unprivileged_userns_clone=1 to /etc/sysctl.conf (or /etc/sysctl.d) and run sudo sysctl -p

openSUSE

• sudo modprobe ip_tables iptable_mangle iptable_nat iptable_filter is required. (This is likely to be required on other distros as well)

Fedora

• Run sudo dnf install -y iptables.
• If doesn't work on Fedora >= 31, try sudo grubby --update-kernel=ALL --args="systemd.unified_cgroup_hierarchy=0" and reboot.

RHEL/CentOS 8

• Run sudo dnf install -y iptables.

RHEL/CentOS 7

• Unsupported since February 2020. Usernetes v20200126.0 (January 26, 2020) should work.

Restrictions

• slirp4netns is used instead of vEth pairs.
• fuse-overlayfs is used instead of overlayfs.
• Following features are not supported:
  • Cgroups
  • Apparmor

Install from binary

Download the latest usernetes-x86_64.tbz from Releases.

$ tar xjvf usernetes-x86_64.tbz
$ cd usernetes

Install from source

Docker 17.05+ is required for building Usernetes from the source. Docker 18.09+ with DOCKER_BUILDKIT=1 is recommended.

$ make

Binaries are generated under ./bin directory.

Quick start

Install

install.sh installs Usernetes systemd units to $HOME/.config/systemd/unit.

To use containerd as the CRI runtime (default):

$ ./install.sh --cri=containerd
[INFO] Base dir: /home/exampleuser/gopath/src/github.com/rootless-containers/usernetes
[INFO] Installing /home/exampleuser/.config/systemd/user/u7s.target
[INFO] Installing /home/exampleuser/.config/systemd/user/u7s-rootlesskit.service
[INFO] Installing /home/exampleuser/.config/systemd/user/u7s-etcd.target
[INFO] Installing /home/exampleuser/.config/systemd/user/u7s-etcd.service
[INFO] Installing /home/exampleuser/.config/systemd/user/u7s-master.target
[INFO] Installing /home/exampleuser/.config/systemd/user/u7s-kube-apiserver.service
[INFO] Installing /home/exampleuser/.config/systemd/user/u7s-kube-controller-manager.service
[INFO] Installing /home/exampleuser/.config/systemd/user/u7s-kube-scheduler.service
[INFO] Installing /home/exampleuser/.config/systemd/user/u7s-node.target
[INFO] Installing /home/exampleuser/.config/systemd/user/u7s-containerd.service
[INFO] Installing /home/exampleuser/.config/systemd/user/u7s-kubelet-containerd.service
[INFO] Installing /home/exampleuser/.config/systemd/user/u7s-kube-proxy.service
[INFO] Starting u7s.target
+ systemctl --user -T enable u7s.target
Created symlink /home/exampleuser/.config/systemd/user/multi-user.target.wants/u7s.target → /home/exampleuser/.config/systemd/user/u7s.target.
+ systemctl --user -T start u7s.target
Enqueued anchor job 522 u7s.target/start.
Enqueued auxiliary job 538 u7s-rootlesskit.service/start.
Enqueued auxiliary job 542 u7s-kubelet-containerd.service/start.
Enqueued auxiliary job 541 u7s-containerd.service/start.
Enqueued auxiliary job 524 u7s-etcd.service/start.
Enqueued auxiliary job 546 u7s-kube-controller-manager.service/start.
Enqueued auxiliary job 523 u7s-etcd.target/start.
Enqueued auxiliary job 543 u7s-master.target/start.
Enqueued auxiliary job 544 u7s-kube-scheduler.service/start.
Enqueued auxiliary job 545 u7s-kube-apiserver.service/start.
Enqueued auxiliary job 539 u7s-node.target/start.
Enqueued auxiliary job 540 u7s-kube-proxy.service/start.
+ systemctl --user --no-pager status
● localhost
    State: running
...
[INFO] Hint: `sudo loginctl enable-linger` to start user services automatically on the system start up.

To use CRI-O:

$ ./install.sh --cri=crio

Use kubectl

$ ./kubectl.sh get nodes

Or

$ ./rootlessctl.sh add-ports 127.0.0.1:8080:8080/tcp
$ export KUBECONFIG=$(pwd)/config/localhost.kubeconfig
$ kubectl get nodes

Or

$ nsenter -U -n -t $(cat $XDG_RUNTIME_DIR/usernetes/rootlesskit/child_pid) \
  kubectl --kubeconfig=./config/localhost.kubeconfig get nodes

Reset to factory defaults

$ ./cleanup.sh

Uninstall

$ ./uninstall.sh

Run Usernetes in Docker

All-in-one Docker image is available as rootlesscontainers/usernetes on Docker Hub.

To build the image manually:

$ docker build -t rootlesscontainers/usernetes .

The image is based on Fedora.

Single node

$ docker run -td --name usernetes-node -p 127.0.0.1:8080:8080 --privileged rootlesscontainers/usernetes -p 0.0.0.0:8080:8080/tcp --cri=containerd
$ export KUBECONFIG=./config/localhost.kubeconfig
$ kubectl run -it --rm --image busybox foo
/ #

Multi node (Docker Compose)

$ docker-compose up -d
$ export KUBECONFIG=./config/localhost.kubeconfig

Flannel VXLAN 10.5.0.0/16 is configured by default.

$ kubectl get nodes -o wide
NAME           STATUS   ROLES    AGE     VERSION           INTERNAL-IP    EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
967e81e90e1f   Ready    <none>   3m42s   v1.14-usernetes   10.0.101.100   <none>        Ubuntu 18.04.1 LTS   4.15.0-43-generic   docker://Unknown
b2204f192e5c   Ready    <none>   3m42s   v1.14-usernetes   10.0.102.100   <none>        Ubuntu 18.04.1 LTS   4.15.0-43-generic   cri-o://1.14.0-dev
ba0133c68378   Ready    <none>   3m42s   v1.14-usernetes   10.0.103.100   <none>        Ubuntu 18.04.1 LTS   4.15.0-43-generic   containerd://1.2.0-168-gb3807c5d
$ kubectl run --replicas=3 --image=nginx:alpine nginx
$ kubectl get pods -o wide
NAME                     READY   STATUS    RESTARTS   AGE   IP          NODE           NOMINATED NODE   READINESS GATES
nginx-6b4b85b77b-7hqrk   1/1     Running   0          3s    10.5.13.3   b2204f192e5c   <none>           <none>
nginx-6b4b85b77b-8rknj   1/1     Running   0          3s    10.5.79.3   967e81e90e1f   <none>           <none>
nginx-6b4b85b77b-r466s   1/1     Running   0          3s    10.5.7.3    ba0133c68378   <none>           <none>
$ kubectl exec -it nginx-6b4b85b77b-7hqrk -- wget -O - http://10.5.79.3
Connecting to 10.5.79.3 (10.5.79.3:80)
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...
$ kubectl exec -it nginx-6b4b85b77b-7hqrk -- wget -O - http://10.5.7.3
Connecting to 10.5.7.3 (10.5.7.3:80)
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...

Advanced guide

Expose netns ports to the host

As Usernetes runs in a network namespace (with slirp4netns), you can't expose container ports to the host by just running kubectl expose --type=NodePort.

In addition, you need to expose Usernetes netns ports to the host:

$ ./rootlessctl.sh add-ports 0.0.0.0:8080:80/tcp

You can also manually expose Usernetes netns ports manually with socat:

$ pid=$(cat $XDG_RUNTIME_DIR/usernetes/rootlesskit/child_pid)
$ socat -t -- TCP-LISTEN:8080,reuseaddr,fork EXEC:"nsenter -U -n -t $pid socat -t -- STDIN TCP4\:127.0.0.1\:80"

Routing ping packets

To route ping packets, you need to set up net.ipv4.ping_group_range properly as the root.

$ sudo sh -c "echo 0   2147483647  > /proc/sys/net/ipv4/ping_group_range"

License

Usernetes is licensed under the terms of Apache License Version 2.0.

The binary releases of Usernetes contain files that are licensed under the terms of different licenses:

• bin/crun: GNU GENERAL PUBLIC LICENSE Version 2, see https://github.com/containers/crun
• bin/fuse-overlayfs: GNU GENERAL PUBLIC LICENSE Version 3, see https://github.com/containers/fuse-overlayfs
• bin/slirp4netns: GNU GENERAL PUBLIC LICENSE Version 2, see https://github.com/rootless-containers/slirp4netns
• bin/socat: GNU GENERAL PUBLIC LICENSE Version 2, see http://www.dest-unreach.org/socat/