Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

starting container "kube-proxy" fails with : mountpoint for cgroup not found #15406

Closed
asridharan opened this issue Aug 7, 2015 · 8 comments
Closed

starting container "kube-proxy" fails with : mountpoint for cgroup not found #15406

asridharan opened this issue Aug 7, 2015 · 8 comments

Comments

@asridharan
Copy link

Description of problem
I am trying to run kubernetes on a docker-based local setup using the following instructions:
http://kubernetes.io/v1.0/docs/getting-started-guides/docker.html

NOTE: Although the problem is with starting kubernetes, I am filling this issue with docker since looks like the issue is with docker not being able to start a specific container.

All the kuberenetes containers start cleanly, except the "kube-proxy" container.

~ $ docker ps --all
CONTAINER ID        IMAGE                                        COMMAND                CREATED             STATUS                           PORTS               NAMES
34fb39961d7a        gcr.io/google_containers/hyperkube:v0.21.2   "/hyperkube proxy --   54 minutes ago      Exited (255) 54 minutes ago                          dreamy_morse                                                                                             
bd4c5c6e89ab        gcr.io/google_containers/hyperkube:v0.21.2   "/hyperkube proxy --   54 minutes ago                                                           tender_babbage                                                                                           
dd493eaa6d65        gcr.io/google_containers/hyperkube:v0.21.2   "/hyperkube proxy --   About an hour ago   Exited (255) About an hour ago                       berserk_colden                                                                                           
88ca0fb914be        gcr.io/google_containers/hyperkube:v0.21.2   "/hyperkube schedule   About an hour ago   Up About an hour                                     k8s_scheduler.b725e775_k8s-master-127.0.0.1_default_9b44830745c166dfc6d027b0fc2df36d_f864727d            
031b1646492f        gcr.io/google_containers/hyperkube:v0.21.2   "/hyperkube apiserve   About an hour ago   Up About an hour                                     k8s_apiserver.70750283_k8s-master-127.0.0.1_default_9b44830745c166dfc6d027b0fc2df36d_fe01717c            
245658f9a17a        gcr.io/google_containers/hyperkube:v0.21.2   "/hyperkube controll   About an hour ago   Up About an hour                                     k8s_controller-manager.aad1ee8f_k8s-master-127.0.0.1_default_9b44830745c166dfc6d027b0fc2df36d_0fd7c4e8   
6566e67ab67e        gcr.io/google_containers/pause:0.8.0         "/pause"               About an hour ago   Up About an hour                                     k8s_POD.e4cc795_k8s-master-127.0.0.1_default_9b44830745c166dfc6d027b0fc2df36d_21bef762                   
ff98d0a62497        gcr.io/google_containers/hyperkube:v0.21.2   "/hyperkube kubelet    About an hour ago   Up About an hour                                     drunk_franklin                                                                                           
97a00a7de5f4        gcr.io/google_containers/etcd:2.0.9          "/usr/local/bin/etcd   About an hour ago   Up About an hour                                     ecstatic_hodgkin

On looking at the output of the kube-proxy container, looks like the remote API is throwing a error while launching the container:

$ docker logs dd493eaa6d65 
W0807 14:26:19.502305       1 server.go:86] Failed to start in resource-only container "/kube-proxy": mountpoint for cgroup not found
I0807 14:26:19.502667       1 proxier.go:121] Setting proxy IP to 192.168.75.10 and initializing iptables
F0807 14:26:19.509773       1 server.go:101] Unable to create proxer: failed to initialize iptables: error appending rule: exit status 1: iptables: No chain/target/match by that name.

This is a bit odd, cause all the cgroup mounts seem to be in place:

~ $ mount | grep cgroup
cgroup_root on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,relatime,size=10240k,mode=755)
openrc on /sys/fs/cgroup/openrc type cgroup (rw,nosuid,nodev,noexec,relatime,release_agent=/lib64/rc/sh/cgroup-release-agent.sh,name=openrc)
cpuset on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cpu on /sys/fs/cgroup/cpu type cgroup (rw,nosuid,nodev,noexec,relatime,cpu)
cpuacct on /sys/fs/cgroup/cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct)
memory on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
devices on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
freezer on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
blkio on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
perf_event on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
hugetlb on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)

Also, there doesn't seem to be an issue with starting any of the other containers. The only difference, I could see, in starting the kube-proxy container versus the other containers (etcd, api-server, scheduler) is that kube-proxy needs to be run with --priveleged . However, I did try running a ubuntu container with --privileged and it seems to be coming up fine.

docker version:
Client version: 1.7.1
Client API version: 1.19
Go version (client): go1.4
Git commit (client): 786b29d
OS/Arch (client): linux/amd64
Server version: 1.7.1
Server API version: 1.19
Go version (server): go1.4
Git commit (server): 786b29d
OS/Arch (server): linux/amd64

docker info:
Containers: 9
Images: 97
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 115
Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.18.4-aufs
Operating System: Gentoo/Linux
CPUs: 4
Total Memory: 3.862 GiB
Name: stitchserver
ID: NSPJ:6OQ3:FURV:IIES:UIEA:WO65:P3QG:SW2N:NFA6:UTBW:ATHA:4XLF

uname -a:

Linux stitchserver 3.18.4-aufs #4 SMP Thu Aug 6 10:53:19 PDT 2015 x86_64 QEMU Virtual CPU version 2.1.0 GenuineIntel GNU/Linux

Environment details (AWS, VirtualBox, physical, etc.):
Gentoo VM, running a custom kernel. Ran the check-confish.sh script to make sure all kernel configs are turned on:

warning: /proc/config.gz does not exist, searching other paths for kernel config...
info: reading kernel config from /boot/config-3.18.4-aufs ...

Generally Necessary:
- cgroup hierarchy: properly mounted [/sys/fs/cgroup]
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_DEVPTS_MULTIPLE_INSTANCES: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MACVLAN: enabled
- CONFIG_VETH: enabled
- CONFIG_BRIDGE: enabled (as module)
- CONFIG_BRIDGE_NETFILTER: enabled (as module)
- CONFIG_NF_NAT_IPV4: enabled (as module)
- CONFIG_IP_NF_FILTER: enabled
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled
- CONFIG_NF_NAT: enabled (as module)
- CONFIG_NF_NAT_NEEDED: enabled
- CONFIG_POSIX_MQUEUE: enabled

Optional Features:
- CONFIG_MEMCG_SWAP: enabled
- CONFIG_MEMCG_SWAP_ENABLED: enabled
- CONFIG_RESOURCE_COUNTERS: enabled
- CONFIG_BLK_CGROUP: enabled
- CONFIG_IOSCHED_CFQ: enabled
- CONFIG_CGROUP_PERF: enabled
- CONFIG_CFS_BANDWIDTH: missing
- Storage Drivers:
  - "aufs":
    - CONFIG_AUFS_FS: enabled
    - CONFIG_EXT4_FS_POSIX_ACL: enabled
    - CONFIG_EXT4_FS_SECURITY: enabled
  - "btrfs":
    - CONFIG_BTRFS_FS: missing
  - "devicemapper":
    - CONFIG_BLK_DEV_DM: enabled
    - CONFIG_DM_THIN_PROVISIONING: enabled
    - CONFIG_EXT4_FS: enabled
    - CONFIG_EXT4_FS_POSIX_ACL: enabled
    - CONFIG_EXT4_FS_SECURITY: enabled
  - "overlay":
    - CONFIG_OVERLAY_FS: missing
    - CONFIG_EXT4_FS_SECURITY: enabled
    - CONFIG_EXT4_FS_POSIX_ACL: enabled
  - "zfs":
    - /dev/zfs: missing
    - zfs command: missing
    - zpool command: missing

How reproducible:
Every time.

Steps to Reproduce:

  1. Follow the steps listed in http://kubernetes.io/v1.0/docs/getting-started-guides/docker.html
  2. After each step maker sure the docker containers have started correctly.
  3. After step 3 you will the kube-proxy exits with the above issue.

Actual Results:
kube-proxy container fails to start

Expected Results:
kube-proxy should start and configure the iptables with the right entries.

Additional info:

This could be something specific to Gentoo. I am not seeing other users (on Ubuntu for e.g.,) complaining about kubernetes not starting. I am actively discussing this issue with the kubernetes developers as well, but seems like they are also not able to triage the issue cleanly since the problem is with starting the container itself.
@thaJeztah
Copy link
Member

/cc @thockin @abronan :-)

@thaJeztah
Copy link
Member

oh, shucks, wrong ping, meant to include @aboch for libnetwork (sorry @abronan .. blame it on autocomplete :-))

@aboch
Copy link
Contributor

aboch commented Aug 10, 2015

@thaJeztah @asridharan From a rough look at the code, this does not seem to have to do with libnetwork, or docker.

It looks it is internal to kubernetes code: Error is thrown by kubernetes/cmd/kube-proxy/app/server.go code when trying to create the proxier here, tracing it to here and then probably here

@asridharan
Copy link
Author

The error with setting up the iptables does look like an kubernetes issue. But what about the cgroup mount error: "Failed to start in resource-only container "/kube-proxy": mountpoint for cgroup not found"

They are not bailing out of the kube-proxy code when they encounter this error, so maybe its not that critical to the creation of kube-proxy? The relevant code piece is here

@aboch
Copy link
Contributor

aboch commented Aug 10, 2015

@asridharan That one comes from libcontainer, pinging @LK4D4 for libcontainer.

@LK4D4
Copy link
Contributor

LK4D4 commented Aug 10, 2015

@asridharan If you found those errors by docker logs, then both of them from kubernetes. I see that it tries to setup Oom, so it should have some cgroups mounted inside container. Latest docker able to mount cgroups inside container, I'm not sure what cgroups tree expected by kube-proxy, though. Maybe @thockin knows.

@asridharan
Copy link
Author

Looks like all the errors are contained within the kubernetes code-base. I will move this issue to kubernetes in that case.

@asridharan asridharan mentioned this issue Aug 10, 2015
Closed
@thaJeztah
Copy link
Member

thanks all! closing this one, and hope you find a solution :-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@LK4D4 @thaJeztah @asridharan @aboch