Kernel Panic when mounting volumes on RHEL 6.5 using Docker 1.4.0, Docker 1.4.1, with latest device mapper #9856

marcellodesales · 2014-12-30T22:08:19Z

The problem

After mounting a host volume with Docker 1.4.0, Docker 1.4.1-dev, the server hangs for a few seconds and the ssh terminal session is lost with the host. Here's a Fig up session:

[root@pppdc9prd6dq newww]# fig up
Creating newww_npmRegistryElasticsearchData_1...
Creating newww_npmRegistryElasticsearchServer_1...
Creating newww_npmRegistryCouchdbData_1...
Creating newww_npmRegistryCouchdbServer_1...
Creating newww_npmRegistryRedisData_1...
Creating newww_npmRegistryRedisServer_1...
Write failed: Broken pipe

The same occurs when running the docker command manually...

[root@pppdc9prd6dq newww]# docker run -ti -v $(pwd)/dev/couch/data:/usr/local/var/lib/couchdb -v $(pwd)/dev/couch/couch-docker.ini:/usr/local/etc/couchdb/local.ini busybox bash
exec: "bash": executable file not found in $PATHFATA[0000] Error response from daemon: Cannot start container e8e53a826d1602e1ca5363109db401856963f47313f2d25a8e6a2d9b3e718c3a: exec: "bash": executable file not found in $PATH 
[root@pppdc9prd6dq newww]# docker run -ti -v $(pwd)/dev/couch/data:/usr/local/var/lib/couchdb -v $(pwd)/dev/couch/couch-docker.ini:/usr/local/etc/couchdb/local.ini busybox sh
/ # ls -la
total 56
drwxr-xr-x   17 root     root          4096 Dec 30 18:51 .
drwxr-xr-x   17 root     root          4096 Dec 30 18:51 ..
-rwxr-x---    1 root     root             0 Dec 30 18:51 .dockerenv
-rwxr-x---    1 root     root             0 Dec 30 18:51 .dockerinit
drwxrwxr-x    2 root     root          4096 May 22  2014 bin
drwxr-xr-x    5 root     root           380 Dec 30 18:51 dev
drwxr-xr-x    6 root     root          4096 Dec 30 18:51 etc
drwxrwxr-x    4 root     root          4096 May 22  2014 home
drwxrwxr-x    2 root     root          4096 May 22  2014 lib
lrwxrwxrwx    1 root     root             3 May 22  2014 lib64 -> lib
lrwxrwxrwx    1 root     root            11 May 22  2014 linuxrc -> bin/busybox
drwxrwxr-x    2 root     root          4096 Feb 27  2014 media
drwxrwxr-x    2 root     root          4096 Feb 27  2014 mnt
drwxrwxr-x    2 root     root          4096 Feb 27  2014 opt
dr-xr-xr-x  158 root     root             0 Dec 30 18:51 proc
drwx------    2 root     root          4096 Dec 30 18:51 root
lrwxrwxrwx    1 root     root             3 Feb 27  2014 run -> tmp
drwxr-xr-x    2 root     root          4096 May 22  2014 sbin
drwxr-xr-x   13 root     root             0 Dec 30 18:28 sys
drwxrwxrwt    3 root     root          4096 May 22  2014 tmp
drwxrwxr-x    7 root     root          4096 Dec 30 18:51 usr
drwxrwxr-x    4 root     root          4096 May 22  2014 var
/ # ls -la /usr/local/var/lib/couchdb/
total 8
drwxr-x---    2 root     root          4096 Dec 30 09:50 .
drwxr-x---    3 root     root          4096 Dec 30 18:51 ..
/ # ls -la /usr/local/etc/couchdb/local.ini 
-rw-------    1 root     root          1359 Dec 30 07:14 /usr/local/etc/couchdb/local.ini
/ # exit
Write failed: Broken pipe

I cannot ssh to the host until I restart the server... ssh to the host becomes unavailable...

mdesales@SDGl121456EB ~ (mac) $ ssh npm.corp.intuit.net 
ssh: connect to host npm.corp.intuit.net port 22: Operation timed out

Reproducing

First, I installed the latest Docker 1.4.0 from yum EPEL, and later built Docker 1.4.1 on a Ubuntu machine and scp the latest version to the host. Still the same problem occurs. Everything started with the following error:

http://stackoverflow.com/questions/27216473/docker-1-3-fails-to-start-on-rhel6-5

So I upgraded the device mapper library to the latest version:

$ cd /etc/yum.repos.d 
$ wget public-yum.oracle.com/public-yum-ol6.repo 
$ wget public-yum.oracle.com/RPM-GPG-KEY-oracle-ol6 -O /etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
$ yum update device-mapper

After that, I was able to restart Docker 1.4.0 or Docker 1.4.1-dev without a problem...

[root@pppdc9prd6dq mdesales]# docker --version
Docker version 1.4.1-dev, build a444643

I can list images without a problem and manipulate them using Fig:

[root@pppdc9prd6dq newww]# fig up
Creating newww_npmRegistryElasticsearchData_1...
Cannot start container 62c57ef8830000496a8c6dfeba637dc8976de47d82431f3df701c6bdccfe6d5d: Cannot find child for /newww_npmRegistryElasticsearchData_1
[root@pppdc9prd6dq newww]# fig rm
Going to remove newww_npmRegistryRedisServer_1, newww_npmRegistryRedisData_1, newww_npmRegistryCouchdbServer_1, newww_npmRegistryCouchdbData_1, newww_npmRegistryElasticsearchServer_1
Are you sure? [yN] y
Removing newww_npmRegistryElasticsearchServer_1...
Removing newww_npmRegistryCouchdbData_1...
Removing newww_npmRegistryCouchdbServer_1...
Removing newww_npmRegistryRedisData_1...
Removing newww_npmRegistryRedisServer_1...

After that, I tried to "fig up" again and the same problem occurs...

[root@pppdc9prd6dq newww]# fig up
Creating newww_npmRegistryElasticsearchData_1...
Creating newww_npmRegistryElasticsearchServer_1...
Creating newww_npmRegistryCouchdbData_1...
Creating newww_npmRegistryCouchdbServer_1...
Creating newww_npmRegistryRedisData_1...
Creating newww_npmRegistryRedisServer_1...
Write failed: Broken pipe

The text was updated successfully, but these errors were encountered:

marcellodesales · 2014-12-30T22:28:56Z

Upgrade from 1.2.0 to 1.4.0 reuses the same /var/lib/docker

It is worth to mention that I also upgrade the host from Docker 1.2.0 to Docker 1.4.0 from the testing EPEL repo initially. That's when I had to upgrade... Also, I've reused the same "/var/lib/docker" directory from the upgrade as my Docker directory resides in a separate partition/disk.

$ /var/lib/docker -> /app/var/lib/docker/
[root@pppdc9prd6dq newww]# ls -la /app/var/lib/docker/
total 76
drwx------  11 root root  4096 Dec 30 13:55 .
drwx------   3 root root  4096 Aug  8 16:00 ..
drwx------  21 root root  4096 Dec 30 13:34 containers
drwx------   5 root root  4096 Dec 30 01:34 devicemapper
drwx------   3 root root  4096 Dec 30 01:34 execdriver
drwx------ 192 root root 20480 Dec 30 13:00 graph
drwx------   2 root root  4096 Dec 30 13:55 init
-rw-r-----   1 root root 12288 Dec 30 13:55 linkgraph.db
-rw-------   1 root root  1501 Dec 30 13:00 repositories-devicemapper
drwx------   2 root root  4096 Dec 30 13:07 tmp
drwx------   2 root root  4096 Dec 30 01:34 trust
drwx------   3 root root  4096 Dec 30 01:44 vfs
drwx------  47 root root  4096 Dec 30 13:34 volumes
[root@pppdc9prd6dq newww]#

Questions

Would the reuse of the /var/docker/lib directory pose any problem when upgrading between versions?
Is there any alternative to verify what's really going on? I don't see any entry under /var/log/messages nor under /var/log/docker.

Workaround: Downgraded to Docker 1.3.3

Downgrading to Docker 1.3.3 works as expected:

[root@pppdc9prd6dq newww]# docker --version
Docker version 1.3.3, build d344625/1.3.3
[root@pppdc9prd6dq newww]# fig --version
fig 1.0.1

The device mapper used is still the newest available:

[root@pppdc9prd6dq newww]# rpm -q device-mapper
device-mapper-1.02.90-2.el6_6.1.x86_64

With the downgrade, everything is "working"...

Fresh /var/lib/docker directory was created
Multiple volumes for Redis, Couch and ElasticSearch were created
Fig up created all the containers correctly (with few runtime errors regarding the databases/volumes, this might be other problems)

[root@pppdc9prd6dq newww]# fig up
Recreating newww_npmRegistryElasticsearchData_1...
Recreating newww_npmRegistryElasticsearchServer_1...
Recreating newww_npmRegistryCouchdbData_1...
Recreating newww_npmRegistryCouchdbServer_1...
  Recreating newww_npmRegistryRedisData_1...
Recreating newww_npmRegistryRedisServer_1...
Creating newww_npmregistryserver_1...
Attaching to newww_npmRegistryElasticsearchServer_1, newww_npmRegistryCouchdbServer_1, newww_npmRegistryRedisServer_1, newww_npmregistryserver_1
npmRegistryElasticsearchServer_1 | [2014-12-30 22:19:04,075][INFO ][node                     ] [Captain Universe] version[1.4.1], pid[1], build[89d3241/2014-11-26T15:49:29Z]
npmRegistryElasticsearchServer_1 | [2014-12-30 22:19:04,075][INFO ][node                     ] [Captain Universe] initializing ...
npmRegistryElasticsearchServer_1 | [2014-12-30 22:19:04,079][INFO ][plugins                  ] [Captain Universe] loaded [], sites []
npmRegistryElasticsearchServer_1 | [2014-12-30 22:19:07,533][INFO ][node                     ] [Captain Universe] initialized
npmRegistryElasticsearchServer_1 | [2014-12-30 22:19:07,533][INFO ][node                     ] [Captain Universe] starting ...
npmRegistryElasticsearchServer_1 | [2014-12-30 22:19:07,595][INFO ][transport                ] [Captain Universe] bound_address {inet[/0.0.0.0:9300]}, publish_address {inet[/192.168.4.23:9300]}
npmRegistryElasticsearchServer_1 | [2014-12-30 22:19:07,609][INFO ][discovery                ] [Captain Universe] elasticsearch_npm_www_dev/lTwVoL-YR8OfQE7d2BguCA
npmRegistryElasticsearchServer_1 | [2014-12-30 22:19:11,403][INFO ][cluster.service          ] [Captain Universe] new_master [Captain Universe][lTwVoL-YR8OfQE7d2BguCA][bc4e33e964b6][inet[/192.168.4.23:9300]], reason: zen-disco-join (elected_as_master)
npmRegistryElasticsearchServer_1 | [2014-12-30 22:19:11,420][INFO ][http                     ] [Captain Universe] bound_address {inet[/0.0.0.0:9200]}, publish_address {inet[/192.168.4.23:9200]}
npmRegistryElasticsearchServer_1 | [2014-12-30 22:19:11,420][INFO ][node                     ] [Captain Universe] started
npmRegistryElasticsearchServer_1 | [2014-12-30 22:19:11,540][INFO ][gateway                  ] [Captain Universe] recovered [0] indices into cluster_state
npmRegistryRedisServer_1         | [1] 30 Dec 22:19:22.720 # You requested maxclients of 10000 requiring at least 10032 max file descriptors.
npmRegistryRedisServer_1         | [1] 30 Dec 22:19:22.720 # Redis

marcellodesales · 2014-12-31T18:48:32Z

Here's a Kernel Panic message captured in the host's console... I'm aware that might be the result of using RHEL 6.5, which uses an older kernel 2.6...

chenchun · 2015-01-14T12:45:06Z

Also encountered the same problem with docker 1.4.1, kernel version 2.6.32-431.el6.x86_64
Run docker run --rm docker.oa.com:8080/library/centos7 env several times.
And then ssh terminal session is lost with that host.

[root@c182 /data/home/tdwadmin]# docker run --rm docker.oa.com:8080/library/centos7 env                                                                                                      
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=08b277ab6868
GAIA_PORT_MAPPING={}
HOME=/root
[root@c182 /data/home/tdwadmin]# docker run --rm docker.oa.com:8080/library/centos7 env
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=facaa35e1d93
GAIA_PORT_MAPPING={}
HOME=/root
Write failed: Broken pipe

[tdwadmin@c162 ~]$ ping x.x.x.182
PING x.x.x.182 (10.149.27.182) 56(84) bytes of data.
From x.x.x.182 icmp_seq=2 Destination Host Unreachable
From x.x.x.182 icmp_seq=3 Destination Host Unreachable
From x.x.x.182 icmp_seq=4 Destination Host Unreachable

unclejack · 2015-01-21T13:47:49Z

@marcellodesales @chenchun Please install system updates using yum update and reboot. I believe these problems may have already been fixed in the latest kernels provided by CentOS.

/cc @vbatts

vbatts · 2015-01-21T16:51:32Z

@marcellodesales wait, so you are reporting a couple of issues.
You initially had a failure in devmapper on a link time reference to a function? Funny i've never seen that before. And not sure that I would fix it by pulling packages from various repos just yet.
Can you provide your docker info, docker version? Also, where are you installing docker from?

marcellodesales · 2015-01-22T09:09:44Z

Hi @vbatts

As I described above, I pulled the 1.3.x version from the EPEL repos from RedHat, while I built Docker 1.4.x from the current dev branches (see details above). Here's what version I have with Docker 1.4.x and 1.3.x.

Docker 1.4.x

This is the version that breaks the server, and similar to what @chenchun described, I can't login to the host after that happens.

mdesales@pppdc9prd6dq ~]$ sudo docker info
Containers: 36
Images: 156
Storage Driver: devicemapper
 Pool Name: docker-253:0-271429-pool
 Pool Blocksize: 65.54 kB
 Data file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata file: /var/lib/docker/devicemapper/devicemapper/metadata
 Data Space Used: 5.667 GB
 Data Space Total: 107.4 GB
 Metadata Space Used: 10.1 MB
 Metadata Space Total: 2.147 GB
 Library Version: 1.02.89-RHEL6 (2014-09-01)
Execution Driver: native-0.2
Kernel Version: 2.6.32-431.1.2.el6.x86_64
Operating System: <unknown>
CPUs: 4
Total Memory: 15.58 GiB
Name: pppdc9prd6dq
ID: Y2SD:MBOV:KMHT:CXED:7YBH:27UF:IHFK:TTE4:4OWL:4SH4:GGWJ:6SVQ

Docker 1.3.x

This version works with Fig.

[mdesales@pppdc9prd6dq ~]$ sudo docker info
Containers: 36
Images: 156
Storage Driver: devicemapper
 Pool Name: docker-253:0-271429-pool
 Pool Blocksize: 65.54 kB
 Data file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata file: /var/lib/docker/devicemapper/devicemapper/metadata
 Data Space Used: 5.667 GB
 Data Space Total: 107.4 GB
 Metadata Space Used: 10.1 MB
 Metadata Space Total: 2.147 GB
 Library Version: 1.02.89-RHEL6 (2014-09-01)
Execution Driver: native-0.2
Kernel Version: 2.6.32-431.1.2.el6.x86_64
Operating System: <unknown>

Updating the versions

I used the versions by downloading the binary and swapping the versions... I once had to remove the lib directory, but upgrading from 1.3.3 to 1.4.0 just was a matter of updating the symlinks and restarting the service...

[mdesales@pppdc9prd6dq ~]$ ls -la /usr/bin/docker
lrwxrwxrwx 1 root root 21 2014-12-31 11:57 /usr/bin/docker -> /usr/bin/docker_1.3.3
[mdesales@pppdc9prd6dq ~]$ sudo rm -f /usr/bin/docker
[mdesales@pppdc9prd6dq ~]$ sudo ln -s /usr/bin/docker_1.4.0 /usr/bin/docker
[mdesales@pppdc9prd6dq ~]$ sudo service docker stop
Stopping docker:                                           [  OK  ]
[mdesales@pppdc9prd6dq ~]$ sudo service docker start
Starting docker:                                       [  OK  ]

What's next?

Let me know if you need any more information about this environment...

@chenchun, could you also describe what is your environment?

thanks guys! We still have till mid March to get RHEL 7.0 and I just can't wait. Until then, I'm still suffering with the lack of Docker 1.4.x ...

chenchun · 2015-01-24T03:11:10Z

I didn't use original docker-1.4.1 package from community, because we did some customer development and mountains our internal version of docker. But I'm sure the bug is not due to the added feathers since we also use our internal version of docker-1.3.1 pretty well.

[root@c114 /tmp]# docker info
Containers: 38
Images: 179
Storage Driver: devicemapper
 Pool Name: docker-8:4-20201477-pool
 Pool Blocksize: 65.54 kB
 Data file: /data/docker/var/lib/docker/devicemapper/devicemapper/data
 Metadata file: /data/docker/var/lib/docker/devicemapper/devicemapper/metadata
 Data Space Used: 12.65 GB
 Data Space Total: 107.4 GB
 Metadata Space Used: 15.75 MB
 Metadata Space Total: 2.147 GB
 Library Version: 1.02.82-git (2013-10-04)
Execution Driver: native-0.2
Kernel Version: 2.6.32-431.el6.x86_64
Operating System: <unknown>
CPUs: 4
Total Memory: 7.685 GiB
Name: c114
ID: JVYV:GAON:QTKM:SQKQ:KGIX:75EQ:VXSH:2GK7:UOEP:3CN4:GXAD:5G7Y

[root@c114 /tmp]# docker -v
Docker version 1.4.1, build bbc84c8

And it is not that easy for us to upgrade kernels because we have to use our internal kernel from the kernel team in our company.

rhvgoyal · 2015-01-29T20:24:45Z

@chenchun

What should I run to reproduce this issue locally?

Tried " docker run --rm docker.oa.com:8080/library/centos7 env" and it fails. docker.oa.com is not even reachable.

chenchun · 2015-01-30T02:08:59Z

@rhvgoyal Thanks for looking at this, you can pull docker.cn/docker/centos.

rhvgoyal · 2015-01-30T14:36:24Z

@chenchun

I tried following on Fedora 21 with 1.4.1 docker and it works just fine.

[root@tiger ~]# docker run --rm centos env
Unable to find image 'centos:latest' locally
Pulling repository centos
8efe422e6104: Download complete
511136ea3c5a: Download complete
5b12ef8fd570: Download complete
Status: Downloaded newer image for centos:latest
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=70e82f87629f
HOME=/root

marcellodesales · 2015-02-07T07:58:01Z

This is RedHat Enterprise Linux 6.5 host... Not centos...

unclejack · 2015-02-07T09:31:35Z

@marcellodesales CentOS 6.5 and RHEL 6.5 have the exact same kernel and they're extremely close to being identical when it comes to devicemapper. That's why CentOS 6.x, RHEL 6.x and other distributions based on RHEL 6.x are frequently discussed together.

marcellodesales · 2015-02-07T16:48:23Z

@unclejack Thanks for the clarification... I just want to make sure they don't have any deltas that could impact with the verification of this...

Today I experienced the same behavior, but this time the --net host switch and I completely lost control of the host.

Using the --net host also breaks

Everything was fine...

# root at pppdc9prd505 in /export/home/mdesales [8:29:45]
$ docker run -ti --net host -e http_proxy=http://qypprdproxy02.ie.intuit.net:80 platform.registry.docker.corp.intuit.net/cicd/jenkins bash

After inspecting the /var/hosts... The server crashed...

root@pppdc9prd505:/var/lib/jenkins/plugins# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
#0.0.0.0  ntp-server1
#0.0.0.0  ntp-server2
10.132.68.233    pppdc9prd505.corp.intuit.net   pppdc9prd505    # NIC <eth5>
root@pppdc9prd505:/var/lib/jenkins/plugins# ping github.intuit.com
PING github-prd.ilb.intuit.com (10.168.0.56) 56(84) bytes of data.
64 bytes from github-prd-lvdc.intuit.com (10.168.0.56): icmp_seq=1 ttl=246 time=45.1 ms
64 bytes from github-prd-lvdc.intuit.com (10.168.0.56): icmp_seq=2 ttl=246 time=45.1 ms
^C
--- github-prd.ilb.intuit.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 45.109/45.143/45.178/0.215 ms
root@pppdc9prd505:/var/lib/jenkins/plugins# exitWrite failed: Broken pipe
mdesales@SDGl121456EB ~/dev/github-intuit/ApplicationServices (master) $ ssh ci.dockerhub.corp.intuit.net

Weird behavior... I will downgrade again...

thanks!

kashyap-parikh · 2015-02-17T06:32:32Z

Any update on this issue? I am able to reproduce this issue on CentOS 6.5 (2.6.32-431.el6.x86_64) + Docker 1.4.1 + Fig 1.0.1.

larrymyers · 2015-02-17T18:33:29Z

I am also seeing this issue with the exact same setup:

CentOS 6.5 (2.6.32-431.el6.x86_64) + Docker 1.4.1 + Fig 1.0.1

The only immediate workaround I've found is to not use Fig and to just run the docker commands manually. In this case Fig seems to be the problem.

kashyap-parikh-ck · 2015-02-18T01:24:29Z

Downgrading to Docker 1.3.2 resolves the issue but would love to move to 1.5. It's in fedora testing repo, hopefully it will make it to stable soon.

Johnsmith1111 · 2015-03-02T09:24:54Z

it seems that I am a little later to arrive here,centos 6.5 + epel source,and the same kernel panic as pircture above.And ... ,Can you Downgrading to Docker 1.3.2 or quickly resolve it?

seanclerkin · 2015-03-02T17:04:43Z

I'm also seeing this regularly with the same setup
CentOS 6.5 (2.6.32-431.el6.x86_64)
Docker version 1.4.1, build 5bc2ff8/1.4.1

Johnsmith1111 · 2015-03-03T02:11:48Z

http://koji.fedoraproject.org/koji/buildinfo?buildID=595471

Here we can get the history 1.3.2 rpm source package,and you can install by the following:
(the is some dependence there,just install it)

rpm -i docker-io-1.3.2-2.el6.src.rpm
cd rpmbuild/SPECS
rpmbuild -ba docker-io.spec
cd rpmbuild/RPMS

and the rpm we want is in rpmbuild/RPMS

Johnsmith1111 · 2015-03-05T05:04:57Z

@jfrazelle,I'm sorry,And Could you please cancel the two label on my response above? It's my mistake,the CentOS's version I use in that comment is 6.5,and the 6.6 is OK for this issue.

hxfeng · 2015-06-03T07:05:33Z

+1

strootman · 2015-06-15T16:00:05Z

This seems to be re-occuring in kernel 2.6.32-504.23.4.el6.x86_64. Had to downgrade to 2.6.32-504.16.2.el6.x86_64.

jamesdobson · 2015-06-15T21:34:52Z

I am also noticing the panic using kernel 2.6.32-504.23.4.el6.x86_64. Downgrading to 2.6.32-504.16.2.el6.x86_64 works for me.

Khazrak · 2015-06-16T14:40:00Z

I am also noticing the panic using kernel 2.6.32-504.23.4.el6.x86_64 (RHEL 6.6, Centos 6.6). Older version kernel 2.6.32-504.8.1.el6.x86_64 works fine.

rhvgoyal · 2015-06-16T15:06:20Z

Can somebody paste the full panic() with any backtrace here. IOW, please paste the output of console.

Khazrak · 2015-06-17T11:38:43Z

This is the only log I've been able to get so far:

SELinux: initialized (dev proc, type proc), uses genfs_contexts
device vethba38ad2 entered promiscuous mode
ADDRCONF(NETDEV_UP): vethba38ad2: link is not ready
ADDRCONF(NETDEV_CHANGE): vethba38ad2: link becomes ready
docker0: port 1(vethba38ad2) entering forwarding state
SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
SELinux: initialized (dev devpts, type devpts), uses transition SIDs
docker0: port 1(vethba38ad2) entering disabled state
device vethba38ad2 left promiscuous mode
docker0: port 1(vethba38ad2) entering disabled state
EXT4-fs (dm-3): mounted filesystem with ordered data mode. Opts:
SELinux: initialized (dev dm-3, type ext4), uses xattr
EXT4-fs (dm-3): mounted filesystem with ordered data mode. Opts:
SELinux: initialized (dev dm-3, type ext4), uses xattr
EXT4-fs (dm-3): mounted filesystem with ordered data mode. Opts:
SELinux: initialized (dev dm-3, type ext4), uses xattr
SELinux: initialized (dev mqueue, type mqueue), uses transition SIDs
lo: Disabled Privacy Extensions
SELinux: initialized (dev proc, type proc), uses genfs_contexts
device vethabe8f70 entered promiscuous mode
ADDRCONF(NETDEV_UP): vethabe8f70: link is not ready
ADDRCONF(NETDEV_CHANGE): vethabe8f70: link becomes ready
docker0: port 1(vethabe8f70) entering forwarding state
SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
SELinux: initialized (dev devpts, type devpts), uses transition SIDs
docker0: port 1(vethabe8f70) entering disabled state
general protection fault: 0000 [#1] SMP
last sysfs file: /sys/devices/virtual/net/vethabe8f70/address
CPU 0
Modules linked in: veth ipt_addrtype xt_conntrack iptable_filter ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables bridge stp llc dm_thin_pool dm_bio_prison dm_persistent_data dm_bufio libcrc32c vboxsf(U) ipv6 ppdev parport_pc parport i2c_piix4 vboxguest(U) e1000 vboxvideo(U) drm i2c_core sg ext4 jbd2 mbcache video output sr_mod cdrom sd_mod crc_t10dif pata_acpi ata_generic ata_piix ahci dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]

Pid: 5166, comm: docker Not tainted 2.6.32-504.23.4.el6.x86_64 #1 innotek GmbH VirtualBox/VirtualBox
RIP: 0010:[] [] list_del+0x10/0xa0
RSP: 0018:ffff880119c4ddc8 EFLAGS: 00010092
RAX: dead000000200200 RBX: ffff88010ad3da58 RCX: 0000000000000010
RDX: 0000000000000002 RSI: 0000000000000003 RDI: ffff88010ad3da58
RBP: ffff880119c4ddd8 R08: 0000000000000010 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: ffff88010ad3da40
R13: ffff88011b4d7f18 R14: 0000000000000010 R15: 0000000000000000
FS: 00007f42d17fb700(0000) GS:ffff880028200000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c20874bf20 CR3: 000000011a091000 CR4: 00000000000006f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process docker (pid: 5166, threadinfo ffff880119c4c000, task ffff88010a146040)
Stack:
0000000000000001 ffff88010ad3da80 ffff880119c4de08 ffffffff810cdc32
0000000100000004 ffff880119881d78 0000000000000000 ffff880119881d90
ffff880119c4de58 ffffffff81057839 0000000000000000 0000000300000001
Call Trace:
[] cgroup_event_wake+0x42/0x70
[] __wake_up_common+0x59/0x90
[] __wake_up+0x48/0x70
[] eventfd_release+0x2d/0x40
[] __fput+0xf5/0x210
[] fput+0x25/0x30
[] filp_close+0x5d/0x90
[] sys_close+0xa5/0x100
[] system_call_fastpath+0x16/0x1b
Code: 01 01 01 01 01 48 0f af c2 48 c1 e8 38 c3 90 90 90 90 90 90 90 90 90 90 90 90 90 55 48 89 e5 53 48 89 fb 48 83 ec 08 48 8b 47 08 <4c> 8b 00 4c 39 c7 75 39 48 8b 03 4c 8b 40 08 4c 39 c3 75 4c 48
RIP [] list_del+0x10/0xa0
RSP
Initializing cgroup subsys cpuset
Initializing cgroup subsys cpu
Linux version 2.6.32-504.23.4.el6.x86_64 (mockbuild@c6b9.bsys.dev.centos.org) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-11) (GCC) ) #1 SMP Tue Jun 9 20:57:37 UTC 2015
Command line: ro root=/dev/mapper/VolGroup-lv_root rd_NO_LUKS rd_NO_MD rd_LVM_LV=VolGroup/lv_swap SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=de rd_LVM_LV=VolGroup/lv_root LANG=de_DE.UTF-8 rd_NO_DM rhgb quiet console=ttyS0,57600 console=tty0 ignore_loglevel irqpoll nr_cpus=1 reset_devices cgroup_disable=memory mce=off disable_cpu_apicid=0 memmap=exactmap memmap=635K@4K memmap=131441K@49787K elfcorehdr=181228K memmap=4K$0K memmap=1K$639K memmap=64K$960K memmap=64K#3669952K memmap=256K$4194048K
KERNEL supported cpus:
Intel GenuineIntel
AMD AuthenticAMD
Centaur CentaurHauls
BIOS-provided physical RAM map:
BIOS-e820: 0000000000000100 - 000000000009fc00 (usable)
BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
BIOS-e820: 0000000000100000 - 00000000dfff0000 (usable)
BIOS-e820: 00000000dfff0000 - 00000000e0000000 (ACPI data)
BIOS-e820: 00000000fffc0000 - 0000000100000000 (reserved)
BIOS-e820: 0000000100000000 - 0000000120000000 (usable)
debug: ignoring loglevel setting.
last_pfn = 0x120000 max_arch_pfn = 0x400000000
user-defined physical RAM map:
user: 0000000000000000 - 0000000000001000 (reserved)
user: 0000000000001000 - 000000000009fc00 (usable)
user: 000000000009fc00 - 00000000000a0000 (reserved)
user: 00000000000f0000 - 0000000000100000 (reserved)
user: 000000000309ec00 - 000000000b0fb000 (usable)
user: 00000000dfff0000 - 00000000e0000000 (ACPI data)
user: 00000000fffc0000 - 0000000100000000 (reserved)
DMI 2.5 present.
SMBIOS version 2.5 @ 0xFFF60
DMI: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
e820 update range: 0000000000000000 - 0000000000001000 (usable) ==> (reserved)
e820 remove range: 00000000000a0000 - 0000000000100000 (usable)
last_pfn = 0xb0fb max_arch_pfn = 0x400000000
MTRR default type: uncachable
MTRR variable ranges disabled:
x86 PAT enabled: cpu 0, old 0x7010600070106, new 0x7010600070106
CPU MTRRs all blank - virtualized system.
initial memory mapped : 0 - 20000000
init_memory_mapping: 0000000000000000-000000000b0fb000
0000000000 - 000b000000 page 2M
000b000000 - 000b0fb000 page 4k
kernel direct mapping tables up to b0fb000 @ 8000-b000
RAMDISK: 0abc5000 - 0b0ee724
ACPI: RSDP 00000000000e0000 00024 (v02 VBOX )
ACPI: XSDT 00000000dfff0030 0003C (v01 VBOX VBOXXSDT 00000001 ASL 00000061)
ACPI: FACP 00000000dfff00f0 000F4 (v04 VBOX VBOXFACP 00000001 ASL 00000061)
ACPI: DSDT 00000000dfff0470 01BF1 (v01 VBOX VBOXBIOS 00000002 INTL 20100528)
ACPI: FACS 00000000dfff0200 00040
ACPI: APIC 00000000dfff0240 00054 (v02 VBOX VBOXAPIC 00000001 ASL 00000061)
ACPI: SSDT 00000000dfff02a0 001CC (v01 VBOX VBOXCPUT 00000002 INTL 20100528)
ACPI: Local APIC address 0xfee00000
Setting APIC routing to flat.
No NUMA configuration found
Faking a node at 0000000000000000-000000000b0fb000
Bootmem setup node 0 0000000000000000-000000000b0fb000
NODE_DATA [0000000000009000 - 000000000003cfff]
bootmap [000000000003d000 - 000000000003e61f] pages 2
(7 early reservations) ==> bootmem [0000000000 - 000b0fb000]
#0 [0000000000 - 0000001000] BIOS data page ==> [0000000000 - 0000001000]
#1 [0000006000 - 0000008000] TRAMPOLINE ==> [0000006000 - 0000008000]
#2 [0004000000 - 0005029be4] TEXT DATA BSS ==> [0004000000 - 0005029be4]
#3 [000abc5000 - 000b0ee724] RAMDISK ==> [000abc5000 - 000b0ee724]
#4 [000009f800 - 0000100000] BIOS reserved ==> [000009f800 - 0000100000]
#5 [000502a000 - 000502a13c] BRK ==> [000502a000 - 000502a13c]
#6 [0000008000 - 0000009000] PGTABLE ==> [0000008000 - 0000009000]
found SMP MP-table at [ffff88000009fff0] 9fff0
[ffffea0000000000-ffffea00003fffff] PMD -> [ffff880003600000-ffff8800039fffff] on node 0
Zone PFN ranges:
DMA 0x00000001 -> 0x00001000
DMA32 0x00001000 -> 0x00100000
Normal 0x00100000 -> 0x00100000
Movable zone start PFN for each node
early_node_map[2] active PFN ranges
0: 0x00000001 -> 0x0000009f
0: 0x0000309f -> 0x0000b0fb
On node 0 totalpages: 33018
DMA zone: 56 pages used for memmap
DMA zone: 100 pages reserved
DMA zone: 2 pages, LIFO batch:0
DMA32 zone: 564 pages used for memmap
DMA32 zone: 32296 pages, LIFO batch:7
ACPI: PM-Timer IO Port: 0x4008
ACPI: Local APIC address 0xfee00000
Setting APIC routing to flat.
ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
ACPI: IOAPIC (id[0x01] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 1, version 17, address 0xfec00000, GSI 0-23
ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
ACPI: IRQ0 used by override.
ACPI: IRQ2 used by override.
ACPI: IRQ9 used by override.
Using ACPI (MADT) for SMP configuration information
SMP: Allowing 1 CPUs, 0 hotplug CPUs
nr_irqs_gsi: 24
PM: Registered nosave memory: 000000000009f000 - 00000000000a0000
PM: Registered nosave memory: 00000000000a0000 - 00000000000f0000
PM: Registered nosave memory: 00000000000f0000 - 0000000000100000
PM: Registered nosave memory: 0000000000100000 - 000000000309f000
Allocating PCI resources starting at b0fb000 (gap: b0fb000:d4ef5000)
Booting paravirtualized kernel on bare hardware
NR_CPUS:4096 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1
PERCPU: Embedded 30 pages/cpu @ffff880003200000 s90968 r8192 d23720 u2097152
pcpu-alloc: s90968 r8192 d23720 u2097152 alloc=1*2097152
pcpu-alloc: [0] 0
Built 1 zonelists in Node order, mobility grouping on. Total pages: 32298
Policy zone: DMA32
Kernel command line: ro root=/dev/mapper/VolGroup-lv_root rd_NO_LUKS rd_NO_MD rd_LVM_LV=VolGroup/lv_swap SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=de rd_LVM_LV=VolGroup/lv_root LANG=de_DE.UTF-8 rd_NO_DM rhgb quiet console=ttyS0,57600 console=tty0 ignore_loglevel irqpoll nr_cpus=1 reset_devices cgroup_disable=memory mce=off disable_cpu_apicid=0 memmap=exactmap memmap=635K@4K memmap=131441K@49787K elfcorehdr=181228K memmap=4K$0K memmap=1K$639K memmap=64K$960K memmap=64K#3669952K memmap=256K$4194048K
Misrouted IRQ fixup and polling support enabled
This may significantly impact system performance
Disabling memory control group subsystem
PID hash table entries: 512 (order: 0, 4096 bytes)
Checking aperture...
No AGP bridge found
Memory: 105540k/181228k available (5336k kernel code, 49156k absent, 26532k reserved, 7018k data, 1288k init)
Hierarchical RCU implementation.
NR_IRQS:33024 nr_irqs:256
Spurious LAPIC timer interrupt on cpu 0
Console: colour VGA+ 80x25
console [tty0] enabled
console [ttyS0] enabled
Fast TSC calibration using PIT
Detected 3494.790 MHz processor.
Calibrating delay loop (skipped), value calculated using timer frequency.. 6989.58 BogoMIPS (lpj=3494790)
pid_max: default: 32768 minimum: 301
Security Framework initialized
SELinux: Initializing.
SELinux: Starting in permissive mode
Dentry cache hash table entries: 16384 (order: 5, 131072 bytes)
Inode-cache hash table entries: 8192 (order: 4, 65536 bytes)
Mount-cache hash table entries: 256
Initializing cgroup subsys ns
Initializing cgroup subsys cpuacct
Initializing cgroup subsys memory
Initializing cgroup subsys devices
Initializing cgroup subsys freezer
Initializing cgroup subsys net_cls
Initializing cgroup subsys blkio
Initializing cgroup subsys perf_event
Initializing cgroup subsys net_prio
using mwait in idle threads.
SMP alternatives: switching to UP code
Freeing SMP alternatives: 36k freed
ACPI: Core revision 20090903
ftrace: converting mcount calls to 0f 1f 44 00 00
ftrace: allocating 21926 entries in 86 pages
APIC routing finalized to flat.
..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
CPU0: Intel(R) Xeon(R) CPU E3-1246 v3 @ 3.50GHz stepping 03
Performance Events: unsupported p6 CPU model 60 no PMU driver, software events only.
NMI watchdog disabled (cpu0): hardware events not enabled
Brought up 1 CPUs
Total of 1 processors activated (6989.58 BogoMIPS).
sizeof(vma)=200 bytes
sizeof(page)=56 bytes
sizeof(inode)=592 bytes
sizeof(dentry)=192 bytes
sizeof(ext3inode)=800 bytes
sizeof(buffer_head)=104 bytes
sizeof(skbuff)=232 bytes
sizeof(task_struct)=2672 bytes
devtmpfs: initialized
regulator: core version 0.5
NET: Registered protocol family 16
ACPI: bus type pci registered
PCI: Using configuration type 1 for base access
bio: create slab at 0
ACPI: EC: Look up EC in DSDT
ACPI: Executed 1 blocks of module-level executable AML code
ACPI: Interpreter enabled
ACPI: (supports S0 S5)
ACPI: Using IOAPIC for interrupt routing
ACPI: No dock devices found.
PCI: Ignoring host bridge windows from ACPI; if necessary, use "pci=use_crs" and report a bug
ACPI: PCI Root Bridge [PCI0](domain 0000 [bus 00-ff])
pci_root PNP0A03:00: host bridge window io 0x0000-0x0cf7
pci_root PNP0A03:00: host bridge window io 0x0d00-0xffff
pci_root PNP0A03:00: host bridge window mem 0x000a0000-0x000bffff
pci_root PNP0A03:00: host bridge window mem 0xe0000000-0xffdfffff
PCI: root bus 00: using default resources
PCI host bridge to bus 0000:00
pci_bus 0000:00: root bus resource [io 0x0000-0xffff]
pci_bus 0000:00: root bus resource [mem 0x00000000-0x7fffffffff]
pci 0000:00:01.1: reg 20: [io 0xd000-0xd00f]
pci 0000:00:02.0: reg 10: [mem 0xe0000000-0xe0ffffff pref]
pci 0000:00:03.0: reg 10: [mem 0xf0000000-0xf001ffff]
pci 0000:00:03.0: reg 18: [io 0xd010-0xd017]
pci 0000:00:04.0: reg 10: [io 0xd020-0xd03f]
pci 0000:00:04.0: reg 14: [mem 0xf0400000-0xf07fffff]
pci 0000:00:04.0: reg 18: [mem 0xf0800000-0xf0803fff pref]
pci 0000:00:08.0: reg 10: [mem 0xf0820000-0xf083ffff]
pci 0000:00:08.0: reg 18: [io 0xd040-0xd047]
pci 0000:00:0d.0: reg 10: [io 0xd048-0xd04f]
pci 0000:00:0d.0: reg 18: [io 0xd058-0xd05f]
pci 0000:00:0d.0: reg 20: [io 0xd070-0xd07f]
pci 0000:00:0d.0: reg 24: [mem 0xf0840000-0xf0841fff]
ACPI: PCI Interrupt Routing Table [SB.PCI0._PRT]
ACPI: PCI Interrupt Link [LNKA](IRQs 5 9 10 *11), disabled.
ACPI: PCI Interrupt Link [LNKB](IRQs 5 9 10 *11), disabled.
ACPI: PCI Interrupt Link [LNKC](IRQs 5 9 *10 11), disabled.
ACPI: PCI Interrupt Link [LNKD](IRQs 5 9 10 *11), disabled.
vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none
vgaarb: loaded
vgaarb: bridge control possible 0000:00:02.0
SCSI subsystem initialized
libata version 3.00 loaded.
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
PCI: Using ACPI for IRQ routing
PCI: old code would have set cacheline size to 32 bytes, but clflush_size = 64
PCI: pci_cache_line_size set to 64 bytes
NetLabel: Initializing
NetLabel: domain hash size = 128
NetLabel: protocols = UNLABELED CIPSOv4
NetLabel: unlabeled traffic allowed by default
Switching to clocksource jiffies
pnp: PnP ACPI init
ACPI: bus type pnp registered
pnp 00:00: [io 0x0cf8-0x0cff]
pnp 00:00: Plug and Play ACPI device, IDs PNP0a03 (active)
pnp 00:01: [io 0x0060]
pnp 00:01: [io 0x0064]
pnp 00:01: [irq 1]
pnp 00:01: Plug and Play ACPI device, IDs PNP0303 (active)
pnp 00:02: [io 0x0000-0x000f]
pnp 00:02: [io 0x0080-0x008f]
pnp 00:02: [io 0x00c0-0x00df]
pnp 00:02: [dma 4]
pnp 00:02: Plug and Play ACPI device, IDs PNP0200 (active)
pnp 00:03: [irq 12]
pnp 00:03: Plug and Play ACPI device, IDs PNP0f03 (active)
pnp 00:04: [io 0x0378-0x037f]
pnp 00:04: [io 0x0778-0x077f]
pnp 00:04: [irq 7]
pnp 00:04: Plug and Play ACPI device, IDs PNP0400 (active)
pnp 00:05: [io 0x03f8-0x03ff]
pnp 00:05: [irq 4]
pnp 00:05: Plug and Play ACPI device, IDs PNP0501 (active)
pnp: PnP ACPI: found 6 devices
ACPI: ACPI bus type pnp unregistered
Switching to clocksource acpi_pm
pci_bus 0000:00: resource 4 [io 0x0000-0xffff]
pci_bus 0000:00: resource 5 [mem 0x00000000-0x7fffffffff]
NET: Registered protocol family 2
IP route cache hash table entries: 1024 (order: 1, 8192 bytes)
TCP established hash table entries: 4096 (order: 4, 65536 bytes)
TCP bind hash table entries: 4096 (order: 4, 65536 bytes)
TCP: Hash tables configured (established 4096 bind 4096)
TCP reno registered
NET: Registered protocol family 1
pci 0000:00:00.0: Limiting direct PCI/PCI transfers
pci 0000:00:01.0: Activating ISA DMA hang workarounds
pci 0000:00:02.0: Boot video device
Trying to unpack rootfs image as initramfs...
hrtimer: interrupt took 4903747 ns
Freeing initrd memory: 5285k freed
platform rtc_cmos: registered platform RTC device (no PNP device found)
futex hash table entries: 256 (order: 2, 16384 bytes)
audit: initializing netlink socket (disabled)
type=2000 audit(1434540923.267:1): initialized
HugeTLB registered 2 MB page size, pre-allocated 0 pages
VFS: Disk quotas dquot_6.5.2
Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
Refined TSC clocksource calibration: 3493.035 MHz.
msgmni has been set to 216
SELinux: Registering netfilter hooks
Switching to clocksource tsc
alg: No test for stdrng (krng)
ksign: Installing public key data
Loading keyring

Added public key A23E2B04ECA5D043
User ID: CentOS (Kernel Module GPG key)
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251)
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
pci_hotplug: PCI Hot Plug PCI Core version: 0.5
pciehp: PCI Express Hot Plug Controller Driver version: 0.4
acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
ACPI: AC Adapter AC
input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
ACPI: Power Button [PWRF]
input: Sleep Button as /devices/LNXSYSTM:00/LNXSLPBN:00/input/input1
ACPI: Sleep Button [SLPF]
ACPI: acpi_idle registered with cpuidle
[Firmware Bug]: No valid trip found
GHES: HEST is not enabled!
Non-volatile memory driver v1.3
Linux agpgart interface v0.103
crash memory driver: version 1.1
Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
\FFserial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
00:05: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
brd: module loaded
loop: module loaded
input: Macintosh mouse button emulation as /devices/virtual/input/input2
Fixed MDIO Bus: probed
ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
uhci_hcd: USB Universal Host Controller Interface driver
PNP: PS/2 Controller [PNP0303:PS2K,PNP0f03:PS2M] at 0x60,0x64 irq 1,12
serio: i8042 KBD port at 0x60,0x64 irq 1
serio: i8042 AUX port at 0x60,0x64 irq 12
mice: PS/2 mouse device common for all mice
input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input3
rtc_cmos rtc_cmos: rtc core: registered rtc_cmos as rtc0
rtc0: alarms up to one day, 114 bytes nvram
cpuidle: using governor ladder
cpuidle: using governor menu
EFI Variables Facility v0.08 2004-May-17
usbcore: registered new interface driver hiddev
input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input4
usbcore: registered new interface driver usbhid
usbhid: v2.6:USB HID core driver
GRE over IPv4 demultiplexor driver
TCP cubic registered
Initializing XFRM netlink socket
NET: Registered protocol family 17
registered taskstats version 1
rtc_cmos rtc_cmos: setting system clock to 2015-06-17 11:35:42 UTC (1434540942)
Initalizing network drop monitor service
Freeing unused kernel memory: 1288k freed
Write protecting the kernel read-only data: 10240k
Freeing unused kernel memory: 788k freed
Freeing unused kernel memory: 1564k freed
device-mapper: uevent: version 1.0.3
device-mapper: ioctl: 4.27.0-ioctl (2013-10-30) initialised: dm-devel@redhat.com
NET: Registered protocol family 10
lo: Disabled Privacy Extensions
alloc irq_desc for 20 on node -1
alloc kstat_irqs on node -1
vboxguest 0000:00:04.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
input: Unspecified device as /devices/pci0000:00/0000:00:04.0/input/input5
vboxguest: misc device minor 57, IRQ 20, I/O port d020, MMIO at 00000000f0400000 (size 0x400000)
vboxguest: Successfully loaded version 4.3.28 (interface 0x00010004)
e1000: Intel(R) PRO/1000 Network Driver - version 7.3.21-k8-NAPI
e1000: Copyright (c) 1999-2006 Intel Corporation.
alloc irq_desc for 19 on node -1
alloc kstat_irqs on node -1
e1000 0000:00:03.0: PCI INT A -> GSI 19 (level, low) -> IRQ 19
e1000 0000:00:03.0: eth0: (PCI:33MHz:32-bit) 08:00:27:2d:82:3a
e1000 0000:00:03.0: eth0: Intel(R) PRO/1000 Network Connection
alloc irq_desc for 16 on node -1
alloc kstat_irqs on node -1
e1000 0000:00:08.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
e1000 0000:00:08.0: eth1: (PCI:33MHz:32-bit) 08:00:27:3e:a5:13
e1000 0000:00:08.0: eth1: Intel(R) PRO/1000 Network Connection
STARTING CRC_T10DIF
ahci 0000:00:0d.0: version 3.0
alloc irq_desc for 21 on node -1
alloc kstat_irqs on node -1
ahci 0000:00:0d.0: PCI INT A -> GSI 21 (level, low) -> IRQ 21
ahci: SSS flag set, parallel bus scan disabled
ahci 0000:00:0d.0: AHCI 0001.0100 32 slots 1 ports 3 Gbps 0x1 impl SATA mode
ahci 0000:00:0d.0: flags: 64bit ncq stag only ccc
scsi0 : ahci
ata1: SATA max UDMA/133 abar m8192@0xf0840000 port 0xf0840100 irq 21
ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
ata1.00: ATA-6: VBOX HARDDISK, 1.0, max UDMA/133
ata1.00: 20766720 sectors, multi 128: LBA48 NCQ (depth 31/32)
ata1.00: configured for UDMA/133
scsi 0:0:0:0: Direct-Access ATA VBOX HARDDISK 1.0 PQ: 0 ANSI: 5
scsi 0:0:0:0: Attached scsi generic sg0 type 0
ata_piix 0000:00:01.1: version 2.13
scsi1 : ata_piix
scsi2 : ata_piix
ata2: PATA max UDMA/33 cmd 0x1f0 ctl 0x3f6 bmdma 0xd000 irq 14
ata3: PATA max UDMA/33 cmd 0x170 ctl 0x376 bmdma 0xd008 irq 15
ata3.00: ATAPI: VBOX CD-ROM, 1.0, max UDMA/133
ata2.00: ATAPI: VBOX CD-ROM, 1.0, max UDMA/133
ata2.00: configured for UDMA/33
ata3.00: configured for UDMA/33
scsi 1:0:0:0: CD-ROM VBOX CD-ROM 1.0 PQ: 0 ANSI: 5
scsi 1:0:0:0: Attached scsi generic sg1 type 5
scsi 2:0:0:0: CD-ROM VBOX CD-ROM 1.0 PQ: 0 ANSI: 5
scsi 2:0:0:0: Attached scsi generic sg2 type 5
ppdev: user-space parallel port driver
parport_pc 00:04: reported by Plug and Play ACPI
piix4_smbus 0000:00:07.0: SMBus base address uninitialized - upgrade BIOS or use force_addr=0xaddr
[drm] Initialized drm 1.1.0 20060810
sd 0:0:0:0: [sda] 20766720 512-byte logical blocks: (10.6 GB/9.90 GiB)
sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
sda: sda1 sda2
sd 0:0:0:0: [sda] Attached SCSI disk
sr0: scsi3-mmc drive: 32x/32x xa/form2 tray
Uniform CD-ROM driver Revision: 3.20
sr 1:0:0:0: Attached scsi CD-ROM sr0
sr1: scsi3-mmc drive: 32x/32x xa/form2 tray
sr 2:0:0:0: Attached scsi CD-ROM sr1
input: Video Bus as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A03:00/LNXVIDEO:00/input/input6
ACPI: Video Device [GFX0](multi-head: yes rom: no post: no)
alloc irq_desc for 18 on node -1
alloc kstat_irqs on node -1
pci 0000:00:02.0: PCI INT A -> GSI 18 (level, low) -> IRQ 18
[drm] Initialized vboxvideo 1.0.0 20090303 for 0000:00:02.0 on minor 0
sda: sda1 sda2
EXT4-fs (dm-0): mounted filesystem with ordered data mode. Opts:
type=1404 audit(1434533752.241:2): enforcing=1 old_enforcing=0 auid=4294967295 ses=4294967295
SELinux: 2048 avtab hash slots, 295064 rules.
SELinux: 2048 avtab hash slots, 295064 rules.
SELinux: 9 users, 12 roles, 4142 types, 228 bools, 1 sens, 1024 cats
SELinux: 81 classes, 295064 rules
SELinux: Completing initialization.
SELinux: Setting up existing superblocks.
SELinux: initialized (dev dm-0, type ext4), uses xattr
SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
SELinux: initialized (dev selinuxfs, type selinuxfs), uses genfs_contexts
SELinux: initialized (dev mqueue, type mqueue), uses transition SIDs
SELinux: initialized (dev hugetlbfs, type hugetlbfs), uses transition SIDs
SELinux: initialized (dev devpts, type devpts), uses transition SIDs
SELinux: initialized (dev inotifyfs, type inotifyfs), uses genfs_contexts
SELinux: initialized (dev anon_inodefs, type anon_inodefs), uses genfs_contexts
SELinux: initialized (dev pipefs, type pipefs), uses task SIDs
SELinux: initialized (dev debugfs, type debugfs), uses genfs_contexts
SELinux: initialized (dev sockfs, type sockfs), uses task SIDs
SELinux: initialized (dev devtmpfs, type devtmpfs), uses transition SIDs
SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs
SELinux: initialized (dev proc, type proc), uses genfs_contexts
SELinux: initialized (dev bdev, type bdev), uses genfs_contexts
SELinux: initialized (dev rootfs, type rootfs), uses genfs_contexts
SELinux: initialized (dev sysfs, type sysfs), uses genfs_contexts
type=1403 audit(1434533753.351:3): policy loaded auid=4294967295 ses=4294967295
sd 0:0:0:0: [sda] Synchronizing SCSI cache
e1000 0000:00:08.0: PCI INT A disabled
e1000 0000:00:03.0: PCI INT A disabled
Restarting system.
machine restart
ACPI MEMORY or I/O RESET_REG.

rhvgoyal · 2015-06-17T12:00:32Z

@Khazrak This sounds like a kernel issue. Can you open a bug at https://bugzilla.redhat.com/ to track this.

Khazrak · 2015-06-17T13:07:53Z

@rhvgoyal Done. https://bugzilla.redhat.com/show_bug.cgi?id=1232779

Bug nr 1232779

strootman · 2015-06-17T15:34:09Z

I'm not able to see the redhat bug. @Khazrak, if you are able, could you add any updates from the bug here, when/if at all possible?

lsm5 · 2015-06-17T15:37:19Z

@strootman perhaps I could try cc'ing your email to that bug, do you have an account on the bz site?

strootman · 2015-06-17T15:56:35Z

@lsm5 yes I do, under jstroot.iplant@gmail.com

lsm5 · 2015-06-17T16:05:02Z

@strootman done, please check

strootman · 2015-06-17T16:07:11Z

@lsm5 I can see it. Ty!

sghosh151 · 2015-06-18T15:38:09Z

https://access.redhat.com/solutions/1378023

The Red Hat provided, maintained and supported version of Docker is only available on RHEL 7.

Red Hat does not recommend running any version of Docker on any RHEL 6 releases. Docker feature developments are tied to RHEL7 infrastructure components for kernel, devicemapper (thin provisioning, direct lvm), sVirt and systemd.

thaJeztah · 2015-06-18T17:07:08Z

@tiborvass @moxiegirl @vbatts ^^ does that mean we should drop RHEL6 from the installation docs as well?

vbatts · 2015-06-18T17:17:27Z

Dropping the docs may confuse some. Likely better to have notice requiring
support. That the usage of docker on rhel6 (or centos6) is at your own
risk, it is not officially supported by redhat or Docker, and fixes are on
a best effort basis.
On Jun 18, 2015 13:07, "Sebastiaan van Stijn" notifications@github.com
wrote:

@tiborvass https://github.com/tiborvass @moxiegirl
https://github.com/moxiegirl @vbatts https://github.com/vbatts ^^
does that mean we should drop RHEL6 from the installation docs as well?

—
Reply to this email directly or view it on GitHub
#9856 (comment).

thaJeztah · 2015-06-18T17:28:44Z

@vbatts agreed; must say I wasn't aware RH communicated this (not recommending Docker on RHEL6). Was a bit surprised, and under the assumption that they did support it and that being the reason we tried to keep Docker working on it.

I'll open a separate issue when I get home, so that it can be further discussed without cluttering the discussion here.

tpires · 2015-06-22T10:30:42Z

While I was trying to setup swarm on my node I got a kernel panic on 2.6.32-504.16.2.el6.x86_64.

visualphoenix · 2015-06-25T01:56:01Z

I must say for both me and my company it would be catastrophic to lose docker support for RHEL/CentOS 6.x. For the longest time Docker has stated support for RHEL/Centos 6.5. Recently updated to a minimum of 6.6. For us, upgrading to 7 is out of the question for the near term.

alexanderilyin · 2015-07-07T02:18:12Z

Just faced with same:

Docker version 1.6.2, build 7c8fca2/1.6.2
Cent OS 2.6.32-504.23.4.el6.x86_64

catrixs · 2015-07-08T08:26:49Z

got panic

Docker version 1.6.2
CentOS 6.5 2.6.32-431.23.3.el6.x86_64

sprijk · 2015-07-08T17:47:54Z

+1 same trace

Khazrak · 2015-07-13T06:36:49Z

@feiyang21687 Correct me if I'm wrong but Docker didn't work on RHEL/CentOS until Cent OS 2.6.32-504.x (you said your version was 2.6.32-431.x).
The reason for this was that Redhat (or someone) backported functionality to the old kernel so that Docker could be used.

catrixs · 2015-07-13T07:37:04Z

@Khazrak yeah, you are right. But it is ok when we run docker 1.3.2 on 2.6.32-431.x.

vbatts · 2015-07-13T12:10:39Z

The version compatibility matrix is all over the place. There have plenty
of versions of RHEL that "worked". As new features are added to docker,
then new capabilities and fixes were backported to rhel6.
On Jul 13, 2015 03:37, "陈飞" notifications@github.com wrote:

@Khazrak https://github.com/Khazrak yeah, you are right. But it is ok
when we run docker 1.3.2 on 2.6.32-431.x.

—
Reply to this email directly or view it on GitHub
#9856 (comment).

amaltson · 2015-07-17T13:26:48Z

Our servers were just upgrade to Red Hat 6.5 2.6.32-504.23.4.el6.x86_64 and I've been having kernel panics probably 10 times during that day. This was original with Docker 1.6.1. We've now upgraded to Docker 1.7.1 and continue to have kernel panics.

$ sudo docker info
Containers: 2
Images: 158
Storage Driver: devicemapper
 Pool Name: docker-253:2-29633562-pool
 Pool Blocksize: 65.54 kB
 Backing Filesystem: extfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 16.76 GB
 Data Space Total: 107.4 GB
 Data Space Available: 90.62 GB
 Metadata Space Used: 16.11 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.131 GB
 Udev Sync Supported: false
 Deferred Removal Enabled: false
 Data loop file: /transient/opt/docker/devicemapper/devicemapper/data
 Metadata loop file: /transient/opt/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.82-git (2013-10-04)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 2.6.32-504.23.4.el6.x86_64
Operating System: <unknown>
CPUs: 4
Total Memory: 13.6 GiB
Name: devbuild1
ID: BURZ:PUHG:ECHK:KRO2:7N3L:HD7O:TJZE:VQP2:LFQL:3N3R:STCS:LBSE

tuesdaythefifth · 2015-07-24T16:44:16Z

does this still occur with 6.7?

https://access.redhat.com/articles/3078#RHEL6
RHEL 6 Update 7 2015-07-22 2015-07-22 RHEA-2015:1423 2.6.32-573

marcellodesales changed the title ~~Host connection completely lot after mounting volume Docker 1.4.0, Docker 1.4.1, RHEL 6.5, with latest device mapper~~ Kernel Panic when mounting volumes on RHEL 6.5 using Docker 1.4.0, Docker 1.4.1, with latest device mapper Dec 31, 2014

SvenDowideit added area/storage/devicemapper bug labels Jan 2, 2015

jessfraz added /os/linux/rhel exp/expert labels Mar 2, 2015

kashyap-parikh mentioned this issue Mar 2, 2015

Machine stuck running DinD on CentOS 6.5 #11072

Closed

spf13 removed /os/linux/rhel area/storage/devicemapper exp/expert labels Mar 21, 2015

Khazrak mentioned this issue Jun 24, 2015

1.7.0/CentOS/RHEL 6.6 - bridge interface creation fails. daemon won't start. #14024

Closed

This was referenced Jun 25, 2015

Continued RHEL/CentOS6.6+ support #14174

Closed

Kernel Panic when build image #14033

Closed

Kernel Panic when mounting volumes on RHEL 6.5 using Docker 1.4.0, Docker 1.4.1, with latest device mapper #9856

Kernel Panic when mounting volumes on RHEL 6.5 using Docker 1.4.0, Docker 1.4.1, with latest device mapper #9856

Comments

marcellodesales commented Dec 30, 2014

The problem

Reproducing

marcellodesales commented Dec 30, 2014

Upgrade from 1.2.0 to 1.4.0 reuses the same /var/lib/docker

Questions

Workaround: Downgraded to Docker 1.3.3

marcellodesales commented Dec 31, 2014

chenchun commented Jan 14, 2015

unclejack commented Jan 21, 2015

vbatts commented Jan 21, 2015

marcellodesales commented Jan 22, 2015

Docker 1.4.x

Docker 1.3.x

Updating the versions

What's next?

chenchun commented Jan 24, 2015

rhvgoyal commented Jan 29, 2015

chenchun commented Jan 30, 2015

rhvgoyal commented Jan 30, 2015

marcellodesales commented Feb 7, 2015

unclejack commented Feb 7, 2015

marcellodesales commented Feb 7, 2015

Using the --net host also breaks

kashyap-parikh commented Feb 17, 2015

larrymyers commented Feb 17, 2015

kashyap-parikh-ck commented Feb 18, 2015

Johnsmith1111 commented Mar 2, 2015

seanclerkin commented Mar 2, 2015

Johnsmith1111 commented Mar 3, 2015

Johnsmith1111 commented Mar 5, 2015

hxfeng commented Jun 3, 2015

strootman commented Jun 15, 2015

jamesdobson commented Jun 15, 2015

Khazrak commented Jun 16, 2015

rhvgoyal commented Jun 16, 2015

Khazrak commented Jun 17, 2015

rhvgoyal commented Jun 17, 2015

Khazrak commented Jun 17, 2015

strootman commented Jun 17, 2015

lsm5 commented Jun 17, 2015

strootman commented Jun 17, 2015

lsm5 commented Jun 17, 2015

strootman commented Jun 17, 2015

sghosh151 commented Jun 18, 2015

thaJeztah commented Jun 18, 2015

vbatts commented Jun 18, 2015

thaJeztah commented Jun 18, 2015

tpires commented Jun 22, 2015

visualphoenix commented Jun 25, 2015

alexanderilyin commented Jul 7, 2015

catrixs commented Jul 8, 2015

sprijk commented Jul 8, 2015

Khazrak commented Jul 13, 2015

catrixs commented Jul 13, 2015

vbatts commented Jul 13, 2015

amaltson commented Jul 17, 2015

tuesdaythefifth commented Jul 24, 2015