Kernel Panic when container restart (docker-compose restart) on centos6.6 #14181
Comments
|
Hi! Please read this important information about creating issues. If you are reporting a new issue, make sure that we do not have any duplicates already open. You can ensure this by searching the issue list for this repository. If there is a duplicate, please close your issue and add a comment to the existing issue instead. If you suspect your issue is a bug, please edit your issue description to include the BUG REPORT INFORMATION shown below. If you fail to provide this information within 7 days, we cannot debug your issue and will close it. We will, however, reopen it if you later provide the information. This is an automated, informational response. Thank you. For more information about reporting issues, see https://github.com/docker/docker/blob/master/CONTRIBUTING.md#reporting-other-issues BUG REPORT INFORMATIONUse the commands below to provide key information from your environment:
Provide additional environment details (AWS, VirtualBox, physical, etc.): List the steps to reproduce the issue: Describe the results you received: Describe the results you expected: Provide additional info you think is important: ----------END REPORT --------- #ENEEDMOREINFO |
|
I don't think we support kernels as old as In either case, this is a Linux kernel bug which is either caused by some CentOS patch or is an upstream bug. Pop an email to the CentOS guys first to see if the problem is on their end, if it isn't send an email to the stable kernel maintainers to see if someone can take a look at it. They'll probably want a decompilation of the relevant kernel code (a general protection fault is caused by invalid memory accesses that violate protection policies of the CPU), as well as some more hardware-specific information. |
|
@cyphar Please don't recommend custom kernels on CentOS 6 and RHEL 6. They're not in any way supported. This might indeed be a kernel issue and needs to be investigated. Marking as a kernel and CentOS issue. |
|
@unclejack The reason I was asking him to try with a more modern kernel is to see if this bug is present in modern kernels (or if it was fixed and has yet to be backported, or if it is a bug in a CentOS patch). I wouldn't dream of running custom kernels in production (especially on CentOS). |
|
Agreed with @unclejack - I'm sure you were trying to be helpful, but the docker team has only ever attempted to support official RHEL/CentOS kernels. 2.6.32-504.23.4.el6.x86_64 is an official security update from RH: https://rhn.redhat.com/errata/RHSA-2015-1081.html See: https://github.com/docker/docker/blob/release/docs/installation/centos.md and https://github.com/docker/docker/blob/release/docs/installation/rhel.md Thanks @unclejack for marking this for investigation. |
|
I downgrade kernel to 2.6.32-504.16.2.el6.x86_64, it seems works fine. |
|
@lostsnow Okay, so I've taken a look at the changelog, here is the list of patches applied during that period: Is it possible for you to see if you can reproduce on |
|
Hey all, Also can confirm running the latest kernel ( |
|
@pmyjavec What is the version of the "older kernel" you booted from? |
|
Hello @cyphar, The working version is |
|
@cyphar I can not find kernel rpm 2.6.32-504.22.* (http://mirror.centos.org/centos/6/centosplus/x86_64/Packages/), Can someone provide it? |
|
Sorry, that means that they didn't actually release it (weird). Anyway, so that means we'll have to git bisect with the entire changeset. I'll take a look at this. Can this be reproduced on a stock CentOS 6.6 install? |
|
My install was: It generates panics. First I removed: And I installed: The 2.6.32-504.16.2 version did not fix it, but the 2.6.32-504.8.1 will do. |
|
@jophofste Can you check if |
|
@cyphar I've been running the EPEL provided docker-io-1.5.0-1 RPMs with kernel-2.6.32-504.16.2 for weeks without any issues. Upon upgrading to kernel-2.6.32-504.23.4, I immediately started seeing panics on container shutdown. I think the problem is between 16.2 and 23.4. I'd install the latest version of docker from the docker-provided RPMs, but they have been built poorly and expect 32-bit packages to be installed that conflict with 64-bit packages. There's another issue opened for this. |
|
@hrunting Since we're seeing a kernel panic, I don't expect an updated Docker version to fix this problem (even if it does, this looks like a kernel bug proper to me). I'm still trying to figure out a nice way of bisecting the CentOS kernel tree. I'll get back to you on this. |
|
I’d like to start this thread with a heartfelt thanks to everyone in the Docker and Red Hat communities who have worked to bring this awesome project to EL6 and maintain it there. Here’s the result of my research on this issue. The most recent RHEL 6.6 kernel version (kernel-2.6.32-504.23.4.el6) has a regression in its handling of cgroups which will often cause kernel panics when used with applications like Apache Mesos, cgroup_monitor, and Docker. In my experience, starting the Docker daemon will nearly immediately panic the system. See https://bugs.centos.org/view.php?id=7538 for an example bug about a different application that uses the cgroups subsystem. The fix was committed to the mainline kernel in 2013 and Red Hat has also reportedly confirmed in that the fix for this will be in the RHEL 6.7 mainline kernel version kernel-2.6.32-564.el6 when that comes out. At this point if you would like to use Docker on EL6, you’ve got 2 major options:
My results are not guaranteed to be conclusive for your workload, but I’ve had very good results running Docker 1.6.2 using the CentOS Plus kernel in a test environment. As part of my research on the CentOS Plus kernel I compared its spec file to the mainline CentOS kernel, and here’s the list of patches applied. They appear to primarily be backported race fixes. |
|
@smerrill Thanks for doing the bisect, you're a much stronger man than I. |
|
Thankfully, no bisect was needed - when I found that other CentOS bug about a cgroups panic, it pointed me to the patch to fix it (and the reported inclusion of the fix in RHEL 6.7.) Also interestingly, it looks like this bug has actually been around for a few RHEL kernel releases, but Docker seems to trigger it instantly with the newest kernel release. |
|
Good! # uname -a
Linux serv-11-1-171 2.6.32-504.23.4.el6.centos.plus.x86_64 #1 SMP Wed Jun 10 13:09:42 UTC 2015 x86_64 x86_64 x86_64 GNU/Linuxthanks @smerrill |
|
Says here 6.7 has been released. Can anyone confirm the problem is fixed? https://access.redhat.com/articles/3078#RHEL6 |
|
@cyphar The version you proposed will fix it. It is a fix till we upgrade to CentOS 6.7. |
|
Today we upgraded two lower environment VMs to CentOS 6.7 kernel 2.6.32-573.3.1.el6 and that seems to have solved the kernel panics we were getting with I have been running a loop to stress the system:
It's been looping for over 20 minutes without panics. On CentOS 6.6 this loop would cause kernel panics after a few cycles. Fingers crossed that this may actually be solved. UPDATE: I stopped that loop test after running it continuously for almost 4 hrs. No kernel panics. |
|
Got confirmation in another issue, that |
|
This issue is the same as #15057. Please keep your systems fully updated to get fixes such as this one. Please keep in mind that CentOS 6 and RHEL 6 are unsupported with Docker. An upgrade to CentOS 7 is recommended. |
docker info
docker version
vmcore-dmesg.txt: http://pastebin.com/Zs6QWLEB
The text was updated successfully, but these errors were encountered: