Kernel Panic // dirperm1 breaks the protection by the permission bits on the lower branch #21081
Comments
|
This is a Linux Kernel bug, you should report using the Linux bug tracker if you're using a vanilla kernel or via your distribution bug tracker if you're using their patched kernel. |
|
@mlaventure could we keep this open and just apply the area/kernel label to the issue? So we won't get another issue (hopefully) if this happens to others and to also keep track of this is it's found to be a kernel bug |
|
@runcom done. |
|
Similar issues have been reported elsewhere (#13940, #14816, on google compute). I understand that this appears to be a kernel bug. It also seems that the mainline kernel has fixed this bug. Unfortunately kernel version 3.16, which we are using on debian jessie, does not include the fix. We only began to have kernel panic on our servers starting with docker 1.10. Docker 1.9 worked perfectly fine. So there must be some changes between docker 1.9 and 1.10 that trigger this panic? |
|
I can confirm that the mentioned setup worked perfectly with kernel 3.16 and docker <= 1.9 for at least 5 month. The kernel panics started when we updated docker to 1.10 - among several other debian security packages. |
|
@haskellar Did you only update docker or did other packages upgrade along with it? |
|
@cpuguy83 We had several debian kernel security updates. For servers that are not rebooted yet and still running kernels older than 3.16.7-ckt20-1+deb8u3, they are fine with docker 1.10 and 1.11 and have had no crash. We are not sure about 3.16.7-ckt20-1+deb8u3, because one server did crash once with this kernel version and docker 1.10. But a few other servers have been running this combination for over 2 months without crash. Versions starting with 3.16.7-ckt20-1+deb8u4 are definitely not happy with docker 1.10 and 1.11. |
|
And once we downgrade docker to 1.9.1, there has been no crash with any kernel version. |
|
We are having the same problem on ubuntu 14.04 with kernel 3.19.0-58-generic and Docker 1.11. It seems like docker problem to me. In our case, the container freezes every time we spawn Process in Python. |
|
I'm seeing the same error on 16.04, but only when using the --userns-remap flag. |
|
Same with 14.04: Was fine with 1.9.1 and 3.16 kernel. After the first random crashes started I updated the kernel but this didn't help. |
|
Seeing this on Ubuntu 16.04 when trying, e.g.: In the logs:
Kernel 4.4.0-21-generic |
|
I'm pretty sure this message about dirperm1 is a warning from aufs, not an actual error. |
|
I get the warnings and in some instances the whole deamon crahes and all containers get restarted. I can not say if the warnings and the crashes are connected but they started appearing at the same time. |
|
@kuhnroyal Do you have a stacktrace for the daemon crash? |
|
getting the same error
with |
|
@cpuguy83 I got a stacktrace of my crash: |
|
@tompson This is unrelated and fixed in 1.11.1 |
|
@cpuguy83 you mean the stacktrace I posted is unrelated to the dirperm1 issue? |
|
I see the same using a privileged container on EC2: Is there anything else I can provide to help debug? It's a debian jessie, as the stack trace says |
|
@rata Please report to the debian team. A kernel panic is a bug in the kernel. |
|
@cpuguy83 ok, will report there then, thanks! |
|
@rata Thanks! Interested to know feedback. |
|
@rata Is there a bug created in debian project or in kernel project yet? |
|
On Tue, Jul 12, 2016 at 04:54:06PM -0700, Qi Luo wrote:
Didn't have the time to look properly nor report with an easy to reproduce case. |
|
Same error happening in Kernel Can I say that the amount of bugs that Docker have is ridiculous? 👎 |
|
Do Docker team test their software on specific dependencies? If Docker would provide a page where they recommend a supported kernel version for each Docker version (and other dependencies of the environment), maybe that would make life of sysadmins easier. We should focus on delivering value, not on fixing code. My perception is that every version of Docker is too bugged. |
|
@bitliner If you have a specific issue, please open it so we can take a look. A kernel panic is always a bug in the kernel. In the stack trace above You should always make sure to run the latest version of the kernel provided by your distro. These updates contain bug fixes for just things such as this. If you are on the latest kernel provided by your distro and you still experience a panic, it is best to report to the distro. You can of course report in docker/docker if it happened and is reproducible by using docker and we can figure out what might need to be done. |
|
@bitliner as @cpuguy83 says, a kernel panic is not a bug in docker it is a bug in the kernel (often a security issue in fact as well). We cannot fix it by changing docker. We test on the major distributions especially RHEL 7, Ubuntu LTS, and Debian stable, and on recent stable long term support kernels (4.4.x currently). Generally it is best to open a new issue, as your kernel panic is very unlikely to be the same as someone else's. This issue is particularly confusing as it references a generic aufs message which many people find in their kernel logs and is unrelated. |
|
@justincormack I understand your point. I want just to point out that your recommendation is to always update the kernel and hope there are no bugs (hope?!?). What I was suggesting is to recommend specific versions of the kernel that are properly tested for every vrsion of Docker, so everyone can setup a proper environment. Who does web applications runs browser testing for multiple browsers, it would be nice if you have something similar for the kernels and multiple distros. You can imagine not everyone has knowledge to put hands into kernel and related issues, and waiting for someone else fixing a bug can be painful. And sometimes I have the sensation that the pain of using Docker overcomes its benefits. my2cents |
|
@bitliner No, I said which versions we largely test on. It is a good idea to update, as bugs often get fixed relatively quickly. We do try to avoid using kernel functions with known issues, but it can be hard to know where all the issues are. You have only posted "Same error happening in Docker version 1.12.2, build bb80604" which is totally unhelpful. We don't know what "same" is, or under what conditions it happened. Can you post your actual kernel panic in a new issue? |
|
@bitliner Just because we test on a specific kernel version doesn't mean someone won't hit a kernel bug on that version. In fact, I don't think we've had a single specific kernel (panic) issue that effects a majority (or even some measurable minority) of users on a particular version except explicitly with extremely old kernels. i.e. we used to recommend at least kernel 3.8, now we recommend at least kernel 3.10 because of known issues in 3.8 (and 3.8 is EOL). We recommended 3.8 in the past because of well known issues (security/breakouts issues even) on less than 3.8... but even then for awhile rhel 6 (kernel 2.6.32) was being supported by RedHat with necessary fixes and feature backports (not anymore). The real recommendation is use the latest kernel provided by your distro for the release you are on. As with all software you must test to make sure an upgrade doesn't break something for you. If it does break something then it's best to report the bug so it can get fixed as quickly as possible. Would love to take a look at your specific kernel panic to see what may be causing an issue. Are you seeing several issues or just one common one? |
|
I hope this is useful My environment is Vagrant using image The kernel panic happens if I run a Docker image containing datbase tokumx, that was previoously normally running. What about Docker maintaining their own distro? It could be expensive, it could be out topic, but apparently frustration for Docker is a common thing. :( |
|
@bitliner ok so that is a lockup on the emulated ethernet card on vagrant. Are you running it with virtualbox? This is not really a kernel bug at all it is virtualisation code trying to emulate physical hardware and not doing it well enough. This is often an issue, especially with timing. Virtual network drivers (PV) rather than emulated hardware drivers work better with VMs. |
|
@justincormack Indeed, moved to vmware fusion (also changed vagrant box, I went for a different ubuntu-16.04 box), now it works fine. |
|
hi reproducted in debian. but system didn't panic just show the error log: docker version is 1.12.0 ps: the same docker version & info and system info some of them is running well . some of them happend this issue. so it can not be reproducted every time. |
|
I'm going to close this, as this is a generic issue for people's various kernel problems based on a log entry that has nothing to do with the problem. For new people coming to this: This is not the issue you are looking forThe log message about dirperm1 is a warning from aufs that docker has enabled the This As always, kernel panics are bugs in the kernel and should be reported upstream in your distro's issue tracker. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
I experienced the above mentioned kernel panic several times while running mysqldump-backups (script attached) started by cron on Debian Jessie (Kernel 3.16 / Docker Engine 1.10.1).
docker_mysql_crash_20101003.txt
Output of
docker version:Output of
docker info:Provide additional environment details (AWS, VirtualBox, physical, etc.):
physical
List the steps to reproduce the issue:
Unfortunately it's not crashing every time.
Describe the results you received:
kernel panics, hardware reset needed
Describe the results you expected:
Provide additional info you think is important:
The text was updated successfully, but these errors were encountered: