Continued RHEL/CentOS6.6+ support

[visualphoenix]

The recent update to https://access.redhat.com/solutions/1378023 suggests that RedHat has decided not to support Docker on RHEL 6.6+ anymore.

After seeing recent discussions in #14024 and resurfaced kernel panics in #9856, it would be nice to know if Docker will continue to support 6.6+.

This ticket is to discuss 6.6+ support (and maybe be a metaticket for 6.6+ issues)

My 2c: I hope to continue to see docker supported on 6.6+ systems.

[visualphoenix]

Also being discussed in #9856

[visualphoenix]

ping folks who might be interested in commenting about 6.x support @shykes @rhatdan @noelob @sgykfjsm @cpuguy83 @Khazrak @vinnyspb @gavinwhyte @pdericson @LK4D4 @icecrime @thaJeztah @jfrazelle @mavenugo @marcellodesales @SvenDowideit @chenchun @unclejack @vbatts @rhvgoyal @kashyap-parikh @larrymyers @kashyapparikh-ck @Johnsmith1111 @seanclerkin @tpires @nekodojo @oscarrenalias @jmzwcn @hxfeng @strootman @jamesdobson @lsm5 @sghosh151

[gavinwhyte]

@shykes @rhatdan @noelob @sgykfjsm @cpuguy83 @Khazrak @vinnyspb @gavinwhyte @pdericson @LK4D4 @icecrime @thaJeztah @jfrazelle @mavenugo @marcellodesales @SvenDowideit @chenchun @unclejack @vbatts @rhvgoyal @kashyap-parikh @larrymyers @kashyapparikh-ck @Johnsmith1111 @seanclerkin @tpires @nekodojo @oscarrenalias @jmzwcn @hxfeng @strootman @jamesdobson @lsm5 @sghosh151

Looks like people are aborting Docker, I feel the same if I cant get it working on RHEL 6, so many organisations will loose out. Have docker no idea about the amount of large organisations that use Rhel or Centos 6. Massive banks, just cannot upgrade and move. I understand innovation is important, but supporting centos 6.x to some degree is important as well.

[cpuguy83]

Docker has made no such announcement.
Clearly we have a regression in Docker 1.7 as part of the networking rewrite.
The issue is known and is being addressed.

[samalba]

@cpuguy83 is right. And 1.7.1 is on the way to address this regression.

[gavinwhyte]

@cpuguy83 I do have question, has anyone bothered to test on Centos 6.x before release.

[visualphoenix]

@cpuguy83 but there is also #9856 which is more than a network regression. There seem to be larger regressions with newer RHEL/CentOS kernels

[visualphoenix]

Furthermore, from the comments in #14024 it sounds like there is a kernel panic with even older versions of docker and the latest RHEL/CentOS kernel. Likewise, we have @thaJeztah asking if support should be removed from docs, @vbatts commenting about the fact that 'the usage of docker on rhel6 (or centos6) is at your own risk, it is not officially supported by redhat or Docker, and fixes are on a best effort basis.', RedHat suddenly backpedaling on 6.x: https://access.redhat.com/solutions/1378023

I realize that 1.7.1 is already planning on fixing the network regression I mentioned before. What I'm more interested in is knowing if docker is actually officially going to state that 6.x is no longer supported and 'use at your own risk'? Is Docker going to work with RedHat to do better than a 'best effort' basis for support?

[gavinwhyte]

I have had kernel panics with docker 1.6. and centos 6.5.

[visualphoenix]

@gavinwhyte burried in issue notes, you'll find that 6.6 is now the minimum supported version.

[visualphoenix]

@gavinwhyte to quote @Khazrak, there seems to be "... a problem with the latest kernel 2.6.32-504.23.4.el6.x86_64."

@gavinwhyte I've had good success in production with both CentOS and RHEL 6.6 using 2.6.32-504.8 and Docker 1.6.2

[visualphoenix]

@gavinwhyte specifically #12814 updated docs changing the minimum requirements to 6.6 and actually references #9856 from my first post

[thaJeztah]

@visualphoenix at this moment, the position of Docker regarding RHEL6 support hasn't changed; the project is fully committed to keeping it supported on those systems. I don't know if this means that all future features and versions can be supported on these systems, given that Docker is relying on some "cutting edge" kernel features. For those, I think we depend on support from RedHat, to make sure these features are supported (in the kernel).

In addition, I know Docker (Inc.) offers a commercially supported "docker engine", which could be a good alternative for those organizations looking for a "long term support" version of Docker

All this is my personal view at things; to my best knowledge

[simar0at]

I always get a kernel panic in Centos kernel.x86_64 0:2.6.32-504.23.4.el6 when trying to stop my docker container. We use a VMWare virtualization environment docker-io is from the EPEL repos: 1.5.0. Using the previous patch level 2.6.32-504.16.2.el6.x86_64 I don't see that problem. RedHat claims to have fixed some serious looking security issues in 23.4 so I'm not very comfortable with using 16.2. I hope there is a solution to this soon.

[tpires]

@simar0at As I reported on #9856 (comment) I got kernel panic while on 2.6.32-504.16.2.el.x86_64 (not so often, but it happened).

Still, I'm stuck on that version rather on 23.4 which would kernel panic when trying to stop a container.

[Khazrak]

I've now tried it on multiple hardware (and some VM's). All the VM's I've tried CentOS 6.6 on get the same kernel panic's. On one "real" server with CentOs the same kernel crash occurred but one laptop with the same CentOS from the same CD worked.

And one of our RHEL 6.6 Test servers (don't know if it's physical or virtual) actually works. So it might be hardware-kernel related? But we can't roll out Docker in production if a future kernel update might crash it too... And you don't want to have production servers that doesn't get the security fixes for years (so not upgrading the kernel is not an option).

Since the kernel problems came after 2.6.32-504.8 it feels like someone bright might find a kernel issue that came after this version.

The 2.6.32 kernel lost it's official support in April (https://en.wikipedia.org/?title=Linux_kernel#2.6.x.y_releases), will Redhat switch or maintain it? If they maintain it they should be able to fix it

[visualphoenix]

Adding a reference to #14181 #13769 #14033 here since it also affects 6.6+

[simar0at]

will Redhat switch? You're joking, aren't you? I can't imagine that happening. They maintain 2.6.18 because it's in RHEL5, they maintained 2.6.9, which was officially broken, for years. No chance they switch. But they do have a problem that might be considered a security issue. Docker isn't running using super powers, is it? If not then it should never be able to cause a kernel panic.

[dblooman]

Would be good for us enterprise users still on 6 to have this resolved soon.

[smerrill]

I wrote up some research about what's happening and what options you have today related to this issue in #14181 (comment) .

(And I'd like to again echo a huge thanks to everyone who has helped get this awesome project running on EL6. 👍)

[LK4D4]

Problem, that support should be from two sides, RedHat and Docker(support is impossible without any of side). And seems like RedHat isn't interested in docker on EL6.

[icecrime]

After discussions among maintainers, we decided on the following:

• Docker 1.7.0 breaks on RHEL6/CentOS6: this was unintentional, and will be fixed in 1.7.1 (tentatively scheduled for July 9th).
• Moving forward, Docker will drop support for RHEL6/CentOS6 starting with version 1.8.0. As mentioned by @LK4D4, RedHat themselves don't support Docker on those versions: we can't possibly do this alone, and compatibility with 2.6.x based kernels is becoming too much of a burden.

I hope this clarifies it, and sorry for those that will suffer from this decision.

[visualphoenix]

1.8 is so soon, could we possibly schedule this for, say, 6 months from now?

[icecrime]

@visualphoenix We cannot afford it: it's too much efforts, and we don't have CI on those older systems.

We can't guarantee that we won't regress. RedHat doesn't guarantee that they won't regress. I'm sorry but dropping support is our only option here.