Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Drop support for RHEL6/CentOS6 #14365

Closed
1 of 3 tasks
icecrime opened this issue Jul 2, 2015 · 57 comments
Closed
1 of 3 tasks

Drop support for RHEL6/CentOS6 #14365

icecrime opened this issue Jul 2, 2015 · 57 comments
Labels
Milestone

Comments

@icecrime
Copy link
Contributor

icecrime commented Jul 2, 2015

Description

Current state of things:

We shipped 1.7.0 without mentioning we would break compatibility, so this will be fixed in 1.7.1, but we will drop support in 1.8.0.

How to

  • Documentation update
  • Consider making --userland-proxy=false a default?
  • Drop older kernel compatibility code (cf. bridge creation fallback, there's probably more)

References

Ping @moxiegirl @thaJeztah to anticipate documentation change and spread the words on related issues / PRs!

@icecrime icecrime added this to the 1.8.0 milestone Jul 2, 2015
@visualphoenix
Copy link

👎 would prefer if we could deprecate 6.x in the future but many institutions haven't migrated to systemd-capable systems yet.

@visualphoenix
Copy link

1.8.0 is so soon. could we schedule dropping support for 6.x in, say, 6 months?

@unclejack
Copy link
Contributor

@visualphoenix No, it's been on life support ever since RHEL7 and CentOS7 were released.

@crosbymichael
Copy link
Contributor

@visualphoenix you could still run docker 1.6 on those distros and 1.7.1 when it is released.

@visualphoenix
Copy link

can docker support the 1.7.x series for 6.x for a while going forward? it seems the 1.7.x series has a bunch of regressions

@visualphoenix
Copy link

@crosbymichael the networking features and volume support for 1.8 was pretty huge for 6.x to get. also there have been so many new things baking for v2 registry which haven't stabilized.

@LK4D4
Copy link
Contributor

LK4D4 commented Jul 2, 2015

@visualphoenix Yes, 1.7 will be supported on RHEL6 until they introduce bug to kernel, which we won't be able to fix on our side.

@visualphoenix
Copy link

@LK4D4 i mean, will there be future 1.7.x releases improving stability. Docker hasnt been great in the past about backporting fixes to earlier release series.

@icecrime
Copy link
Contributor Author

icecrime commented Jul 2, 2015

@visualphoenix We stick to the project rules of supporting our latest version, and backport security fixes (where applicable) to version N-1. Right now 1.7.0 is our focus: we are fixing regressions in 1.7.1, and we'll fix more in 1.7.2 if necessary.

@visualphoenix
Copy link

👎 this sucks.

@icecrime
Copy link
Contributor Author

icecrime commented Jul 2, 2015

@visualphoenix I don't think it does: as I said, we will make 1.7.x solid, even on RHEL6. However, we simply don't have the capacity to do this later on once 1.8.0 will be out.

@unclejack
Copy link
Contributor

Red Hat has been doing a lot of work on the 2.6.32 kernel in order to make it work with Docker. As far as I'm concerned, this was mostly work done to make Docker work on RHEL 6 / CentOS 6 until RHEL 7 / CentOS 7 would get released and the customers could use RHEL 7 / CentOS 7 to run containers.

Unfortunately, it seems that the 2.6.32 kernel requires too much work to backport code and keep it fully functional when it comes to containers. The 2.6.32 kernel had only some of the features containers use nowadays on modern kernels. Expecting this very old kernel to be stable when running containers isn't reasonable, even if a lot of kernel developers work on it and if they've put a lot of time into making it better. Since this is a very old kernel, backporting some of the latest kernel features and fixes takes considerable effort, and is an ever increasing effort with diminishing returns (more effort needed as the kernel's code changes, people going straight to RHEL 7 / CentOS 7 for new deployments).

There are a lot of companies and people out there still running RHEL 6 / CentOS 6, but this is not a good environment for running containers.

Moving forward, the Docker project can't support RHEL 6 / CentOS 6 because those two no longer support it. That means that those two distributions shouldn't be used for new deployments where containers need to be used. The reason behind this is that they'd have to put too much effort into making the kernel work properly with containers. Since they're not going to patch such kernel bugs, we wouldn't be able to fix 99.9% of the reported issues which only occur on those systems, thus not being able to properly support those systems.

This isn't a matter of not wanting to help or being mean to someone by dropping support for RHEL 6 / CentOS 6. It's simply about allowing Red Hat and the CentOS project to focus on making RHEL 7 / CentOS 7 better for running containers. The Docker project will also benefit from not having to support the only family of distributions with such an old kernel and not having to hold back on making it more stable everywhere else. The user land proxy had to be kept around because of RHEL 6 / CentOS 6.

Long story short: supporting containers on RHEL 6 / CentOS 6 was increasingly challenging for Red Hat and they've decided to stop supporting it; since they're no longer going to make changes to the kernel and to the associated user land components, we can't support it any further because we can't fix the kernel bugs. We can only try to make sure Docker still runs on it and that's about it. Anyone who runs into a kernel panic on those systems while running containers will have a hard time getting help.

As a Docker maintainer who's put a few hundred hours into supporting people who run into problems with Docker on RHEL 6/CentOS 6 and who has tested Docker on these distros, I can only recommend the following:

  1. upgrade to RHEL 7 / CentOS 7
  2. run the software which requires RHEL 6 / CentOS 6 in RHEL 6 / CentOS 6 containers
  3. keep your systems fully updated with yum update
  4. do not use custom kernels (including kernels from elrepo, mainline kernels, custom rpms, etc)

@logicrime
Copy link

Visualphoenix hit the nail on the head, this DOES suck.

@visualphoenix
Copy link

@icecrime i'm confused when you say:

we will make 1.7.x solid, even on RHEL6.

By solid, that would mean fixing a bunch of 1.7.0 bugs which you guys currently have set to fix for 1.8.0, like: #14337 #13681 #12080 #11394 #11065 #9570 to name a few.

@crosbymichael
Copy link
Contributor

@visualphoenix Docker is opensource so you are free to help out the project and yourself by submitting patches for various bugs that you care about. The maintainers are always available to answer any questions if you get stuck.

I think everyone would be very happy to do another point release with a batch of bug fixes.

@ghost
Copy link

ghost commented Jul 2, 2015

Congrats for breaking your software on the most widely used OS distro on planet Earth.

@icecrime
Copy link
Contributor Author

icecrime commented Jul 2, 2015

@fcntl Again, the most widely used OS distro on the planet dropped support for our software. We cannot fix this alone.

@visualphoenix
Copy link

@icecrime actually the support document says:

Red Hat does not recommend running any version of Docker on any RHEL 6 releases. Docker feature developments are tied to RHEL7 infrastructure components for kernel, devicemapper (thin provisioning, direct lvm), sVirt and systemd.

Docker features are only tied to whatever components Docker wants to support. Right now, docker works on the upcoming 6.7 kernel. I could see making the minimum supported kernel version be the 6.7 kernel, but does not recommend is not the same as does not support.

@lostsnow
Copy link

lostsnow commented Jul 3, 2015

Bad news 💔
Most of my servers are running CentOS 6.x

@bhuvan
Copy link

bhuvan commented Jul 3, 2015

@icecrime @unclejack @crosbymichael To be clear, it's more to do with kernel version than the OS version. By saying Drop support for RHEL6/CentOS6 you claim you wont support kernel =<2.6.x. right?

if users use redhat certified 3.x kernel with RHEL6/CentOS6, they can still use docker 1.8+. correct me if i'm wrong.

@FomalhautWeisszwerg
Copy link

@visualphoenix Just fork and do yourself. Docker is opensource. Why don't you do?

@jamshid
Copy link
Contributor

jamshid commented Jul 3, 2015

+1 Thanks for not trying to support RHEL6/CentOS6 through another release, and instead spending that time on general, forward-moving fixes and features.
Heartbroken users, the pain of upgrading your OS is less than the pain of trying to run docker in an environment that really doesn't want to run it.

@leafmulch
Copy link

+1 for @bhuvan's question.

@thaJeztah
Copy link
Member

@bhuvan @leafmulch see the earlier comment; #14365 (comment)

As a Docker maintainer who's put a few hundred hours into supporting people who run into problems with Docker on RHEL 6/CentOS 6 and who has tested Docker on these distros, I can only recommend the following:

  1. upgrade to RHEL 7 / CentOS 7
  2. run the software which requires RHEL 6 / CentOS 6 in RHEL 6 / CentOS 6 containers
  3. keep your systems fully updated with yum update
  4. do not use custom kernels (including kernels from elrepo, mainline kernels, custom rpms, etc)

We don't support using custom kernels, only the kernel that ships with a distribution. It could work, but it's asking for trouble and you're fully at your own risk there. The steps above are really the better approach if you want something stable and care about your data.

@leafmulch
Copy link

@thaJeztah understood, but here's my current situation: https://docs.docker.com/installation/oracle/

Docker requires the use of the Unbreakable Enterprise Kernel Release 3 (3.8.13) or higher on Oracle
Linux. This kernel supports the Docker btrfs storage engine on both Oracle Linux 6 and 7.

Thanks.

@thaJeztah
Copy link
Member

@leafmulch thanks for pointing that out, not sure what the situation will be there. That docs may need to be updated as well? @unclejack

@JadeQin
Copy link

JadeQin commented Jul 3, 2015

If I use 3.10.80-1.el6.elrepo.x86_64 kernel in centos 6 ,can I continue use docker 1.8?
Thanks

@unclejack
Copy link
Contributor

@leafmulch @loveai88 You can try that, but you will probably run into bugs if the devicemapper libraries are in any way incompatible with your kernel or if there's any kind of incompatibility between that newer kernel and the user land components.

The user land libraries are more adequate for kernel 2.6.32 and it wouldn't be a surprise to see that there are problems caused by switching to a newer kernel.

Please keep in mind that kernel 3.8 is only supported by Oracle Linux today and that might mean you will also run into unique bugs.

Running at least kernel 3.10 on RHEL 6 / CentOS 6 might fix the problems you were seeing, but that's not something we (the people of the Docker project) can support. The reason behind that is that the vendors of those distributions don't support those custom kernels on the distributions and we can't do that either - we can't provide support for something we've got no way to fix bugs on and we can't ask the distribution vendors to fix problems in their distribution which have been encountered in unsupported setups so that Docker can be used.

Please consider upgrading to RHEL 7 / CentOS 7, instead of installing custom kernels.

@barkerd427
Copy link

This is a huge blow to those enterprises who have not even been able to complete the testing of rhel 7. At least we'll be able to upgrade to 1.7 when rhel approves of it, but it's disappointing this was announced so late relative to how fast rhel 6 users probably move.

@thaJeztah
Copy link
Member

@barkerd427 I feel your pain, but this was effectively out of our control, once RHEL upstream ended support. Recent RHEL 6 kernels resulted in panics and strange behavior when using Docker (even on older versions of Docker) . The docker maintainers decided to drop support, because we cannot provide a stable solution without the backing of the distribution upstream, and decided it was better to not support a platform, than to "support" a system that is really instable.

@barkerd427
Copy link

@thaJeztah I know all too well the pains of rhel6 and Docker! I want to move as quickly as I can to rhel7, but we just aren't ready as it had so many changes that have to be tested. I certainly understand the requirement to move away from it. I wish Redhat would have handled this better with their customers. You guys have done an awesome job with this project and supporting outdated kernels. Thank you.

@pdericson
Copy link
Contributor

RHEL / CentOS 6.7 was released on 2015-07-22. The base kernel is now kernel-2.6.32-573.el6.x86_64. This kernel works well with Docker.

Will Docker Inc consider un-dropping support from RHEL / CentOS 6.7 with the 573 kernel?

@rtoma
Copy link

rtoma commented Aug 14, 2015

@thaJeztah would you please answer @pdericson 's question?

We are considering upgrading from RHEL 6.6 to 6.7. Which is so much easier than upgrading to RHEL 7.

If Docker Inc would/could support the 6.7 kernel, that would make a lot of RHEL 6 users very happy.

@thaJeztah
Copy link
Member

@rtoma @pdericson

I don't think it's wise to revert this decision; without the backing of RHEL upstream, we cannot guarantee a stable solution (which, I think, would defeat the purpose of running a long term support OS)

In addition, having to keep the 2.6 kernel into account when developing new Docker features would hold-back the project as a whole, affecting all other users.

You are still able to install and run docker 1.7.1 on RHEL/CenOS 6;
http://yum.dockerproject.org/repo/main/centos/6/Packages/docker-engine-1.7.1-1.el6.x86_64.rpm

But updating to docker 1.8 and above, will require an updated system, or building docker from source.

@Djelibeybi
Copy link
Contributor

Or switch to Oracle Linux 6 and the UEK3, which supports Docker 1.8 and higher.

@pdericson
Copy link
Contributor

I am happily running Docker 1.7.1 on RHEL / CentOS 6.7 :) Note that at night and on weekends I absolutely do not run RHEL / CentOS 6.7!

My point is, Docker Inc. needs to make the choice between meeting their customers where they are right now and pushing for the future without worrying about supporting a major segment of their install base.

I love the fact that so many cool new features are being added to Docker 1.8 and beyond but I also love the fact that Docker 1.7.1 makes getting my job done on RHEL / CentOS 6.7 so, so much easier.

Please stop using Redhat as an excuse. If Docker Inc. breaks compatibility with RHEL / CentOS 6.7 it's on Docker Inc. only and engineers stuck with RHEL / CentOS 6.7 will feel that pain and be forced to look for alternatives.

@gavinwhyte
Copy link

@pdericson Docker doesn't care about doing market research on OS in corporates that could earn potential revenue.

@gavinwhyte
Copy link

@pdericson Maybe if they hire a data scientist, do some proper analysis, maybe they might consider supporting Docker on RHEL 6, quiet possible, just don't care about revenue.

@pdericson
Copy link
Contributor

Good points @gavinwhyte but also keep in mind that automated testing is hard - it's much easier just to support the kernels developers happen to have running on their dev machines

@gavinwhyte
Copy link

@pdericson We all know supporting developers is what makes this product works, not all OS are equal some are more equal than others. But if people are smart enough to figure out market share, then maybe there is an opportunity considering dockers popularity that distros within reason are supported. Mind you RedHad has fixed issues in ver 6.7. At the least Docker should reciprocate. Quiet funny when they use Redhat 6 or Centos 6 as an excuse.

@pdericson
Copy link
Contributor

Alright, I'll stop being a troll and act serious.

I understand why @icecrime raised this issue. But the situation has changed. The 2.6.32-573 kernel that comes with RHEL / CentOS 6.7 is stable. We all just went through a bad time where RedHat released two broken kernels but they have fixed the situation now and it's back to smooth sailing.

Please Docker Inc, @icecrime and others - we think you guys rock, we love docker (why else would we introduce it to enterprise?) - can you please seriously consider making RHEL / CentOS 6.7 with 2.6.32-573 the new minimum support OS + kernel.

@LK4D4
Copy link
Contributor

LK4D4 commented Aug 15, 2015

@pdericson We personally broke nothing intentionally in 1.8, we kept all code(userland proxy, bridge creation through ioctl etc.) for 2.6 kernel. I think you can use it in RHEL6. But we can't just say: all is okay, guys, we recommend you to use docker on RHEL6, because all is not okay - redhat will never again fix 2.6 kernel for docker, me or you.

@gavinwhyte
Copy link

@LK4D4 I totally agree with you broke nothing, Docker has introduced a product that is amazing. Docker depends on Operating Systems. The Operating System concerned is Centos 6. Right now, Redhat has provided fixes, meaning Docker runs on the 2.6.32-573 kernel. Meaning Docker should provide support for this kernel, Redhat has made the attempt to fix. Making a comment like "Redhat will never again fix 2.6 kernel" in rather sad. The mere fact that it works on the kernel currently Docker should support it. When it comes to a point when it does work, we should address it then. Don't assume problems will happen in the future and based on that hypothesis Docker cannot support it now. It works now, Docker 1.8 should happen now. The future has more in store.

@ocontant
Copy link

It is really unfortunate that docker is not supported on RHEL 6. It comes often that the realm of developer is often incompatible with the realm of operations. It's fun as a developer to be on the latest cutting edge version of thing as it ease development by a lot. But operation for many consideration and reason must stay away from the cutting edge version.

It is not true that majority of people will move to RHEL 7 quickly. The majority will complete to move from RHEL 5 to RHEL 6 by 2016 to be just on time before 2017 End of Extended Support and will stick wtih RHEL 6 until 2020 before moving to RHEL 7.

I don't know any bank or fortune 500 company rolling out RHEL 7 already in day to day operations. Even less migrating existing system to RHEL 7. Most company going to RHEL 7 for now are startup, PME and company that doesn't have to comply with ISO, PCI and other management consideration and constraint. They are also the type of company deploying Ubuntu for production, giving even less market for RHEL 7.

In our context, unfortunately for us, it is a complete turn off for Docker until 2018-2020. Seems like container technology is not yet mature enough for our market. (In all due respect for this technology and the people working on it, just meaning it's still too early for us to be able to adopt it. Continue your good work and in a few years we will be able to catch up). My disappointment only comes from the hope I had that we could be able to use it in our context. Can't wait until we can start deploying RHEL 7 in enough volume for our environment, to put in place a new standard using container.

Wish you the best! I'll still play with it in my lab in the mean time :)

@pdericson
Copy link
Contributor

Wow has it been 4 months? I'm still running docker 1.7.1 on rhel / centos 6.7 and it's working ok.

@ocontant
Copy link

Do you have any stability issue so far like network disconnection?

Sent from my Samsung device

-------- Original message --------
From: Peter Ericson notifications@github.com
Date: 2015-12-12 17:27 (GMT-05:00)
To: docker/docker docker@noreply.github.com
Cc: Olivier Contant contant.olivier@gmail.com
Subject: Re: [docker] Drop support for RHEL6/CentOS6 (#14365)

Wow has it been 4 months? I'm still running docker 1.7.1 on rhel / centos 6.7 and it's working ok.


Reply to this email directly or view it on GitHub.

@pdericson
Copy link
Contributor

Network has been ok, did come across #14738 recently but it's rare - I'm considering maintaining an internal fork of 1.7.1 to apply the fix.

@ourway
Copy link

ourway commented May 1, 2017

It's simply impossible for us moving to RHEL7 and I loved docker. Sad!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests