New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add a Celery chart for Helm #4213
Comments
See #4079 (comment) regarding implementing both the livenessProbe and the readinessProbe. |
+1 for a template for such chart. |
Was there ever any progress on this? |
Not yet. I will try to do this in the next version. |
For information, I did an airflow chart based on celery, but I think there could be much more in a celery chart (multi queue, autoscaler,...) |
For what it's worth, a lot of us that use k8s do not like helm. Getting a nice docker image for celery would be great and help a bunch of different use cases. Helm is basically some yaml that would be glued on TOP of the docker image, and IMHO is not that valuable. I suspect helm will slowly die out as people move away, unless perhaps there is a big revamp that fixes all the "bad" parts of helm. |
The big upside of helm is the community. You need a rabbitMQ production ready in your cluster? Helm install. You need to embeded rabbit + reddis + ... in your own chart ? => helm dependency build. In short you do not reimplement everything, you reuse a lot what already exists in your own chart. Our Airflow package embeds celery, but also reuse reddis chart for example, so that I do not have to reimplement the deployment. Same for postgresQL. And if people wants to use their own instance (DBaaS) they could without having to rewrite the deployment. That's very underevaluated. This is what makes helm the defacto standard for the moment, not the template things others do, in a mucher simpler way (kustomize, ksonnet,...).
Worth a try. |
The problem with Helm, is the 75% problem. Often a chart solves 75% of your problem. Missing some flags that you need? Make a PR. Some new issue with a chart? Make a PR or do a local hack. It ends up taking much much more time to "fix" a chart, compared to just creating your own k8s yamls. This is a much bigger topic than this issue, so perhaps not the right forum for it. But google around and you will see a lot of people that dipped toes in Helm have left, it does not seem to much a lot of momentum going forward. Either way - To use helm you need a good docker image. For me to create my own deployment yaml, I need a good docker image. So the next step in either case is to create good docker images ;) If people want to ALSO create a helm chart - more power to them (but I won't use it). But I might indeed consider a docker image if made well. |
I agree. Making a docker image is not difficult (we do have celery in several docker images here). BUT making a generic one is really tricky indeed. |
👋 Helm user here. I must say that having a helm chart ease a lot the deployment (I just hit this issue while trying to package a big app that use celery). I would have loved to see an actively maintained chart with production ready examples and real life feedbacks ^^" I might draft one in the following days and, if it doesn't get too uggly, I might open source it if you don't might (I would be happy to share ownership as I have no experience whatsoever with celery). Also, if you consider publishing a chart, I highly suggest hosting your own repo and publish your chart on https://hub.helm.sh/ (So you don't need to go through Edit: Don't get me wrong: I'm also not very happy with the state of helm as it is now, like many helm users but the draft for helm v3 dropping tiller and moving to an operator pattern will hopefully make everyone happy :D |
As a maintainer for Helm Charts like redis, rabbitmq, etc, and heavy user of celery deployed using helm (currently ~100 celery pods running in prod), here are my 1/ A good docker image is a must-have and comes before a good helm chart. A good Docker image does not necessarily means complex; it can be a few lines long. Edit: after some discussion with some member here, I realize that to use celery, one has to package its code onto the docker image, so one needs to rebuild for each project/commit the docker image. In this case, maybe all that Celery can do is create a base "abstract" image to be "inherited" by custom-celery-image-with-code that would just do some |
Is there any progress/update on this? I was wondering if Celery would like to see the adopt operator pattern in addition to the Helm chart. Let me know if I should create a separate issue for the same. I built a very much POC(https://github.com/brainbreaker/Celery-Kubernetes-Operator) of Celery operator for solving the manual steps needed to manage celery clusters at my work. There's also a discussion going on at celery/ceps#24 for the same. I'm willing to commit a certain number of hours every week to build a production-ready version of the Operator. But I'd need inputs from the community around what will be the requirements, production use cases/opportunities that the operator needs to solve. I also presented this operator POC in my recent talk at EuroPython 2020(Slides link - https://bit.ly/europython20-ppt) around automating the management of Kubernetes infra while staying in the Python ecosystem. |
Yes, we're planning to replace @Brainbreaker If you have experience with Celery and you need that kind of thing at work I suggest you help us. |
I am checking your work. feel free to ping if you want me to collaborate on that effort |
I'd love to. I'm not an expert with Celery internals right now but willing to explore and learn. We use it at our work for various asynchronous workloads and there are pain points of managing all the workers, flower, and scaling things across multiple celery use cases.
Thanks, @auvipy. Code is certainly not the best quality right now so please bear with that. I'll definitely need your inputs. @thedrow Also, please correct me if I'm understanding wrong - The Controller concept we're introducing in Celery 5 will be platform agnostic, and then we'll provide packages(Platform manager), one of which could be the Kubernetes Operator which will be using that controller internally. |
you can currently start with celery 4.4.x and later when the celery 5 enters in rc1 stage we can move the effort to that. (I expect 6-8 months more around) |
No actually we're close to releasing an RC. |
I was actually thinking that the Controller becomes a Kubernetes Operator when it is deployed in k8s. |
this is what we should investigate too. |
you can easily gitsync your code pretty easily, we do that for airflow. |
Is there any official Celery helm chart ? Or Any celery helm chart /Kubernetes yamls that can be used to deploy Celery on Kubernetes ? |
Kubernetes is a very common deployment target these days.
The most common way to deploy an application to Kubernetes is using the Helm package manager.
We need to create a public chart and contribute it to the main charts repository.
The text was updated successfully, but these errors were encountered: