-
Notifications
You must be signed in to change notification settings - Fork 441
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Expanding PVC Volume Template Results in Data Loss #1385
Comments
Thanks. Would it be possible to attach the operator log as a file to this case? I would like to see if there is an issue with operator reconciliation. If you can access rook logs, please attach those as well. |
@tman5 , could you show your storage classes?
And it would be useful to see one of PVCs created by an operator. |
This is one of the PVCs that will perpetually be in a terminating state:
|
it means someone like ArgoCD changed PVC could you try to deploy CHI without argocd |
Is there a way to make it work with argo? |
Errors can not lead to PVC deletion. I wonder if this is actually ArgoCD that deleted it? |
@tman5 Assuming you are using Argo CD can you describe how you have configured CI/CD and exactly what are the steps you apply to make a change to volume size? It seems possible that multiple actors are trying to manage the CHI resources or at least the underlying volume. p.s., Argo CD normally is fine with changes to storage size. I've done it many times on AWS EBS volumes. |
This is my argo-cd config:
It points to a repo that has a kustomize file:
|
In the clickhouse-operator directory, this is the kustomize file:
|
When trying to expand the PVC volume template the operator will delete/re-create the PVC volumes instead of just resizing them. We are using Rook-Ceph as the storage provider and have successfully resized PVCs without delete/re-create. We can also manually edit the PVC itself and it will expand. We are using version 0.22.2 of the operator. I've reproduced it in multiple clusters.
We have tried it without the
storageManagement
options as well and it just results in a loop where the operator will continually try to delete/re-create the PVCsThe text was updated successfully, but these errors were encountered: