ArgoCD Application Stuck In Syncing/Terminating State #8113

[utkarsh-devops]

Hello ArgoCDians, 👋

We’re facing a weird issue in production where one of the applications is stuck in a terminating and sync state.

Version: 2.1.7

Background:
We manually terminated the sync of an application and now that the application is stuck in the terminating sync state and the disable auto-sync button also not working.

Note: Creating a new application with similar configs is working but we want to investigate why the current application is stuck.

Steps performed:

• We tried disabling the auto-sync from the UI and CLI
• We tried terminating the application
• We tried syncing the app

Commands tried

argocd app terminate-op APPNAME
argocd app sync APPNAME
argocd app sync APPNAME --force --prune 

Some Logs (not sure if these are relevant) :

time="2022-01-07T16:19:36Z" level=debug msg="Failed to apply normalization: error in remove for path: '/spec/preserveUnknownFields': Unable to remove nonexistent key: preserveUnknownFields: missing value"
time="2022-01-07T16:19:36Z" level=debug msg="Failed to apply normalization: error in remove for path: '/status': Unable to remove nonexistent key: status: missing value"
time="2022-01-07T16:19:36Z" level=debug msg="Failed to apply normalization: error in remove for path: '/spec/preserveUnknownFields': Unable to remove nonexistent key: preserveUnknownFields: missing value"
time="2022-01-07T16:19:36Z" level=debug msg="Failed to apply normalization: error in remove for path: '/status': Unable to remove nonexistent key: status: missing value"
time="2022-01-07T16:19:36Z" level=debug msg="Failed to apply normalization: error in remove for path: '/spec/preserveUnknownFields': Unable to remove nonexistent key: preserveUnknownFields: missing value"
time="2022-01-07T16:19:36Z" level=debug msg="Failed to apply normalization: error in remove for path: '/spec/preserveUnknownFields': Unable to remove nonexistent key: preserveUnknownFields: missing value"
time="2022-01-07T16:19:36Z" level=debug msg="Failed to apply normalization: error in remove for path: '/status': Unable to remove nonexistent key: status: missing value"
time="2022-01-07T16:19:36Z" level=debug msg="Failed to apply normalization: error in remove for path: '/spec/preserveUnknownFields': Unable to remove nonexistent key: preserveUnknownFields: missing value"
time="2022-01-07T16:19:36Z" level=debug msg="patch: {\"status\":{\"reconciledAt\":\"2022-01-07T16:19:36Z\"}}" application=argocd
time="2022-01-07T16:19:36Z" level=info msg="Failed to Update application operation state: etcdserver: request is too large, retrying in 1s"time="2022-01-07T16:19:36Z" level=info msg="Update successful" application=argocd
time="2022-01-07T16:19:36Z" level=info msg="Reconciliation completed" application=argocd dedup_ms=0 dest-name= dest-namespace=argocd dest-server="https://417901E660A9365B4057207C70C682EE.gr7.ap-south-1.eks.amazonaws.com" diff_ms=174 fields.level=2 git_ms=13 health_ms=3 live_ms=1 settings_ms=0 sync_ms=0 time_ms=249
time="2022-01-07T16:19:36Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=argocd

[utkarsh-devops]

Some more errors:
time="2022-01-07T18:08:00Z" level=warning msg="finished unary call with code FailedPrecondition" error="rpc error: code = FailedPrecondition desc = another operation is already in progress" grpc.code=FailedPrecondition grpc.method=Sync grpc.service=application.ApplicationService grpc.start_time="2022-01-07T18:08:00Z" grpc.time_ms=484.012 span.kind=server system=grpc

[vimalvillavan]

Facing the same issue. Any solution for the same?

[cbrdy]

I'm also facing the same problem. I'm on 2.4.11 and still see it.

[rahul-mourya]

I am also facing this issue with v2.4.12, although restarting the application controller statefulset seems to resolve it and could also be recreated consistently triggering the invalidation cluster cache sequence on application controller replica. Restarting/Deleting one of the two deployed argocd-server pod, would trigger the cluster cache invalidation and reinitialization in application controller replica and one of the replica out of three(we have 3 instance of application controller statefulset) would show the hang/refresh stuck issue.

More details on #10842

[yoadduani]

we facing this issue with v2.1.2
we tried restart all the pods - nothing help
also, we tried to upgrade argocd to 2.6.7 - didn't help.
any solution ?

[joaogl]

I've been having the same issue with argo v2.7.8.
Tried scaling up and down all the argo components it's just stuck in Terminating phase for ever... I have workloads that have been in this state for 20 days.
Only solution i found was to delete all the resources and application using kubectl, and then re-creating it again 🤦

[kraftaa]

The same. I had such an issue, it was stuck for several hours in sync. Initially I had one syntax error in manifest, but after I pushed updates, I had the correct values in desired manifest, but it was syncing forever. So I also deleted statefulsets in kubernetes, copied 'desired manifest' into yaml file and applied it. Then It was complaining about few missing keys, I took a template from helm and created them too via kubectl apply -f ... . After that I was able to run Sync, it went through and properly recreated/configured all the stuff.

[crenshaw-dev]

I believe the original poster was facing this issue: #14486

Basically, the number of resources to sync blew up the size of the Application's operation.sync.resources field, and etcd rejected it.

Argo CD needs to handle this more gracefully. I think gzipping some fields in the App status would help.

[tibers]

upgoating this because I came here googling for the error and I have the exact same problem. Is there a solution?

[crenshaw-dev]

Not yet. Upgrading to 2.8 should at least expose the actual error messages related to the problem. But actually solving the problem will require feature work (i.e. compression in the status field).

[tooptoop4]

🐐

[adecchi-2inno]

Facing the same issue. Any solution or workaround ? I am using argocd version v2.7.8+92949f6.dirty

[gravops]

Hello @crenshaw-dev ,
We are facing similar issue on multiple EKS clusters where argo CD getting stuck while refreshing status of an App. We have enabled Auto sync for the apps and it gets stuck on some apps. We are using Argo CD 2.5.1 and it goes away if we do rolling restart of Argo CD pods.
We tried using command "argocd app terminate-op APP-NAME" but it doesn't help. Getting logs which says refresh is getting timed out. Is there any fix or workaround for this found till now?

[gravops]

In logs we found that it times out after 3 mins, then again it retries to refresh the app status and again fails. This keeps on happening in loop infinitely.
time="2023-09-15T08:04:19Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=argocd/XXXX
time="2023-09-15T08:04:04Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=argocd/XXXX
time="2023-09-15T08:03:19Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=argocd/XXXX
time="2023-09-15T08:03:04Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=argocd/XXXX
time="2023-09-15T08:04:34Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=argocd/XXXX
time="2023-09-15T08:05:05Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=argocd/XXXX
time="2023-09-15T08:05:20Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=argocd/XXXX
time="2023-09-15T08:05:35Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=argocd/XXXX
time="2023-09-15T08:06:05Z" level=info msg="Refreshing app status (comparison expired, requesting refresh. reconciledAt: 2023-09-15 08:03:04 +0000 UTC, expiry: 3m0s), level (2)" application=argocd/XXXX

[adecchi-2inno]

Same problem running
v2.8.2+dbdfc71

[delfilip]

Hi, I found out the solution (version 2.2.2):
From UI, delete the Application with the "Non cascading" option. It will delete only the Application manifest, no its resources.
Then, sync App of Apps (if you have one, or deploy the Application manifest again).
Run sync.
Should be fine right now

[ndlanier]

This resolved the issue for me.

[jauyzed]

Same

[teddy-wahle]

Created an issue here with my experience of a similar problem:
#17155