kapp should support CiliumIdentity resources out of the box

[praveenrewar]

What steps did you take:
I am using [Cilium](https://cilium.io] and I deployed a simple app with a Deployment and a Service.

$ kapp deploy -a simple-app -f https://github.com/carvel-dev/kapp/blob/develop/examples/simple-app-example/config-1.yml
Target cluster 'https://xx.xxx.xx.xxx' (nodes: gk3-xxx-xxx-default-pool, 2+)

Changes

Namespace  Name        Kind        Age  Op      Op st.  Wait to    Rs  Ri
default    simple-app  Deployment  -    create  -       reconcile  -   -
^          simple-app  Service     -    create  -       reconcile  -   -

Op:      2 create, 0 delete, 0 update, 0 noop, 0 exists
Wait to: 2 reconcile, 0 delete, 0 noop

Continue? [yN]: y

11:07:03AM: ---- applying 2 changes [0/2 done] ----
Warning: Autopilot set default resource requests for Deployment default/simple-app, as resource requests were not specified. See http://g.co/gke/autopilot-defaults
11:07:04AM: create service/simple-app (v1) namespace: default
11:07:06AM: create deployment/simple-app (apps/v1) namespace: default

...snip...

11:09:13AM: ---- applying complete [2/2 done] ----
11:09:13AM: ---- waiting complete [2/2 done] ----

Succeeded

What happened:
When I delete the app, kapp gets stuck on waiting for the CiliumIdentity resource to get deleted by the cluster.

$ kapp delete -a simple-app
Target cluster 'https://xx.xxx.xx.xxx' (nodes: gk3-xxx-xxx-default-pool, 2+)

Changes

Namespace  Name                         Kind            Age  Op      Op st.  Wait to  Rs  Ri
(cluster)  22690                        CiliumIdentity  3m   -       -       delete   ok  -
default    simple-app                   Deployment      5m   delete  -       delete   ok  -
^          simple-app                   Endpoints       5m   -       -       delete   ok  -
^          simple-app                   Service         5m   delete  -       delete   ok  -
^          simple-app-64dccdbdf5        ReplicaSet      5m   -       -       delete   ok  -
^          simple-app-64dccdbdf5-smkjb  CiliumEndpoint  3m   -       -       delete   ok  -
^          simple-app-64dccdbdf5-smkjb  Pod             5m   -       -       delete   ok  -
^          simple-app-64dccdbdf5-smkjb  PodMetrics      2s   -       -       delete   ok  -
^          simple-app-7mdbq             EndpointSlice   5m   -       -       delete   ok  -

Op:      0 create, 2 delete, 0 update, 7 noop, 0 exists
Wait to: 0 reconcile, 9 delete, 0 noop

Continue? [yN]: y

11:12:13AM: ---- applying 9 changes [0/9 done] ----
11:12:13AM: noop ciliumendpoint/simple-app-64dccdbdf5-smkjb (cilium.io/v2) namespace: default
11:12:13AM: noop pod/simple-app-64dccdbdf5-smkjb (v1) namespace: default
11:12:13AM: noop replicaset/simple-app-64dccdbdf5 (apps/v1) namespace: default
11:12:13AM: noop endpoints/simple-app (v1) namespace: default
11:12:13AM: noop endpointslice/simple-app-7mdbq (discovery.k8s.io/v1) namespace: default
11:12:13AM: noop podmetrics/simple-app-64dccdbdf5-smkjb (metrics.k8s.io/v1beta1) namespace: default
11:12:13AM: noop ciliumidentity/22690 (cilium.io/v2) cluster
11:12:13AM: delete deployment/simple-app (apps/v1) namespace: default
11:12:13AM: delete service/simple-app (v1) namespace: default

...snip...

11:12:17AM: ---- waiting on 1 changes [8/9 done] ----
11:13:15AM: ongoing: delete ciliumidentity/22690 (cilium.io/v2) cluster
11:13:18AM: ---- waiting on 1 changes [8/9 done] ----
11:14:16AM: ongoing: delete ciliumidentity/22690 (cilium.io/v2) cluster
11:14:19AM: ---- waiting on 1 changes [8/9 done] ----
^C

What did you expect:
I expected kapp delete to succeed.

Anything else you would like to add:
When we create a an app using kapp which has a pod resource (like Deployment), then the ownership labels are injected into the Pods and the ReplicaSets as well, and into the CiliumIdentity resource in case of GKE autopilot cluster. While deleting the app, since CiliumIdentity is a cluster owned resource, kapp waits for the cluster to delete it, but these resources are not deleted immediately so kapp gets stuck waiting for the cluster to delete them.

Slack thread: https://kubernetes.slack.com/archives/CH8KCCKA5/p1683315623187379

Environment:

• kapp version (use kapp --version):
• OS (e.g. from /etc/os-release):
• Kubernetes version (use kubectl version)

---

Vote on this request

This is an invitation to the community to vote on issues, to help us prioritize our backlog. Use the "smiley face" up to the right of this comment to vote.

👍 "I would like to see this addressed as soon as possible"
👎 "There are other more important things to focus on right now"

We are also happy to receive and review Pull Requests if you want to help working on this issue.

[praveenrewar]

kapp v0.56.0 contains a fix for this issue which excludes the CiliumIdentity resources during listing. Next steps would be to provide this functionally via the kapp configuration and also provide a a way to disable it. Keeping this issue open to track that work.