KubernetesClient informer stopped silently after running couple weeks

[hkbuddy]

Describe the bug

After running KubernetesClient informers for couple weeks, it just stopped working silently until the pod being restarted. No logs were found related the this issue. Or let me know how can I debug this issue. Thanks.

Fabric8 Kubernetes Client version

5.12.3

Steps to reproduce

run KubernetesClient informers for couple weeks

Expected behavior

informers should continue to process events.

Runtime

Kubernetes (vanilla)

Kubernetes API Server version

1.22

Environment

Linux

Fabric8 Kubernetes Client Logs

No response

Additional context

No response

[shawkins]

We have informers running for weeks at a time as well and have not experienced any issue. You can check the informer isWatching method as part of a health check and turn on debug logging if you didn't see anything at info / warn. If the informer thinks that it's watching, but you are not getting events, then you are hitting an issue similar to kubernetes/kubernetes#102464 - but different as that one was based on the api server mishandling the timeout for service watches. If that's the case we'd need to know the type of resource and also file an upstream issue.

[hkbuddy]

ok thanks @shawkins! Let us add isWatching method as part of a health check and see how it goes.

[andreaTP]

You can check the informer isWatching method as part of a health check

I would say that it should be part of the "Ready" check, but, apart from this detail, I think this is something that can be provided by the SDK itself cc. @csviri @metacosm

[shawkins]

Could be related to #4369 which is addressed by #4365

[stale]

This issue has been automatically marked as stale because it has not had any activity since 90 days. It will be closed if no further activity occurs within 7 days. Thank you for your contributions!