Enhance documentation for the Gardenlet's /healthz endpoint

[danielfoehrKn]

How to categorize this PR?

/kind bug
/priority normal
/area documentation

What this PR does / why we need it:
Based on #2925 but uses direct clients instead of listers.

The health manager changes the gardenlets /healthz endpoint to DOWN if it cannot renew the lease in the Garden cluster.
However, this does not cover the case when the Gardener Extended API Server is down.

The problem is, that Seeds can be queried from the lister even though the Gardener Extended API Server is down. This can be prevented by using a direct client.

Advantage:

• Gardenlet fails when Gardener Extended API Server is down. I think this is the correct behaviour, as the Gardenlet cannot work without it.

Disadvantage:

• more network traffic (one request per 10 s)
• I propose to decrease the ping interval from 10s -> 20s to save network traffic.

UPDATE: THis PR now only includes enhanced documentation for the Gardenlet's /healthz endpoint.

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:

Release note:

Enhance documentation for Gardenlet's /healthz endpoint.

[timuthy]

However, this does not cover the case when the Gardener Extended API Server is down.

I'm generally not objecting but curious if you experienced any particular problems leaving the Gardenlet up and running while the API server was down? So far I was quite confident that a component like Gardenlet can deal with such intermittent issues.

[rfranzke]

I'm still struggling a bit with this with respect to the advantages we get compared with the network I/O?

[timebertt]

If network I/O is a big concern, we might wait for #3109 and use the metadata only client. This should be enough for this scenario, as we don't need the spec and saves network traffic.

[rfranzke]

Yeah, maybe. I'm still in the phase of understanding the motivation for this change and if we experienced any particular problem with it (or if we foresee problems if we don't introduce this now).

[timebertt]

Alternatively, we could also keep the watch but use some event handlers for delete events instead of using the lister.
Though, this would make the logic event-driven instead of level-driven, which might lead to issues, when we miss events.

[danielfoehrKn]

I'm generally not objecting but curious if you experienced any particular problems leaving the Gardenlet up and running while the API server was down?

No, I did not.
My main motivation was that I "expected" it to fail because I thought that the /healthzendpoint should always indicate whether the Gardenlet is healthy / can work.

Thinking about it again, we might be better off just documenting the behaviour - the /healthzendpoint only states that Gardenlet has a lease with the Garden cluster.

[rfranzke]

Yeah, @danielfoehrKn, maybe that's a good first step to clarify on what can be expected.

[timebertt]

I guess, we should generally improve our health checks and do a clean separation between liveness and readiness (ref gardener-attic/gardener-resource-manager#102 (comment)).
Then we can report an unready state, when caches haven't been synced for a minute or so. This would also cover the stale lease/seed case addressed here.

[danielfoehrKn]

I guess, we should generally improve our health checks and do a clean separation between liveness and readiness (ref gardener-attic/gardener-resource-manager#102 (comment)).
Then we can report an unready state, when caches haven't been synced for a minute or so. This would also cover the stale lease/seed case addressed here.

Maybe you can open an issue for this describing this in more detail?
For this PR, I would only include the documentation - if that is fine with you.

[rfranzke]

/lgtm