Kubectl: Keepalive connection to API server for exec and logs.

[nielsbasjes]

What would you like to be added:

A feature (enabled by default) that any "long running" kubectl command (like exec and logs) send a periodic keep alive signal to the API server.

At this point this is only available for the proxy command where it is disabled by default.

I propose to have this keepalive made a part of also the exec and logs (and others?) and to set the default value to 5s.

Why is this needed:

When running a kubernetes cluster in High Available mode (i.e. multiple api servers behind a loadbalancer) it is common that client side inactivity timeouts are set in the load balancer so it is able to cleanup stale connections.

The advised haproxy config (which I have right now) advises to set the timeouts for client and server to 20s.
https://github.com/kubernetes/kubeadm/blob/master/docs/ha-considerations.md#haproxy-configuration

What I ran into is that I listen to the logs of a pod (i.e. kubectl logs -f) and the process at hand took a "long time" between the log messages (I'm debugging a new application). After the configured timeout (the mentioned 20 seconds) the connection would be lost and I had to restart the 'logs -f' to see the rest of the messages.

Also exec a shell in a pod to examine what is happening, looking something up on a webpage and then returning to the shell to find it has been closed because I was idle for more than 20 seconds is not very productive.

See also: #58486

[nielsbasjes]

/sig api-machinery
/sig cli

[nielsbasjes]

Note that at this point the only available mitigation for these practical problems is to increase the timeouts on the loadbalancer (haproxy in my case).
Yet this not desirable because

1. also truly stale connections will remain in the memory of the loadbalancer for a longer time which may overload things in busy/large installations.
2. the connections will still be closed after the longer timeout even if this is not wanted.

[fedebongio]

/assign @lavalamp
(who volunteered to find the previous one related)

[lavalamp]

I think #94170 was the one I was thinking of. It adds the feature, but we would need to turn it on in various places. Probably both the kubectl <-> apiserver and the apiserver <-> kublet paths would need this on.

[lavalamp]

I don't think there's much harm in just turning it on, I don't really think we need a flag. A ping every 20s or so shouldn't break the bank.

[caesarxuchao]

#94170 might fix this.

edit: lavalamp beat me

[lavalamp]

Yeah, #94170 doesn't actually enable it anywhere, it just adds the parameter.

…

On Tue, Sep 1, 2020 at 2:00 PM Chao Xu ***@***.***> wrote: #94170 <#94170> might fix this. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#94301 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAE6BFSSXPZMPHXUQS5BP2LSDVOGJANCNFSM4QN3GKJQ> .

[nielsbasjes]

Given that the recommended timeout is 20 seconds I would set the default ping time to something like 5 or 10 seconds.

[djzager]

/assign

[Nit123]

/assign

[vinayvenkat]

/assign

[joshfix]

+1 This would be super useful.

[bevank]

+1

[knight42]

Hi, I have filed #97083 to enable SPDY pings to address this issue.

[nielsbasjes]

As I understand #97083 addresses the exec issue but not the logs -f issue.

[knight42]

As I understand #97083 addresses the exec issue but not the logs -f issue.

More precisely, #97083 should address exec and portforward.

As for the logs -f, I think this is a different problem because logs -f actually sends a HTTP request to the REST API, and the connection might be terminated if the kubectl client, or the apiserver, or the load balancer between the client and server thought the connection has been idle for a specific period.

IMHO I guess we need to reconnect until user interrupts to address the logs -f issue.

[tiloso]

Hey, for logs -f the issue has been addressed in #95981 as far as I understand. (At least for environments that use HTTP/2.)

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten

[nielsbasjes]

As far as I understand right now:

• feat: enable SPDY pings on connections #97083 should address exec and portforward issues.
• Enables HTTP/2 health check #95981 should address logs -f in HTTP/2 (SPDY) situations.

@djzager / @vinayvenkat / @Nit123 Does that mean this issue has been fully resolved (i.e. can be closed)?
Or is there something remaining?

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Reifier]

/reopen

[k8s-ci-robot]

@Reifier: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.