-
Notifications
You must be signed in to change notification settings - Fork 90
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ci(ssh): avoid ssh timeouts from the server #5370
Conversation
This reverts commit b366d6e.
Motivation: We've been trying multiple solutions to our SSH connection issues, our last try solving this issues was PR https://github.com/ZcashFoundation/zebra/pull/5367/files Depends-On: #5367 Expected behavior: An SSH connection should not be terminated by the server, the connection must be kept alive indefinitely until it's killed by GitHub Actions Solution: Disable TCP keepalive messages from the server and set `ClientAliveCountMax` to 0, which disables connection termination
@@ -11,6 +14,8 @@ sudo cat /etc/ssh/sshd_config | |||
echo | |||
echo 'Modifying config:' | |||
echo 'MaxStartups 500' | sudo tee --append /etc/ssh/sshd_config \ | |||
echo 'TCPKeepAlive no' | sudo tee --append /etc/ssh/sshd_config \ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Disabling TCP keepalive packets will make some firewalls terminate the connection as inactive.
Is there some documentation supporting this change?
@@ -11,6 +14,8 @@ sudo cat /etc/ssh/sshd_config | |||
echo | |||
echo 'Modifying config:' | |||
echo 'MaxStartups 500' | sudo tee --append /etc/ssh/sshd_config \ | |||
echo 'TCPKeepAlive no' | sudo tee --append /etc/ssh/sshd_config \ | |||
echo 'ClientAliveCountMax 0' | sudo tee --append /etc/ssh/sshd_config \ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We might want to set this to a high number instead (6 hours), to avoid stuck connections exhausting the connection limit.
Or we can also increase the authenticated connection limit so it's higher than the number of connections we expect from the longest workflow. (About 20.)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just wondering if there is a specific log message or failure you're trying to fix here?
Edit: yes, this is a known issue.
I'm also going to ask the new CI change questions from the retro:
|
This is actually happening in CI:
|
We're no longer needing this. We might consider to open this once again if we see SSH issues reappear |
Motivation
We've been trying multiple solutions to our SSH connection issues, our last try solving this issues was PR https://github.com/ZcashFoundation/zebra/pull/5367/files
Depends-On: #5367
Expected behavior
An SSH connection should not be terminated by the server, the connection must be kept alive indefinitely until it's killed by GitHub Actions
Solution
Disable TCP keepalive messages from the server and set
ClientAliveCountMax
to 0, which disables connection terminationReview
Anyone can review this
Reviewer Checklist