ssh: fix error on commandconn close, add ping and default timeout

[laurazard]

- Related

• cli: set timeout connection ping on sockets as well #3722
• cli hangs on dead connection #3652
• Fix ssh process killed when context is done #3900
• docker.example.com is down, so docker compose doesn't work anymore #4221

- TLDR

During CLI initialization we ping the configured host. Initially, we set no context timeout for these connections, and after b397391 we set timeouts but only for TCP connections (which then got changed in #3722 to only specifically exclude SSH connections and not sockets).

This looks to be due to errory looking logs from cli/connhelper/commandconn/commandconn.go on Close() – which gets called by http.Transport when the request context times out.
However, these logs are being thrown erroneously – commandconn.kill() states that it will return nil if the command terminates regardless of it's exit status:

cli/cli/connhelper/commandconn/commandconn.go

Line 113 in 26a7357

// kill returns nil if the command terminated, regardless to the exit status.

But the implementation does not hold up to this contract:

cli/cli/connhelper/commandconn/commandconn.go

Line 130 in 26a7357

if wExitErr.ProcessState.Exited() {

since ProcessState.Exited() returns true only if the program exited due to calling exit and not if the program was terminated by a signal (this probably does works on Windows since here ProcessState.Exited() returns true if the program has exited either way, but not on POSIXy OSs).

If commandconn.kill() is fixed to always return nil when the command has terminated, setting timeouts on contexts passed to SSH connections works fine and we no longer need special handling.

Other than that, I also added -o ConnectTimeout=30 to the created SSH command, which serves to align with the timeout from the client returned for non-SSH connections in

cli/cli/context/docker/load.go

Line 134 in 26a7357

Timeout: 30 * time.Second,

since it will be used only for "dialing" i.e., ssh command is unable to establish a connection within 30s it will error out, but if it can establish a connection it will keep running indefinitely, so commands like docker stats or an attach won't time out after 30s.

During this I also found a racy call to cmd.Wait() between Close() and onEOF(), since calling Close will close the command's stdin/out pipes, which will throw an EOF to pending reads, triggering a call to onEOF(), but afterwards Close will also call kill() which will also call cmd.Wait()

- What I did

• cli/connhelper/commandconn/commandconn.go:
  • kill(): Fix implementation so we don't return an error when process is successfully closed
  • Write()/Read(): Prevent race by checking if connection is being closed
• cli/command/cli.go:
  • initializeFromClient(): remove SSH special handling so we also add timeout for SSH connections
• cli/context/docker/load.go:
  • ClientOpts(): add 30s timeout to align with non-SSH client

- How to verify it

• DOCKER_HOST=ssh://example.com docker: no longer hangs forever, now it returns after the default 2s CLI init timeout
• DOCKER_HOST=ssh://example.com docker version: no longer hangs forever, now it will timeout after 30s (can be more if the host resolves to more than 1 IP address)

- Description for the changelog

- A picture of a cute animal (not mandatory but encouraged)

[codecov-commenter]

Codecov Report

Merging #4226 (50d81c8) into master (20923df) will increase coverage by 0.22%.
The diff coverage is 84.50%.

❗ Current head 50d81c8 differs from pull request most recent head a5ebe22. Consider uploading reports for the commit a5ebe22 to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4226      +/-   ##
==========================================
+ Coverage   59.07%   59.29%   +0.22%     
==========================================
  Files         287      288       +1     
  Lines       24752    24769      +17     
==========================================
+ Hits        14623    14688      +65     
+ Misses       9249     9197      -52     
- Partials      880      884       +4     

[thaJeztah]

Thanks! Left some nits, suggestions, and questions 😅

[thaJeztah]

/cc @AkihiroSuda perhaps you want to have a look as well

[thaJeztah]

Looking good (thanks!); could use some additional set of eyes on this in case I missed things.

I left some comments/suggestions inline; let me open a PR with those suggestions (feel free to ignore/update/modify them if you want)

[thaJeztah]

opened laurazard#1 (please squash with your commits, and (as mentioned) feel free to update/ignore/whatever).

[laurazard]

@thaJeztah I merged your changes (thank you!)

[thaJeztah]

LGTM

[thaJeztah]

@rumpl @AkihiroSuda ptal 🤗

[AkihiroSuda]

Can't we use errors.Is(readErr, io.EOF)?

[tianon]

Similar to moby/moby#45966 (comment), this means we should update our go 1.18 line in vendor.mod to at least 1.19 😅

# github.com/docker/cli/cli/connhelper/commandconn
cli/connhelper/commandconn/commandconn.go:71:22: undefined: atomic.Bool
cli/connhelper/commandconn/commandconn.go:76:22: undefined: atomic.Bool
cli/connhelper/commandconn/commandconn.go:77:22: undefined: atomic.Bool
cli/connhelper/commandconn/commandconn.go:78:22: undefined: atomic.Bool

[thaJeztah]

Hello @tianon, are you interested in contributing to open source and opening a PR? 😉

[tianon]

Naw, open source isn't really my thing 😂

(totally fair callout, I'll see what I can do)