Investigate cause of intermittent "connection refused" errors in TestDetailedConnectionCloseErrorPropagatesToRpcError

[apolcyn]

This is a spinoff from the review conversation in #4311 (comment)

If the rpcStartedOnServer channel is removed from that test, some of RPCs will fail with:

end2end_test.go:1372: &{0xc0000ba360}.Recv() = _, rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:37309: connect: connection refused", want _, rpc error containing substring: |connection reset by peer| OR |error reading from server: EOF|

Note that the stub server Start method waits until the client conn is ready before returning. So the only explanation I can think of is that the LB policy is picking a connection for the RPC asynchronously (i.e. ss.Client.FullDuplexCall(ctx) does not wait for the LB policy to pick a connection - FWIW this behavior differs e.g. from C++), and that the LB policy's picking of a connection is sometimes losing the race against ss.S.Stop() - in that case, the previously ready connection would tear down and not come back up, and RPC's would fail with such an error.

Filing this bug to confirm the root cause though.

[menghanl]

@apolcyn Have you got time to look at this?

[apolcyn]

I'll likely be busy with other issues for at least the next couple of weeks.

[github-actions]

This issue is labeled as requiring an update from the reporter, and no update has been received after 6 days. If no update is provided in the next 7 days, this issue will be automatically closed.