Fixes: #2614
This is a problem with pion/webrtc's OnConnectionStateChange handler. It should wait for the handler to finish. Right now connection notifications might get reordered leading to listener thinking the connection has established and the dialer thinks connection hasn't established yet.

The fix in pion/webrtc is here:
https://github.com/pion/webrtc/pull/2702/files

The same error in pion/ice
https://github.com/pion/ice/pull/656/files

Update: the text below helped me explains how I went about debugging this
I think this fixes the problem This might help debug the problem of tests taking 10m. My theory is that the ICE connection succeeds on dialer and times out on listener(20s). So the listener removes the ufrag from its udp mux. After keepalive time, the dialer sends a STUN Request and initiates a new connection on the listener. This of course will fail because the dialer has exited the ice connection establishment mode and is simply sending STUN packets to keep the connection alive.

If we look at the failure here: https://github.com/libp2p/go-libp2p/actions/runs/8244362080/job/22546464639

listener is stuck on
2024-03-12T06:24:47.2310488Z /home/runner/work/go-libp2p/go-libp2p/p2p/transport/webrtc/listener.go:234 +0x886

	select {
	case <-ctx.Done():
		return nil, ctx.Err()
	case err := <-errC:
		if err != nil {
			return nil, fmt.Errorf("peer connection failed for ufrag: %s", candidate.Ufrag)
		}

This context expires after 20 seconds.
And all the goroutines on the dialer side are stuck for 9 minutes. And all the goroutines on the listener side are stuck for < minute(which is why that's not shown in the logs)
The logs also show that the dialer believes that ice connection on its side has completed.

Unfortunately this is very difficult to test. This has not flaked in the 16 tries I have done after that.

We still have to explain why the dialer doesn't exit on DTLS failure. DTLS failure should reflect in the OnConnectionStateChange callback.