Flaky test Test/ServerSideXDS_FileWatcherCerts

[easwars]

See: https://github.com/grpc/grpc-go/pull/4388/checks?check_run_id=2531319580

[easwars]

The test actually passed.

This goroutine is leaked:

        grpctest.go:39: Leaked goroutine: goroutine 402 [select]:
            github.com/envoyproxy/go-control-plane/pkg/server/sotw/v3.(*server).StreamHandler.func1(0x7f06edc78598, 0xc0002a5150, 0xc00011f020, 0xc00067fe00, 0xc000088bc0)
            	/home/runner/go/pkg/mod/github.com/envoyproxy/go-control-plane@v0.9.9-0.20210217033140-668b12f5399d/pkg/server/sotw/v3/server.go:409 +0xf6
            created by github.com/envoyproxy/go-control-plane/pkg/server/sotw/v3.(*server).StreamHandler
            	/home/runner/go/pkg/mod/github.com/envoyproxy/go-control-plane@v0.9.9-0.20210217033140-668b12f5399d/pkg/server/sotw/v3/server.go:399 +0xa5
        grpctest.go:60: Leak check disabled for future tests

This goroutine should exit when the management server Stop() is called, which is clearly done in this case because the management server's call to grpc.Serve() is not blocked. So, this is strange. If we see this again, maybe we should remove this goroutine also from our leakcheck.

[easwars]

This is another flake: https://github.com/grpc/grpc-go/pull/4406/checks?check_run_id=2531642404

The issue here is different though. What is happening here is that we have a log statement from one of xdsClients goroutines after the test goroutine has completed. This is because the xdsClient is not getting strictly cleaned up inline. This should get fixed once we fix the balancer_conn_wrapper.close()

[dfawley]

@easwars is the root cause here a dupe of #4403? If so we can close it

[easwars]

This has been fixed as part of envoyproxy/go-control-plane#429. We need to pin to the latest version of the go-control-plane from our go.mod.