Flaky Test: Test/ServerSideXDS_FileWatcherCerts

[timmyyuan]

What version of gRPC are you using?

master

What version of Go are you using (go version)?

go1.16.3

What operating system (Linux, Windows, …) and version?

ubuntu 20.04

What did you do?

If possible, provide a recipe for reproducing the error.

git clone https://github.com/grpc/grpc-go grpc
pushd grpc
go mod edit -replace=google.golang.org/grpc=github.com/grpc/grpc-go@latest
go mod tidy
go mod vendor
go build -mod=vendor
make test

What did you see instead?

...
127.0.0.1:43797 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:43797: connect: connection refused". Reconnecting...  (t=+9.615940176s)
        tlogger.go:99: clientconn.go:1113 [core] Subchannel Connectivity change to TRANSIENT_FAILURE  (t=+9.615959276s)
        tlogger.go:99: clientconn.go:434 [core] Channel Connectivity change to TRANSIENT_FAILURE  (t=+9.615981576s)
        tlogger.go:99: clientconn.go:434 [core] Channel Connectivity change to SHUTDOWN  (t=+10.022709025s)
        tlogger.go:99: xds_resolver.go:91 [xds] [xds-resolver 0xc000578f00] Watch cancel on resource name my-service with xds-client 0x17e7f90  (t=+10.022749225s)
        tlogger.go:99: xds_resolver.go:266 [xds] [xds-resolver 0xc000578f00] Shutdown  (t=+10.022769825s)
        tlogger.go:99: server.go:150 [xds-e2e] Stopped the xDS management server...  (t=+10.022842826s)
        tlogger.go:99: clientconn.go:434 [core] Channel Connectivity change to SHUTDOWN  (t=+10.022909326s)
        tlogger.go:99: clientconn.go:1113 [core] Subchannel Connectivity change to SHUTDOWN  (t=+10.022932526s)
        tlogger.go:99: client.go:610 [xds] [xds-client 0xc00053ae00] Shutdown  (t=+10.022961426s)
        --- FAIL: Test/ServerSideXDS_FileWatcherCerts/mtls (10.00s)
            xds_server_integration_test.go:304: Using tmpdir: /tmp/testServerSideXDS134943499
            xds_server_integration_test.go:304: Wrote file at: /tmp/testServerSideXDS134943499/cert.pem
            xds_server_integration_test.go:304: Wrote file at: /tmp/testServerSideXDS134943499/key.pem
            xds_server_integration_test.go:304: Wrote file at: /tmp/testServerSideXDS134943499/ca.pem
            xds_server_integration_test.go:304: Using tmpdir: /tmp/testClientSideXDS476431598
            xds_server_integration_test.go:304: Wrote file at: /tmp/testClientSideXDS476431598/cert.pem
            xds_server_integration_test.go:304: Wrote file at: /tmp/testClientSideXDS476431598/key.pem
            xds_server_integration_test.go:304: Wrote file at: /tmp/testClientSideXDS476431598/ca.pem
            xds_server_integration_test.go:355: rpc EmptyCall() failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded
        tlogger.go:99: transport_helper.go:275 [xds] [xds-client 0xc00053ae00] ADS request for {target: ["grpc/server?xds.resource.listening_address=127.0.0.1:44301"], type: ListenerResource, version: "1", nonce: "1"} failed: xds: stream.Send(version_info:"1" node:{id:"973f6dea-f59d-4ffd-a0b9-7ca9da582b2d" user_agent_name:"gRPC Go" user_agent_version:"1.38.0-dev" client_features:"envoy.lb.does_not_support_overprovisioning"} resource_names:"grpc/server?xds.resource.listening_address=127.0.0.1:44301" type_url:"type.googleapis.com/envoy.config.listener.v3.Listener" response_nonce:"1") failed: EOF  (t=+10.024867838s)
        tlogger.go:99: listener_wrapper.go:141 [xds] [xds-server-listener 0xc00051e420] Watch cancelled on resource name grpc/server?xds.resource.listening_address=127.0.0.1:44301  (t=+10.024937339s)
        tlogger.go:99: transport_helper.go:205 [xds] [xds-client 0xc00053ae00] xds: ADS stream creation failed: rpc error: code = Canceled desc = grpc: the client connection is closing  (t=+10.024985639s)
FAIL
FAIL	google.golang.org/grpc/xds/internal/test	14.594s

[dfawley]

#4402 is also about this and has more info. Thanks for filing this.