xds: concurrent map read and map write

[02santoshphegde]

What version of gRPC are you using?

1.45.0

What version of Go are you using (go version)?

go1.17

What operating system (Linux, Windows, …) and version?

Linux (Ubuntu)

What did you do?

Using xds as client side loadbalancer for GCP traffic director service.

_ "google.golang.org/grpc/xds"

What did you expect to see?

client work normal and was not expecting to crash.

What did you see instead?

concurrent map read and map write

goroutine 4099 [running]:
runtime.throw({0x18e3978, 0x545780})
	/usr/local/go/src/runtime/panic.go:1198 +0x71 fp=0xc0023f1250 sp=0xc0023f1220 pc=0x43b571
runtime.mapaccess2_faststr(0x1872220, 0xc001b7e8f0, {0xc0024d3b40, 0x37})
	/usr/local/go/src/runtime/map_faststr.go:116 +0x3d4 fp=0xc0023f12b8 sp=0xc0023f1250 pc=0x4172f4
google.golang.org/grpc/xds/internal/xdsclient.(*clientImpl).newAuthority(0xc0000a1c80, 0xc001be0230)
google.golang.org/grpc/xds/internal/xdsclient.(*clientImpl).ReportLoad(0xc0000a1c80, 0xc001be0230)
	/home/build/go/pkg/mod/google.golang.org/grpc@v1.45.0/xds/internal/xdsclient/loadreport.go:31 +0x2a fp=0xc0023f13d8 sp=0xc0023f1360 pc=0x115aa0a
google.golang.org/grpc/xds/internal/xdsclient.(*clientRefCounted).ReportLoad(0x2a1bef0, 0xc000c77dd0)
	<autogenerated>:1 +0x2a fp=0xc0023f13f8 sp=0xc0023f13d8 pc=0x115c1ea
google.golang.org/grpc/xds/internal/xdsclient.(*onceClosingClient).ReportLoad(0xc001860900, 0xc001be0230)
	<autogenerated>:1 +0x2f fp=0xc0023f1418 sp=0xc0023f13f8 pc=0x115c64f
google.golang.org/grpc/xds/internal/balancer/clusterimpl.(*clusterImplBalancer).updateLoadStore(0xc00090b200, 0xc0025e2540)
	/home/build/go/pkg/mod/google.golang.org/grpc@v1.45.0/xds/internal/balancer/clusterimpl/clusterimpl.go:210 +0x2b0 fp=0xc0023f1470 sp=0xc0023f1418 pc=0x1321e70
google.golang.org/grpc/xds/internal/balancer/clusterimpl.(*clusterImplBalancer).UpdateClientConnState(0xc00090b200, {{{0xc0019597a0, 0x2, 0x2}, 0x0, 0xc000010838}, {0x1b40ae0, 0xc0025e2540}})
	/home/build/go/pkg/mod/google.golang.org/grpc@v1.45.0/xds/internal/balancer/clusterimpl/clusterimpl.go:250 +0x2cb fp=0xc0023f1788 sp=0xc0023f1470 pc=0x13221ab
google.golang.org/grpc/internal/balancergroup.(*subBalancerWrapper).updateClientConnState(...)
	/home/build/go/pkg/mod/google.golang.org/grpc@v1.45.0/internal/balancergroup/balancergroup.go:144
google.golang.org/grpc/internal/balancergroup.(*BalancerGroup).UpdateClientConnState(0xc0009d4750, {0xc000137770, 0xbc0897}, {{{0xc0019597a0, 0x2, 0x2}, 0x0, 0xc000010838}, {0x1b40ae0, 0xc0025e2540}})

We are communicating with two different xds host here, will that cause any issue here?

[menghanl]

Thanks for the report. This seems like a bug. The caller should hold a mutex before calling the method.
I made a fix in #5331. Please give it a try. Thanks