Cleanup prometheus metrics more predictably

[aschmahmann]

At the moment we have a global variable for TCP metrics

go-tcp-transport/metrics.go

Line 23 in 1b96803

var collector *aggregatingCollector

Which has a map of connections

go-tcp-transport/metrics.go

Line 58 in 1b96803

conns map[uint64] /* id */ *tracingConn

That we only clear out when the metrics are collected

go-tcp-transport/metrics.go

Line 106 in 1b96803

func (c *aggregatingCollector) Collect(metrics chan<- prometheus.Metric) {

We should clear out these connections more predictably (e.g. on connection close or some background goroutine).

---

There might also be a bug related to the cleanup itself

go-tcp-transport/metrics.go

Lines 110 to 120 in 1b96803

	var bytesSent, bytesRcvd uint64
	for _, conn := range c.conns {
	info, err := conn.getTCPInfo()
	if err != nil {
	if strings.Contains(err.Error(), "use of closed network connection") {
	c.closedConn(conn)
	continue
	}
	log.Errorf("Failed to get TCP info: %s", err)
	continue
	}

where if we're unable to get the TCP info we never clean it up. e.g. on Windows I see the following error:

2021-07-08T21:39:47.925-0400    ERROR   tcp-tpt go-tcp-transport@v0.2.2/metrics.go:118  Failed to get TCP info: raw-control tcp 192.168.1.6:4001: getsockopt: not implemented