gRPC server metrics

[pglass]

Description

This updates the internal and external gRPC servers with the following labeled metrics, where the server_type label is either internal or external.

Server metrics:

• grpc.server.connection.count{server_type=internal|external}
• grpc.server.connections{server_type=internal|external}
• grpc.server.request.count{server_type=internal|external}
• grpc.server.stream.count{server_type=internal|external}
• grpc.server.streams{server_type=internal|external}

Client metrics (a client only exists for the internal gRPC server)

• grpc.client.connection.count{server_type=internal}
• grpc.client.connections{server_type=internal}
• grpc.client.request.count{server_type=internal}

These are the same metrics that currently exist for the internal gRPC server, except those are now emitted with a server_type = internal label.

The stats handler is moved to the agent/grpc-middleware package, and some shared unit test stuff is moved to the agent/grpc-middleware/testutil package.

Testing & Reproduction steps

• Unit tests

PR Checklist

• updated test coverage
• external facing docs updated
• not a security concern

[pglass]

This copies the TestHandler_EmitStats test. I did some deduplication of the helpers by moving things over to grpc-middleware/testutil. (I could do more deduplication but kind of wanted a sanity check before I go too far.)

[boxofrad]

Nice work! 🚢

[boxofrad]

What do you think of passing the metrics object as a parameter to NewServer so we don't have to change global state like this?

func NewServer(logger agentmiddleware.Logger, metrics *metrics.Metrics) *grpc.Server {
	if metrics == nil {
		metrics = agentmiddleware.DefaultMetrics
	}

[boxofrad]

I also appreciate that these tests were copied from grpc-internal so please feel free to leave as-is! 🙇🏻‍♂️

[pglass]

💯 I removed the global metrics instance from both grpc-internal and grpc-external