Expose service 2 service call stats

[fantayeneh]

• Feature request

Expose stats scoped by the caller service, so linkerd can be used as the source of truth for all service to service call metrics.
Considering this can explode easily, it is probably better so expose only a limited numbers of stats.

For each route i would suggest to expose the same metrics exposed by https://github.com/twitter/finagle/blob/master/finagle-http/src/main/scala/com/twitter/finagle/http/filter/StatsFilter.scala

• status/<statusCode>
  A counter of the number of responses received, or returned for servers, that had this statusCode.
• status/<statusClass>
  Same as status/statusCode but aggregated per category, e.g. all 500 range responses count as 5XX for this counter.
• time/<statusCode>
  A histogram on duration in milliseconds per HTTP status code.
• time/<statusCategory>
  A histogram on duration in milliseconds per HTTP status code category.

[adleong]

@fantayeneh can you go into more detail? How do we identify the source of a request? How do these sources get rendered into metric names?

How would this be configured? It should probably be an opt-in feature that should only be enabled if the number of sources is known to be small. Can this be done as a Linkerd plugin?

[fantayeneh]

So
Q. How do we identify the source of a request?
A. This should be configurable/pluggable solution
- Injecting this info in a configurable header name, can be on solution
- If you are using tracing, you might have already the desired info

Q. How do these sources get rendered into metric names?
A. route.src.dest.metric e.g route.frontend.backend.status

Q. How would this be configured? It should probably be an opt-in feature that should only be enabled if the number of sources is known to be small.
A. Absolutely this should be an opt-in feature.
Not sure how/where should go the config but i see see some options for the config

routers:
- protocol: http
  route_stats:
    # max_size: maximum number route stats allowed
    max_size: 10

Q. Can this be done as a Linkerd plugin?
A. I am not sure if it is possible to have hold the stats object in a Plugin

You were saying that there is some work being done for Linkerd2.
Is this different from what you folks thinking?