pubsub: Emit outstanding messages metrics for publishers

[schallert]

Is your feature request related to a problem? Please describe.

The PubSub package emits two metrics related to outstanding messages: OutstandingMessages and OutstandingBytes. The docs mention

OutstandingMessages is a measure of the number of outstanding messages held by the client before they are processed.

As a user, "messages held by the client" is something I care about in two contexts:

1. In the context of publishing, I want to know how many messages the pubsub client has buffered that have not yet been published and persisted by the PubSub service.
2. In the context of subscribing, I want to know how many messages the pubsub client has buffered that my application has not yet processed.

From the docs, I would have expected OutstandingMessages to emit metrics for both of those contexts. However, in practice, it seems like the client only increments OutstandingMessages in the context of (2) (code link). At first this was surprising, as it seems like the flow controller (which is used for both publishing and subscribing) is responsible for recording this metric. However, I saw that the metric is only registered with the subscription tag, which makes sense as to why it doesn't record for publishing to a topic.

This is frustrating, because I cannot tell from the pubsub client's internal metrics how many messages have been buffered but not yet persisted to PubSub. This is helpful for debugging cases where the PubSub service may be experiencing latency on Publish RPCs, I am experiencing network problems, etc.

Describe the solution you'd like

I would like the pubsub package to emit metrics on how many messages are outstanding in the context of publishing. I don't mind if this requires a separate View for producer outstanding metrics and subscriber outstanding metrics.

Describe alternatives you've considered

We have implemented our own metrics of how many messages have not yet been fully published by keeping track of PublishResults and decrementing counters when .Get() returns. It's not as clean as having this natively in the client.

[hongalex]

Hi, yeah as you surmised, outstanding messages/bytes only applies on the subscribe side. This is probably confusing because historically we did not have publisher-side flow control, so there was no concept of MaxOutstandingMessages/Bytes for the publisher side.

I'll consider this a feature request to add something like PublisherOutstandingMessages and PublisherOutstandingBytes to keep track of this.

[schallert]

I'll consider this a feature request to add something like PublisherOutstandingMessages and PublisherOutstandingBytes to keep track of this.

Gave this a shot.