Add MaximumOverIntervalGauge #469
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Jan Berktold <jberktold@roblox.com>
Signed-off-by: Jan Berktold <jberktold@roblox.com>
JanBerktold
commented
Nov 14, 2022
| // The length of a given interval. | ||
| interval_duration: Duration, | ||
| // The time at which the current interval will expose. | ||
| interval_expiry: Arc<RwLock<Instant>>, |
There was a problem hiding this comment.
We actually use an ArcSwap for this in our internal codebase. I've switched this over to an RwLock here to avoid taking on the ArcSwap crate dependency.
lucab
reviewed
Nov 15, 2022
| #[derive(Clone, Debug)] | ||
| pub struct MaximumOverIntervalGauge { | ||
| // The current real-time value. | ||
| value: Arc<AtomicI64>, |
There was a problem hiding this comment.
Here and everywhere else, gauges typically operate on f64 values.
There was a problem hiding this comment.
This was purely an i64 because we weren't clear on how to implement an atomic f64 (nor did we care enough). I just discovered your awesome https://github.com/tikv/rust-prometheus/blob/master/src/atomic64.rs#L89 setup -- will switch this over!
|
Interesting, I'll need to read a bit more about the underlying logic. Does this have an equivalent in Go world (or any other client library)? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Depends on #405.
This is another "fancy" gauge type from other internal codebase that I believe to be beneficial to others.
We have a few use cases where we want to monitor a value that moves around a lot. One example is measuring the number of concurrent requests for a gRPC server.
With a normal gauge, we're only able to inspect the value as of the time of the scraping (typically once every 10 seconds). This actually got us in production -- We had queue build ups due to downstream TCP re-connects that were manifesting themselves as increased latency for our customers but our metrics looked fine.
Enter the
MaximumOverIntervalGauge: Instead of reporting the current value, we report the maximum of the value over the last (user configured) interval. This allows us to observe shifts that resolve themselves in less than the configured scrape interval.