How can I extend OpenTelemetry Java with a new "digest" instrument?

[pedrolamarao]

In my understanding of the ticket below, "summaries" were removed from OpenTelemetry Metrics before the initial stable release over concerns about the summability of summary deltas.
open-telemetry/opentelemetry-proto#199

T-Digests are a distributional sketch capable of maintaining high accuracy of quantiles that also allows merging while maintaining accuracy.
https://github.com/tdunning/t-digest

I have raised a discussion point on the possibility of applying T-Digests to achieve "summable summaries" in the OpenTelemetry Specification repository:
open-telemetry/opentelemetry-specification#3118

I would like to experiment adding such an instrument to try and advance this idea to the prototype phase. I am very confortable writing in Java and other languages for the JVM. Could someone point me in the direction of extending OpenTelemetry Java to add new experimental instruments?

[jkwatson]

I don't think you want a new instrument for this, but a new aggregation for the Histogram instrument, correct?

[jkwatson]

The histogram instrument only has methods to record values. The implementation of that is provided by the aggregation, which lives in the SDK. Currently, a histogram can be aggregated with a number of different aggregations, including both fixed-bucket histograms and base-2-exponential histograms.

I believe what you want is to add a new aggregation option for the t-digest. Does OTEL have a way to export t-digest data via OTLP? If not, that seems like the first step, which will require work in the OTLP specification.

[tdunning]

OK. Adding values is a piece of cake.

What are OTEL and OTLP?

[jkwatson]

Otel is OpenTelemetry. OTLP is the OpenTelemetry protocol, the standard wire format for otel.

I suggest you spend some time with the otel specification before tackling something as complex as adding an entirely new new aggregation.

[pedrolamarao]

I am familiar with OTEL and OTLP and indend to support this work. If I understand the conversation so far correctly, it should be possible to define "buckets" dinamically to corresponde with T-Digest's "centroids". I originally assumed that the histogram instrument required "buckets" with fixed, predefined size.

For public reference, please correct me Ted!, this is the most current published article explaining T-Digest: https://www.sciencedirect.com/science/article/pii/S2665963820300403

[jkwatson]

I recommend getting familiar with how the other aggregations for the histogram instrument work. Hopefully that will clear things up a bit about what this work might entail.

Here is the implementation of explicit buckets: https://github.com/open-telemetry/opentelemetry-java/blob/main/sdk/metrics/src/main/java/io/opentelemetry/sdk/metrics/internal/view/ExplicitBucketHistogramAggregation.java

Here is the implementation of exponential buckets: https://github.com/open-telemetry/opentelemetry-java/blob/main/sdk/metrics/src/main/java/io/opentelemetry/sdk/metrics/internal/view/Base2ExponentialHistogramAggregation.java

The main question you'll have to answer is how to convert your t-digest things into metric points that are exportable. And whether we even have appropriate point implementations at this point. And, then, how you might export this data somewhere so it could be used by a backend that supported this shape of data.

[tdunning]

John, Thanks for answering my questions. T-digest can produce any percentiles of interest. Commonly, these include median (0.5 quantile) and various tail measurements (0.9, 0.95, 0.99, 0.999, 0.9999) for instance. You can specify which quantiles at any time.

…

On Sat, Feb 11, 2023 at 11:55 AM John Watson ***@***.***> wrote: I recommend getting familiar with how the other aggregations for the histogram instrument work. Hopefully that will clear things up a bit about what this work might entail. Here is the implementation of explicit buckets: https://github.com/open-telemetry/opentelemetry-java/blob/main/sdk/metrics/src/main/java/io/opentelemetry/sdk/metrics/internal/view/ExplicitBucketHistogramAggregation.java Here is the implementation of exponential buckets: https://github.com/open-telemetry/opentelemetry-java/blob/main/sdk/metrics/src/main/java/io/opentelemetry/sdk/metrics/internal/view/Base2ExponentialHistogramAggregation.java The main question you'll have to answer is how to convert your t-digest things into metric points that are exportable. And whether we even have appropriate point implementations at this point. And, then, how you might export this data somewhere so it could be used by a backend that supported this shape of data. — Reply to this email directly, view it on GitHub <#5186 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAB5E6ROKFIUCNQJKYSNCTLWW7VEHANCNFSM6AAAAAAUVNYFK4> . You are receiving this because you commented.Message ID: <open-telemetry/opentelemetry-java/repo-discussions/5186/comments/4945308@ github.com>

[pedrolamarao]

This week I have time to dedicate to this activity. I'm having trouble building both main and release/1.23.x. I've filed #5250 about this. If someone can give me guidance on how to build opentelemetry-java, please advise.

[pedrolamarao]

In order to help our conversation move forward, I have implemented a new type of aggregator, the TDigestAggregator, that you can find here: https://github.com/pedrolamarao/opentelemetry-java/blob/digests/sdk/metrics/src/main/java/io/opentelemetry/sdk/metrics/internal/aggregator/TDigestAggregator.java

This aggregator records data by feeding it to a TDigest. It generates a HistogramDataPoint by transforming each TDigest centroid into a histogram datum by assigning centroid mean to histogram boundary and centroid count to histogram count. But this seems conceptually wrong: a TDigest centroid mean is not a boundary.

[jkwatson]

Yes, this is where I was wondering if OTLP has the right data structures currently to support tdigest export. You might need to go to the spec and work with the technical committee (paging @jack-berg ) to figure out how to extend OTLP to support tdigest structures.

[pedrolamarao]

I'll attempt to understand this from the perspective of the collector, building an experimental collector capable of understanding this data.

[jack-berg]

One of the reasons that custom aggregators aren't supported (at least yet) is that all aggregators have to output metric points as defined in the metric data model. Point kinds include: sum, gauge, histogram, exponential histogram, and summary (legacy). There are only so many ways to usefully aggregate measurements into these metric points, and most are covered by the small set of built in aggregators.

I'm not familiar with T-Digest, so I don't have intuition about the information needs to be encoded to be sent over the wire. I assume that in order to be summable, certain data has to be present. You'll need to study the data model (perhaps take a look at the metric proto definitions to for a concrete description) to determine if its feasible to encode T-Digest data into one of the existing metric point kinds without violating the semantics of the point fields. If its not possible, you'll have to propose extending the data model.

To extend the data model, you'll have to open an issue in the specification describing your proposal and convince others of its merit. I see you've opened a discussion - you'll want to file an issue instead. Extending the metric data model is a significant enough change that it requires going through the OTEP process. However, I suggest starting with an issue, and perhaps attending and presenting the idea in the specification SIG, which meets each Tuesday at 8am PST.

[tdunning]

You can estimate boundaries between centroids based on their counts. I would not recommend serializing a t-digest with boundaries, but they can be used for visualization just fine. You can also use the centroids as boundaries and assign half of the points to either side unless the count = 1. For unit count, there is a singleton point at the centroid so you should handle that with a special case.

…

On Mon, Feb 27, 2023 at 12:43 PM Pedro Lamarão ***@***.***> wrote: In order to help our conversation move forward, I have implemented a new type of aggregator, the TDigestAggregator, that you can find here: https://github.com/pedrolamarao/opentelemetry-java/blob/digests/sdk/metrics/src/main/java/io/opentelemetry/sdk/metrics/internal/aggregator/TDigestAggregator.java This aggregator records data by feeding it to a TDigest. It generates a HistogramDataPoint by transforming each TDigest centroid into a histogram datum by assigning centroid mean to histogram boundary and centroid count to histogram count. But this seems conceptually wrong: a TDigest centroid mean is not a boundary. — Reply to this email directly, view it on GitHub <#5186 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAB5E6T3N3SRSH6S7HQ747LWZUGW3ANCNFSM6AAAAAAUVNYFK4> . You are receiving this because you commented.Message ID: <open-telemetry/opentelemetry-java/repo-discussions/5186/comments/5127960@ github.com>

[pedrolamarao]

I think the end game is to reconstruct the TDigest at the point of collection, enabling the collector to sum TDigest "deltas". This would require the collector to understand the digest semantics. I'll try to demonstrate an experimental collector capable of doing this.

[tdunning]

OK. Take a digest where the centroid positions are x_i and the centroid weights are c_i where i = 1..n. Note that c_1 = c_n = 1. You can encode this digest as a histogram in a couple of ways that will look kind of OK when viewed as a histogram and which allow perfect reconstruction of the histogram (if you know the compression factor separately). If you can only have integer weights in the histogram, then use x_i as the bounds of n-1 bins and take the histogram weights as h_1 = c_1 + c_2 = c_2 + 1 h_j = c_{j+1} where j = 1..n-1. Reconstruction is trivial. If you can have non-integer weights in the histogram and want ever so slightly better looking histograms, take x_i as the bounds of n-1 bins and use weights h_1 = 1 + c_2/2 h_j = c_j/2 + c_{j+1}/2 where j = 2 ... n-2 h_{n-1} = c_{n-1}/2 + 1 Reconstruction is only slightly less trivial.

…

On Tue, Feb 28, 2023 at 8:39 AM Pedro Lamarão ***@***.***> wrote: I think the end game is to reconstruct the TDigest at the point of collection, enabling the collector do sum TDigest "deltas". This would require the collector to understand the digest semantics. I'll try to demonstrate an experimental collector capable of doing this. — Reply to this email directly, view it on GitHub <#5186 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAB5E6VZRT3QW7LAKGJB3MDWZYS5ZANCNFSM6AAAAAAUVNYFK4> . You are receiving this because you commented.Message ID: <open-telemetry/opentelemetry-java/repo-discussions/5186/comments/5154125@ github.com>

[pedrolamarao]

I'll put some time to try and demonstrate all alternatives for everyone's consideration.