Add openmetrics and exemplars support #482

voltbit · 2021-11-26T08:52:31Z

Overview

Added support for the OpenMetrics standard
Added support for automatically providing exemplars populated with trace information from OpenTelemetry

The main purpose of the PR is to enable the use of Exemplars for NodeJS codebase.

Relevant resources:

Grafana exemplars
Prometheus exemplars
OpenMetics spec
Prometheus format

Design

The OpenTelemetry standard is very close to the original Prometheus format. In order to keep the library as backwards compatible as possible the default format is kept unchanged (Prometheus) and the use of exemplars is disabled.

The new features can be toggled:

The format at the registry level (prometheus/openmetrics)
The exemplars at the metric level

Each registry instance has an attribute (contentType) that will decide the format.
The two possible formats are defined by the constants OPENMETRICS_CONTENT_TYPE and PROMETHEUS_CONTENT_TYPE which contain the HTTP content type.
Future versions should default to the 1.0.0 version.

Each metric has a flag for enabling the exemplar, the flag is put on the metrics supeclass for simplicity, but out of the currently implemented metric types only histograms and counters can have exemplars.

The biggest change to the code is the creation of separate functions for Counter increment and Histogram observe. Because the functions need to support a third optional parameter (exemplar labels) I have changed the way parameters are passed to the functions. Instead of using plain (label, value) the users will need to provide a single object with the format ({labels, value, exemplarLabels}).
The change should not impact existing users, but users who want to use exemplars will need to use the new call format.

Exemplar object

Timestamp - is the time when the exemplar was created
Reference from the Golang client: https://github.com/prometheus/client_golang/blob/1b145cad6847a692bd07e872d64b7102d33213c6/prometheus/histogram.go#L432.

There is a hard 128 UTF-8 character limit on exemplar length.

The labels use for out of the box traces are traceId and spanId, it feels more like JavaScript to me, there is no other reason for the name choice. The golang implementation seems to be using traceID here and the Java impl. uses trace_id here. The label used for exemplars can be changed in Grafana.

Counters in OpenMetrics

Counters have a brekaing change in the form of an enforced _total suffix, it is not just a convention anymore. Examples:

Prometheus

# HELP mycounter help
# TYPE mycounter counter
mycounter 0

# HELP mycounter2_total help
# TYPE mycounter2_total counter
mycounter2_total 0

OpenMetrics

# HELP mycounter help
# TYPE mycounter counter
mycounter_total 0

Prometheus ignores the comments related to name and type, but the name of the metrics changes too and has the potential to break dashboards/alerts etc. The current implementation follows the same approach as the Java implementation here:

https://github.com/prometheus/client_java/blob/master/simpleclient/src/main/java/io/prometheus/client/Counter.java#L72-L108

However, instead of applying the suffix at the level of the Counter object, this implementation applyes the change in the Registry object. The disadvantage is that the code is less elegant. The advantage is that the change is not breaking in any way for the existing users - the _total suffix will only be enforced by OpenMetrics registries, not Prometheus registries.

In the future, when OpenMetrics becomes more widely adopted, the behaviour can be moved inside Counter object and made mandatory.

Benchmarks

Benchmark tests were not changed. They are using the default registry type (Prometheus) and no exemplars, so it is a check to see the impact for current users of the library. Ran 4 tests (results in gists bellow). The highest impact was on the registry benchmark with a ~10-15% performance hit.

⚠ registry ➭ getMetricsAsJSON#1 with 64 is 5.345% acceptably slower.
⚠ registry ➭ getMetricsAsJSON#2 with 8 is 3.076% acceptably slower.
⚠ registry ➭ getMetricsAsJSON#2 with 4 and 2 with 2 is 4.063% acceptably slower.
✓ registry ➭ getMetricsAsJSON#2 with 2 and 2 with 4 is 0.1468% faster.
⚠ registry ➭ getMetricsAsJSON#6 with 2 is 4.137% acceptably slower.
✗ registry ➭ metrics#1 with 64 is 11.50% slower.
✓ registry ➭ metrics#2 with 8 is 0.2174% faster.
⚠ registry ➭ metrics#2 with 4 and 2 with 2 is 1.046% acceptably slower.
⚠ registry ➭ metrics#2 with 2 and 2 with 4 is 3.413% acceptably slower.
✗ registry ➭ metrics#6 with 2 is 15.03% slower.
⚠ histogram ➭ observe#1 with 64 is 0.2735% acceptably slower.
⚠ histogram ➭ observe#2 with 8 is 0.5325% acceptably slower.
⚠ histogram ➭ observe#2 with 4 and 2 with 2 is 0.09087% acceptably slower.
⚠ histogram ➭ observe#2 with 2 and 2 with 4 is 0.3243% acceptably slower.
⚠ histogram ➭ observe#6 with 2 is 0.6302% acceptably slower.
✓ gauge ➭ inc is 16.96% faster.
⚠ gauge ➭ inc with labels is 1.991% acceptably slower.
⚠ summary ➭ observe#1 with 64 is 2.261% acceptably slower.
✓ summary ➭ observe#2 with 8 is 2.166% faster.
✓ summary ➭ observe#2 with 4 and 2 with 2 is 0.4552% faster.
⚠ summary ➭ observe#2 with 2 and 2 with 4 is 1.407% acceptably slower.
⚠ summary ➭ observe#6 with 2 is 1.116% acceptably slower.

https://gist.github.com/voltbit/55bfdafccb5a0458d0b2aff9703dae43
https://gist.github.com/voltbit/1e1097e6400638334e11da52fefcd5d4
https://gist.github.com/voltbit/41828df848a7132c1aad196414ea2d69
https://gist.github.com/voltbit/70539929453a2b95d4f9ac2df6707a9b

TODO

Complete test coverage of the new features
Performance tests
Add better examples and readme info
~~Strategy for registry merge when there are different registry formats~~
- decided to consider the merge of two different types of registries undefined behaviour, the users should always use the same type - Prometheus or OpenMetrics - if merging
Hadling the _total suffix on counters

Not implemented

Support for the _created suffix on any metrics

SimenB · 2021-11-26T10:32:53Z

Very exciting, thanks for working on this!

SimenB · 2021-11-26T10:36:19Z

example/server.js

@@ -91,6 +92,8 @@ setInterval(() => {

 server.get('/metrics', async (req, res) => {
 	try {
+		// register.contentType = Registry.OPENMETRICS_CONTENT_TYPE;
+		register.contentType = Registry.PROMETHEUS_CONTENT_TYPE;


should go through setContentType I guess?

Thanks, yeah there is a lot of inconsistency in how that flag is set, I'll iron them out.

voltbit · 2022-01-13T11:21:50Z

Hi @zbjornson could you please trigger the tests again? The PR should be ready for review :).

zbjornson · 2022-01-13T16:56:40Z

CI is green! I'll try to review this this weekend and hopefully @SimenB and/or @siimon can also review soon.

shyimo · 2022-01-23T08:37:56Z

thanks @voltbit & @zbjornson! we are excited about this feature!

shyimo · 2022-02-23T10:06:48Z

Hi @zbjornson.
Any news on that ?

SimenB

skimmed through, looks good! Lots of code, but existent tests seems to pass, so safe enough?

SimenB · 2022-02-23T10:12:07Z

CHANGELOG.md

+- feat: new option for calling `observe()` and `inc()` methods on the histogram
+  and counter metric types that can be passed an object of format
+  `{labels: a, value: b, exemplarLabels: c}`


should this be a separate PR?

Right now the implementation is quite tightly coupled to the feature for exemplars. I though about making this more generic from the beginning but I was worried about the performance impact. I tried to provide 100% backwards compatibility and minimal performance impact with the change.

Right now the call with inc({value, labels, exemplarLabels}) can only be done if the metric was created with exemplars enabled.

If I make this call more generic - allow inc() and observe() to be called with an object - I would need to differentiate between the inc({value, labels, exemplarLabels}) and the call that already exists inc({labelNames}). This might not be that bad, but it implies a check on the keys of the object every time inc/obs are used. I should also add it to all metrics, not just counter and histogram for the sake of consistency.

I am not sure which is the better aproach, let me know if you have feedback about this.

So the alternative to what we have now in the PR is: implement an object type check on all metric types and allow them to receive a single object containing all the relevant data for ins/obs operations. This might be useful on the long run, but it may as well be a feature that slows down the library for years until we need more fields like we do with exemplars (which are kind of a fringe feature anyway).

So the alternative to what we have now in the PR is: implement an object type check on all metric types and allow them to receive a single object containing all the relevant data for ins/obs operations. This might be useful on the long run, but it may as well be a feature that slows down the library for years until we need more fields like we do with exemplars (which are kind of a fringe feature anyway).

One thing we could do is drop support for passing labels (inc({labelNames}) like you mention). That way you either pass a single value or an object. I don't typeof arg === 'number' adds much overhead, and then the object shape would be what we want.

Would require a major version of course, but that's not an issue. We wanna drop old nodes anyways

README.md

SimenB · 2022-02-23T10:13:43Z

README.md

+Merging registries of different types is undefined. The user needs to make sure
+all used registries have the same type (Prometheus or OpenMetrics versions).


again, can we throw a usweful error?

https://github.com/siimon/prom-client/pull/482/files#diff-c76cf141522a047fee59ae738cbc8a0dc2c48ed15666d958d407d867c3a4b69fR237

Docs should be updated to say it throws, no? Not undefined behavior

index.d.ts

lib/counter.js

test/metrics/maxFileDescriptorsTest.js

dnutels · 2022-04-24T08:27:09Z

Hi guys. Are there, by chance, any updates on how this is progressing and an estimate, hopefully?

voltbit · 2022-04-24T12:10:48Z

Hi guys. Are there, by chance, any updates on how this is progressing and an estimate, hopefully?

Hi @dnutels I will start working again on this in a couple of weeks, I will implement the changes requested around mid May, but I cant work on it earlier.

voltbit · 2022-05-18T09:12:23Z

New runs for benchmarks with the latest changes.

Expand for benchmark results

Summary:

✗ registry ➭ getMetricsAsJSON#1 with 64 is 11.43% slower.
✗ registry ➭ getMetricsAsJSON#2 with 8 is 155.0% slower.
✓ registry ➭ getMetricsAsJSON#2 with 4 and 2 with 2 is 58.48% faster.
✗ registry ➭ getMetricsAsJSON#2 with 2 and 2 with 4 is 23.80% slower.
⚠ registry ➭ getMetricsAsJSON#6 with 2 is 6.297% acceptably slower.
⚠ registry ➭ metrics#1 with 64 is 5.984% acceptably slower.
⚠ registry ➭ metrics#2 with 8 is 1.545% acceptably slower.
⚠ registry ➭ metrics#2 with 4 and 2 with 2 is 0.09495% acceptably slower.
✓ registry ➭ metrics#2 with 2 and 2 with 4 is 35.38% faster.
⚠ registry ➭ metrics#6 with 2 is 3.828% acceptably slower.
✓ histogram ➭ observe#1 with 64 is 0.2496% faster.
⚠ histogram ➭ observe#2 with 8 is 0.8472% acceptably slower.
⚠ histogram ➭ observe#2 with 4 and 2 with 2 is 0.7701% acceptably slower.
✓ histogram ➭ observe#2 with 2 and 2 with 4 is 3.590% faster.
⚠ histogram ➭ observe#6 with 2 is 3.784% acceptably slower.
✓ gauge ➭ inc is 2.087% faster.
✓ gauge ➭ inc with labels is 0.2869% faster.
✓ summary ➭ observe#1 with 64 is 3.924% faster.
✓ summary ➭ observe#2 with 8 is 0.2697% faster.
✓ summary ➭ observe#2 with 4 and 2 with 2 is 1.130% faster.
✓ summary ➭ observe#2 with 2 and 2 with 4 is 0.4797% faster.
✓ summary ➭ observe#6 with 2 is 1.389% faster.

Summary:

✓ registry ➭ getMetricsAsJSON#1 with 64 is 1.990% faster.
✓ registry ➭ getMetricsAsJSON#2 with 8 is 321.0% faster.
⚠ registry ➭ getMetricsAsJSON#2 with 4 and 2 with 2 is 8.586% acceptably slower.
✓ registry ➭ getMetricsAsJSON#2 with 2 and 2 with 4 is 13.94% faster.
✗ registry ➭ getMetricsAsJSON#6 with 2 is 16.18% slower.
✓ registry ➭ metrics#1 with 64 is 20.45% faster.
⚠ registry ➭ metrics#2 with 8 is 0.8526% acceptably slower.
✓ registry ➭ metrics#2 with 4 and 2 with 2 is 5.139% faster.
⚠ registry ➭ metrics#2 with 2 and 2 with 4 is 6.645% acceptably slower.
✓ registry ➭ metrics#6 with 2 is 0.7693% faster.
⚠ histogram ➭ observe#1 with 64 is 3.102% acceptably slower.
✓ histogram ➭ observe#2 with 8 is 1.538% faster.
✓ histogram ➭ observe#2 with 4 and 2 with 2 is 0.4875% faster.
✓ histogram ➭ observe#2 with 2 and 2 with 4 is 0.7254% faster.
✓ histogram ➭ observe#6 with 2 is 2.165% faster.
✓ gauge ➭ inc is 22.42% faster.
⚠ gauge ➭ inc with labels is 1.116% acceptably slower.
✓ summary ➭ observe#1 with 64 is 2.565% faster.
⚠ summary ➭ observe#2 with 8 is 0.2203% acceptably slower.
⚠ summary ➭ observe#2 with 4 and 2 with 2 is 0.08091% acceptably slower.
✓ summary ➭ observe#2 with 2 and 2 with 4 is 1.113% faster.
✓ summary ➭ observe#6 with 2 is 1.459% faster.

Summary:

✗ registry ➭ getMetricsAsJSON#1 with 64 is 12.81% slower.
✗ registry ➭ getMetricsAsJSON#2 with 8 is 563.2% slower.
✓ registry ➭ getMetricsAsJSON#2 with 4 and 2 with 2 is 7.191% faster.
⚠ registry ➭ getMetricsAsJSON#2 with 2 and 2 with 4 is 8.622% acceptably slower.
✗ registry ➭ getMetricsAsJSON#6 with 2 is 13.17% slower.
✗ registry ➭ metrics#1 with 64 is 25.55% slower.
⚠ registry ➭ metrics#2 with 8 is 2.849% acceptably slower.
⚠ registry ➭ metrics#2 with 4 and 2 with 2 is 3.033% acceptably slower.
⚠ registry ➭ metrics#2 with 2 and 2 with 4 is 2.256% acceptably slower.
⚠ registry ➭ metrics#6 with 2 is 4.078% acceptably slower.
⚠ histogram ➭ observe#1 with 64 is 1.686% acceptably slower.
✓ histogram ➭ observe#2 with 8 is 0.3382% faster.
⚠ histogram ➭ observe#2 with 4 and 2 with 2 is 0.2078% acceptably slower.
⚠ histogram ➭ observe#2 with 2 and 2 with 4 is 6.005% acceptably slower.
⚠ histogram ➭ observe#6 with 2 is 2.634% acceptably slower.
✓ gauge ➭ inc is 12.39% faster.
✓ gauge ➭ inc with labels is 2.485% faster.
✓ summary ➭ observe#1 with 64 is 9374% faster.
✓ summary ➭ observe#2 with 8 is 2.167% faster.
✗ summary ➭ observe#2 with 4 and 2 with 2 is 19.97% slower.
⚠ summary ➭ observe#2 with 2 and 2 with 4 is 1.552% acceptably slower.
⚠ summary ➭ observe#6 with 2 is 0.3165% acceptably slower.

shyimo · 2022-06-07T07:15:57Z

Hi @zbjornson. any news regarding the PR ?

hitwill · 2022-06-22T03:35:32Z

Bumping for interest

vjsamuel · 2022-07-19T20:58:15Z

Bumping for interest again :)

isaac-elvt · 2022-08-15T22:47:46Z

Bumping for interest as well, this would save my organization so much pain!

xal3xhx · 2022-10-01T17:03:16Z

bump for intrest. i really need this

shyimo · 2022-10-12T11:43:09Z

bump for interest. really important task

ejba · 2022-10-27T10:23:00Z

bumping for interest!

SimenB

this generally LGTM. Would love another set of eyes, tho.

Thanks for a fantastic contribution, @voltbit!

SimenB · 2022-10-27T10:36:35Z

example/exemplars.ts

@@ -0,0 +1,66 @@
+import * as prom from '../index';


I don't think we need both a JS and a TS example - could you remove this one?

SimenB · 2022-10-27T10:37:52Z

index.d.ts

@@ -150,16 +185,29 @@ interface MetricConfiguration<T extends string> {
 	name: string;
 	help: string;
 	labelNames?: T[] | readonly T[];
-	registers?: Registry[];
+	registers?: Registry<PrometheusContentType | OpenMetricsContentType>[];


Suggested change

registers?: Registry<PrometheusContentType | OpenMetricsContentType>[];

registers?: (Registry<PrometheusContentType> | Registry<OpenMetricsContentType>)[];

right? A registry itself cannot contain either

SimenB · 2022-10-27T10:38:53Z

lib/cluster.js

@@ -84,19 +84,30 @@ class AggregatorRegistry extends Registry {
 		});
 	}

+	get contentType() {


is this needed? won't the class just inherit from super?

SimenB · 2022-10-27T10:41:07Z

lib/histogram.js

@@ -27,14 +43,21 @@ class Histogram extends Metric {
 			return acc;
 		}, {});

+		this.bucketExemplars = this.upperBounds.reduce((acc, upperBound) => {


should this be created even if config.enableExemplars isn't true?

package.json

SimenB · 2022-10-27T10:51:28Z

@zbjornson @siimon PTAL 🙂

voltbit · 2022-10-27T11:15:28Z

Thanks for all the feedback and all the interest shown! I'll rebase and check the comments as soon as possible (this weekend most likely).

ejba · 2022-10-31T18:15:18Z

@voltbit is there anything we can do to help you?

- Added support for OpenMetrics format, including exemplars - Added support for exemplars with OpenTelemetry tracing data on default metrics - Added the option of passing params as one object to observe() and inc() methods

SimenB · 2022-11-02T11:30:39Z

lib/metrics/processCpuTotal.js

@@ -1,6 +1,8 @@
 'use strict';

+const OtelApi = require('@opentelemetry/api');


this should be added to package.json

SimenB · 2022-11-02T11:31:19Z

lib/registry.js

+			regContentType !== Registry.PROMETHEUS_CONTENT_TYPE &&
+			regContentType !== Registry.OPENMETRICS_CONTENT_TYPE
+		) {
+			throw new TypeError('Content type unsupported');


Suggested change

throw new TypeError('Content type unsupported');

throw new TypeError(`Content type ${regContentType} is unsupported`);

voltbit · 2022-11-14T11:27:32Z

hi @shyimo & @ejba & other contributors
I am not sure I will get the time to work on this again before the holiday season. If this work is urgent for you and you want to pick up the task feel free to do so.

SimenB · 2023-03-06T14:28:37Z

Superseded by #544

SimenB · 2023-03-09T12:03:00Z

Released in https://github.com/siimon/prom-client/releases/tag/v15.0.0-0

SimenB reviewed Nov 26, 2021

View reviewed changes

voltbit force-pushed the add-openmetrics-support branch 5 times, most recently from bb10a72 to 6ae459f Compare December 3, 2021 10:49

voltbit force-pushed the add-openmetrics-support branch 4 times, most recently from e9492ce to e9f6c44 Compare January 12, 2022 13:47

voltbit marked this pull request as ready for review January 12, 2022 13:48

voltbit mentioned this pull request Jan 12, 2022

Add suport for registry types, OpenMetrics and exemplars joao-fontenele/express-prometheus-middleware#74

Draft

SimenB requested review from zbjornson and siimon February 23, 2022 10:11

SimenB reviewed Feb 23, 2022

View reviewed changes

voltbit marked this pull request as draft May 12, 2022 10:26

voltbit force-pushed the add-openmetrics-support branch 4 times, most recently from d16917e to 5e10b87 Compare May 18, 2022 08:45

voltbit force-pushed the add-openmetrics-support branch 2 times, most recently from d24e4fc to ebdbc91 Compare May 19, 2022 11:41

voltbit marked this pull request as ready for review May 20, 2022 08:22

voltbit requested a review from SimenB May 20, 2022 08:23

SimenB approved these changes Oct 27, 2022

View reviewed changes

voltbit marked this pull request as draft November 1, 2022 15:58

voltbit added 2 commits November 1, 2022 17:31

Add support for OpenMetrics format

97fc813

- Added support for OpenMetrics format, including exemplars - Added support for exemplars with OpenTelemetry tracing data on default metrics - Added the option of passing params as one object to observe() and inc() methods

Enforce stronger typing on registries

bd807c6

voltbit force-pushed the add-openmetrics-support branch from ebdbc91 to bd807c6 Compare November 1, 2022 16:34

SimenB reviewed Nov 2, 2022

View reviewed changes

This comment was marked as off-topic.

Sign in to view

This comment was marked as spam.

Sign in to view

karlodwyer mentioned this pull request Mar 3, 2023

Add openmetrics and exemplars support #544

Merged

karlodwyer added a commit to karlodwyer/prom-client that referenced this pull request Mar 3, 2023

Fix tests and address comments on PR: siimon#482

17830bc

karlodwyer added a commit to karlodwyer/prom-client that referenced this pull request Mar 3, 2023

Address comments and fix tests on PR: siimon#482

ef89282

SimenB closed this Mar 6, 2023

		Merging registries of different types is undefined. The user needs to make sure
		all used registries have the same type (Prometheus or OpenMetrics versions).

	registers?: Registry<PrometheusContentType \| OpenMetricsContentType>[];
	registers?: (Registry<PrometheusContentType> \| Registry<OpenMetricsContentType>)[];

		@@ -1,6 +1,8 @@
		'use strict';

		const OtelApi = require('@opentelemetry/api');

	throw new TypeError('Content type unsupported');
	throw new TypeError(`Content type ${regContentType} is unsupported`);

Add openmetrics and exemplars support #482

Add openmetrics and exemplars support #482

Conversation

voltbit commented Nov 26, 2021 • edited

Overview

Design

Counters in OpenMetrics

Benchmarks

SimenB commented Nov 26, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

voltbit commented Jan 13, 2022

zbjornson commented Jan 13, 2022

shyimo commented Jan 23, 2022

shyimo commented Feb 23, 2022

SimenB left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

voltbit May 20, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dnutels commented Apr 24, 2022

voltbit commented Apr 24, 2022

voltbit commented May 18, 2022

Summary:

Summary:

Summary:

shyimo commented Jun 7, 2022

hitwill commented Jun 22, 2022

vjsamuel commented Jul 19, 2022

isaac-elvt commented Aug 15, 2022

xal3xhx commented Oct 1, 2022

shyimo commented Oct 12, 2022

ejba commented Oct 27, 2022

SimenB left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SimenB commented Oct 27, 2022

voltbit commented Oct 27, 2022

ejba commented Oct 31, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

voltbit commented Nov 14, 2022

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as spam.

SimenB commented Mar 6, 2023

SimenB commented Mar 9, 2023

voltbit commented Nov 26, 2021 •

edited

voltbit May 20, 2022 •

edited