prometheusbridge: Creates invalid exemplars for Prometheus counters

[gouthamve]

Description

I was testing #5281 and I found errors in Mimir which looked like the following:

ts=2024-04-14T14:27:50.976027765Z caller=push.go:171 level=error user=12690 msg="push error" err="received an exemplar with no valid labels, timestamp: 0 series: blackbox_module_unknown_total{job=\"unknown_service:blackbox_exporter\"} labels: {} (err-mimir-exemplar-labels-missing)"
 ts=2024-04-14T14:27:30.96869299Z caller=push.go:171 level=error user=12690 msg="push error" err="received an exemplar with no valid labels, timestamp: 0 series: go_memstats_mallocs_total{job=\"unknown_service:blackbox_exporter\"} labels: {} (err-mimir-exemplar-labels-missing)"

Digging into the errors showed me that the Prometheus bridge is actually adding 0 exemplars even when none existed before. This is only happening with counters and not histograms from what I could see.

Environment

• OS: MacOS
• Architecture: arm64
• Go Version: go version go1.22.2 darwin/arm64
• go.opentelemetry.io/contrib version: go.opentelemetry.io/contrib/bridges/prometheus v0.50.0

Steps To Reproduce

A minimal reproduction is below:

package main

import (
	"time"

	"github.com/prometheus/client_golang/prometheus"
	prometheusbridge "go.opentelemetry.io/contrib/bridges/prometheus"
	"go.opentelemetry.io/otel/exporters/stdout/stdoutmetric"
	"go.opentelemetry.io/otel/sdk/metric"
)

func main() {
	reg := prometheus.NewRegistry()
	someDummyMetric := prometheus.NewCounter(prometheus.CounterOpts{
		Name: "dummy_metric",
		Help: "dummy metric",
	})
	reg.MustRegister(someDummyMetric)

	someDummyMetric.Inc()
	someDummyMetric.Add(2.0)
	someDummyMetric.Add(3.0)

	producer := prometheusbridge.NewMetricProducer(prometheusbridge.WithGatherer(reg))
	exporter, err := stdoutmetric.New()
	checkErr(err)

	reader := metric.NewPeriodicReader(exporter, metric.WithProducer(producer), metric.WithInterval(1*time.Second))
	metric.NewMeterProvider(metric.WithReader(reader))

	time.Sleep(2 * time.Second)
}

func checkErr(err error) {
	if err != nil {
		panic(err)
	}
}

➜  otel-prom-bridge-ghost-exemplars go run .
{"Resource":[{"Key":"service.name","Value":{"Type":"STRING","Value":"unknown_service:otel-prom-bridge-ghost-exemplars"}},{"Key":"telemetry.sdk.language","Value":{"Type":"STRING","Value":"go"}},{"Key":"telemetry.sdk.name","Value":{"Type":"STRING","Value":"opentelemetry"}},{"Key":"telemetry.sdk.version","Value":{"Type":"STRING","Value":"1.25.0"}}],"ScopeMetrics":[{"Scope":{"Name":"go.opentelemetry.io/otel/bridge/prometheus","Version":"","SchemaURL":""},"Metrics":[{"Name":"dummy_metric","Description":"dummy metric","Unit":"","Data":{"DataPoints":[{"Attributes":[],"StartTime":"2024-04-14T14:31:03.473136Z","Time":"2024-04-14T16:31:04.474453+02:00","Value":6,"Exemplars":[{"FilteredAttributes":[],"Time":"1970-01-01T00:00:00Z","Value":0}]}],"Temporality":"CumulativeTemporality","IsMonotonic":true}}]}]}

You can see that we are inserting an exemplar there, but it looks like a null exemplar.

Expected behavior

The exemplar array is empty.

cc @dashpole

[fstab]

According to the OpenMetrics spec it's ok to have Exemplars without labels:

Exemplars without Labels MUST represent an empty LabelSet as {}.

There's even an example in the spec:

foo_bucket{le="0.1"} 8 # {} 0.054

Why does Mimir reject this?

[gouthamve]

Good catch @fstab, filed grafana/mimir#7904

However, this bug is still relevant here because the bridge inserts a zero exemplar even when no exemplars are present.