Out of order exemplars when sending metrics using OpenTelemetry spanmetrics connector

[tiithansen]

What did you do?

We are seeing a lot of "out of order exemplars" logs when sending generated metrics from OpenTelemetry collector (spanmetrics connector) component. Specifically that components sends all exemplars attached to data point with same timestamp.

I have doubts about the following condition what checks if exemplar is out of order. It compares if the value of new exemplar is less then its considered to be out of order. Though exemplars value can be less or more it really doesn't matter. Same applies for the labels hash. It should be sufficient to just compare if exemplar timestamp is less assuming each series has its own circular buffer.

Code in question:

prometheus/tsdb/exemplar.go

Lines 266 to 268 in 4a6f870

	if e.Ts < newestExemplar.Ts ||
	(e.Ts == newestExemplar.Ts && e.Value < newestExemplar.Value) ||
	(e.Ts == newestExemplar.Ts && e.Value == newestExemplar.Value && e.Labels.Hash() < newestExemplar.Labels.Hash()) {

What did you expect to see?

No response

What did you see instead? Under which circumstances?

Out of order errors when actually everything was in order.

System information

No response

Prometheus version

No response

Prometheus configuration file

No response

Alertmanager version

No response

Alertmanager configuration file

No response

Logs

No response