Flushing of metrics in PeriodicReader

[Achooo]

Hi team :)

I've been diving into the code for the PeriodicReader (see here). I noticed that the collectAndExport function only exports metrics using the client. It does not clear any stored metrics from my understanding. The flush operation seems to only relate to client export.

I'm trying to understand, after reading through the overall SDK code (pipelines, instruments, etc.), how does instrumented metric data in memory get flushed/replaced?

Thanks a lot!

Edit: Think I figured parts of it out!

• A Reader - i.e. a PeriodicReader will collectAndExport metrics every X seconds (code)
• When it collects metrics, it calls the produce function of a pipeline (code)
• The pipeline calls the Aggregation method on the aggregators (code)
• The different aggregators, when Aggregation is called, return the aggregated data and delete existing data -> when metrics get flushed from memory essentially
  • Code example for LastValue or DeltaSum)
  • I’m a bit confused why the CummulativeSum doesn’t do so (code)

Does this understanding seem correct?

[MrAlias]

I’m a bit confused why the CummulativeSum doesn’t do so (code)

Cumulative temporality means data from all collection cycles is exported. It is not flushed from memory due to this essential behavior it needs to provide.

[Achooo]

Thanks that makes sense! I was expecting it to flush after X cycles or some sort of memory management there. I see this issue is open #3006 to address this.

[JobberRT]

Sorry for asking here, but I can't get this to work. I'm using PeriodRead + DeltaTemporality. But Prometheus still says it's "XXX_counter_total" and the data is cummulative. I don't know which part did I do wrong. I also read the Official Doc

Example Code

1. Create the Exporter

func newGRPCExporter(endpoint string) sdkmetric.Exporter {
	exp, err := otlpmetricgrpc.New(nil,
		otlpmetricgrpc.WithEndpoint(endpoint),
		otlpmetricgrpc.WithCompressor("gzip"),
		otlpmetricgrpc.WithTimeout(15*time.Second),
		otlpmetricgrpc.WithInsecure(),
		otlpmetricgrpc.WithTemporalitySelector(temporalityDeltaSelector),
	)
	if err != nil {
		panic("failed to create grpc exporter")
	}
	return exp
}

func temporalityDeltaSelector(_ sdkmetric.InstrumentKind) metricdata.Temporality {
	return metricdata.DeltaTemporality
}

By default, Counter will use SumAggregation, then I assign DeltaTemporality to the exporter

opentelemetry-go/sdk/metric/reader.go

Lines 147 to 160 in e2f8eca

	func DefaultAggregationSelector(ik InstrumentKind) aggregation.Aggregation {
	switch ik {
	case InstrumentKindCounter, InstrumentKindUpDownCounter, InstrumentKindObservableCounter, InstrumentKindObservableUpDownCounter:
	return aggregation.Sum{}
	case InstrumentKindObservableGauge:
	return aggregation.LastValue{}
	case InstrumentKindHistogram:
	return aggregation.ExplicitBucketHistogram{
	Boundaries: []float64{0, 5, 10, 25, 50, 75, 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000},
	NoMinMax: false,
	}
	}
	panic("unknown instrument kind")
	}

2. Setup the MeterProvider with the Exporter I just created

func setMeterProvider(res *resource.Resource, exp sdkmetric.Exporter, interval time.Duration) {
	var meterProvider *sdkmetric.MeterProvider
	meterProvider = sdkmetric.NewMeterProvider(
		sdkmetric.WithResource(res),
		sdkmetric.WithReader(sdkmetric.NewPeriodicReader(
			exp,
			sdkmetric.WithTimeout(15*time.Second),
			sdkmetric.WithInterval(interval)),
		),
	)
	otel.SetMeterProvider(meterProvider)
	providerShutdown = meterProvider.Shutdown
}

3. Do some load

var (
	counter metric.Int64Counter
)

func do() {
	counter.Add(context.Background(), rand.Int63n(10))
}

// metrics.Meter() just a wrapper function of otel.Meter()
func ExecuteMetrics() {
	counter, _ = metrics.Meter().Int64Counter("test counter")
	for i := 0; i < 10; i++ {
		time.Sleep(time.Duration(rand.Intn(6000)) * time.Millisecond)
		fmt.Println("do!")
		do()
	}
}

4. Finally shutdown the MeterProvider

Prometheus behavior

What I except to happen

Counter will be reset every export. For the demo, the data is randomly generated within 10 so I think in Prometheus the line chart should never beyond 10, and the line randomly go up or down.

I know there's a UpDownCounter but I think the UpDownCounter is mainly used for counter's data could add or sub, not the graph can go up or down(Within a export loop, data in SDK can be add or sub).

[MrAlias]

Prometheus only works with cumulative temporality. I'm guessing you are exporting to a collector that is then being scraped by Prometheus. The collector is performing a delta-to-cumulative transform to ensure Prometheus gets the correct data.

[Achooo]

Exactly as @MrAlias mentioned, Prometheus doesn't support Delta. Counters should be aggregated continuously (no resets) in a cummulative fashion.

In our collector, the otel contrib's prometheusremotewrite that we use will even drop any metrics that aren't cumulative (https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/db8208c23af6f5ac8782b2c2c241ac16fcf36131/pkg/translator/prometheusremotewrite/helper.go#L247)

We ended up changing the temporality to cumulative when using the Metrics SDK.

Some useful links:

• https://opentelemetry.io/docs/specs/otel/compatibility/prometheus_and_openmetrics/#sums
• https://opentelemetry.io/docs/specs/otel/metrics/data-model/#temporality

[JobberRT]

Sorry for replying via email if this is inconvenient to read. Then could I use Gauge instead？Before using Otel sdk， I was using this to achieve what I need. Will try tomorrow, got off work today Tyler Yahn ***@***.***> 于 2023年6月1日周四 22:19写道：

…

Prometheus only works with cumulative temporality. I'm guessing you are exporting to a collector that is then being scraped by Prometheus. The collector is performing a delta-to-cumulative transform to ensure Prometheus gets the correct data. — Reply to this email directly, view it on GitHub <#3993 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEFQFD423QH72V76EZEDF3TXJCQGFANCNFSM6AAAAAAW4P2LXI> . You are receiving this because you commented.Message ID: <open-telemetry/opentelemetry-go/repo-discussions/3993/comments/6061894@ github.com>