Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for BatchLogRecordProcessor autoconfiguration #4811

Merged
merged 1 commit into from Oct 17, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
14 changes: 14 additions & 0 deletions sdk-extensions/autoconfigure/README.md
Expand Up @@ -11,8 +11,10 @@ environment variables, e.g., `OTEL_TRACES_EXPORTER=zipkin`.
## Contents

* [General notes](#general-notes)
* [Disabling OpenTelemetrySdk](#disabling-opentelemetrysdk)
* [Exporters](#exporters)
+ [OTLP exporter (span, metric, and log exporters)](#otlp-exporter-span-metric-and-log-exporters)
+ [OTLP exporter retry](#otlp-exporter-retry)
+ [Jaeger exporter](#jaeger-exporter)
+ [Zipkin exporter](#zipkin-exporter)
+ [Prometheus exporter](#prometheus-exporter)
Expand All @@ -23,8 +25,11 @@ environment variables, e.g., `OTEL_TRACES_EXPORTER=zipkin`.
+ [Disabling automatic ResourceProviders](#disabling-automatic-resourceproviders)
* [Batch span processor](#batch-span-processor)
* [Sampler](#sampler)
* [Attribute limits](#attribute-limits)
* [Span limits](#span-limits)
* [Exemplars](#exemplars)
* [Periodic Metric Reader](#periodic-metric-reader)
* [Batch log record processor](#batch-log-record-processor)
* [Customizing the OpenTelemetry SDK](#customizing-the-opentelemetry-sdk)

## General notes
Expand Down Expand Up @@ -276,6 +281,15 @@ These properties can be used to control the maximum size of spans by placing lim
|-----------------------------|-----------------------------|----------------------------------------------------------------------------------------------|
| otel.metric.export.interval | OTEL_METRIC_EXPORT_INTERVAL | The interval, in milliseconds, between the start of two export attempts. Default is `60000`. |

## Batch log record processor

| System property | Environment variable | Description |
|---------------------------------|---------------------------------|------------------------------------------------------------------------------------|
| otel.blrp.schedule.delay | OTEL_BLRP_SCHEDULE_DELAY | The interval, in milliseconds, between two consecutive exports. Default is `5000`. |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm never going to be able to see this without yelling "BLURP!"

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

or maybe "BLORP!"

| otel.blrp.max.queue.size | OTEL_BLRP_MAX_QUEUE_SIZE | The maximum queue size. Default is `2048`. |
| otel.blrp.max.export.batch.size | OTEL_BLRP_MAX_EXPORT_BATCH_SIZE | The maximum batch size. Default is `512`. |
| otel.blrp.export.timeout | OTEL_BLRP_EXPORT_TIMEOUT | The maximum allowed time, in milliseconds, to export data. Default is `30000`. |

## Customizing the OpenTelemetry SDK

Autoconfiguration exposes SPI [hooks](../autoconfigure-spi/src/main/java/io/opentelemetry/sdk/autoconfigure/spi) for customizing behavior programmatically as needed.
Expand Down
3 changes: 2 additions & 1 deletion sdk-extensions/autoconfigure/build.gradle.kts
Expand Up @@ -25,7 +25,7 @@ dependencies {

annotationProcessor("com.google.auto.value:auto-value")

testImplementation(project(path = ":sdk:trace-shaded-deps"))
testImplementation(project(":sdk:trace-shaded-deps"))

testImplementation(project(":sdk:testing"))
testImplementation("com.linecorp.armeria:armeria-junit5")
Expand Down Expand Up @@ -86,6 +86,7 @@ testing {
implementation(project(":exporters:zipkin"))
implementation(project(":sdk-extensions:resources"))
implementation(project(":sdk:testing"))
implementation(project(":sdk:trace-shaded-deps"))
implementation(project(":semconv"))

implementation("com.google.guava:guava")
Expand Down
Expand Up @@ -14,8 +14,10 @@
import io.opentelemetry.sdk.logs.LogRecordProcessor;
import io.opentelemetry.sdk.logs.SdkLoggerProviderBuilder;
import io.opentelemetry.sdk.logs.export.BatchLogRecordProcessor;
import io.opentelemetry.sdk.logs.export.BatchLogRecordProcessorBuilder;
import io.opentelemetry.sdk.logs.export.LogRecordExporter;
import io.opentelemetry.sdk.logs.export.SimpleLogRecordProcessor;
import java.time.Duration;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
Expand All @@ -38,13 +40,15 @@ static void configureLoggerProvider(
configureLogRecordExporters(
config, serviceClassLoader, meterProvider, logRecordExporterCustomizer);

configureLogRecordProcessors(exportersByName, meterProvider)
configureLogRecordProcessors(config, exportersByName, meterProvider)
.forEach(loggerProviderBuilder::addLogRecordProcessor);
}

// Visible for testing
static List<LogRecordProcessor> configureLogRecordProcessors(
Map<String, LogRecordExporter> exportersByName, MeterProvider meterProvider) {
ConfigProperties config,
Map<String, LogRecordExporter> exportersByName,
MeterProvider meterProvider) {
Map<String, LogRecordExporter> exportersByNameCopy = new HashMap<>(exportersByName);
List<LogRecordProcessor> logRecordProcessors = new ArrayList<>();

Expand All @@ -57,14 +61,41 @@ static List<LogRecordProcessor> configureLogRecordProcessors(
LogRecordExporter compositeLogRecordExporter =
LogRecordExporter.composite(exportersByNameCopy.values());
logRecordProcessors.add(
BatchLogRecordProcessor.builder(compositeLogRecordExporter)
.setMeterProvider(meterProvider)
.build());
configureBatchLogRecordProcessor(config, compositeLogRecordExporter, meterProvider));
}

return logRecordProcessors;
}

// VisibleForTesting
static BatchLogRecordProcessor configureBatchLogRecordProcessor(
ConfigProperties config, LogRecordExporter exporter, MeterProvider meterProvider) {
BatchLogRecordProcessorBuilder builder =
BatchLogRecordProcessor.builder(exporter).setMeterProvider(meterProvider);

Duration scheduleDelay = config.getDuration("otel.blrp.schedule.delay");
if (scheduleDelay != null) {
builder.setScheduleDelay(scheduleDelay);
}

Integer maxQueue = config.getInt("otel.blrp.max.queue.size");
if (maxQueue != null) {
builder.setMaxQueueSize(maxQueue);
}

Integer maxExportBatch = config.getInt("otel.blrp.max.export.batch.size");
if (maxExportBatch != null) {
builder.setMaxExportBatchSize(maxExportBatch);
}

Duration timeout = config.getDuration("otel.blrp.export.timeout");
if (timeout != null) {
builder.setExporterTimeout(timeout);
}

return builder.build();
}

// Visible for testing
static LogLimits configureLogLimits(ConfigProperties config) {
LogLimitsBuilder builder = LogLimits.builder();
Expand Down
Expand Up @@ -16,8 +16,11 @@
import io.opentelemetry.sdk.logs.export.BatchLogRecordProcessor;
import io.opentelemetry.sdk.logs.export.LogRecordExporter;
import io.opentelemetry.sdk.logs.export.SimpleLogRecordProcessor;
import io.opentelemetry.sdk.trace.internal.JcTools;
import java.util.Collections;
import java.util.HashMap;
import java.util.Map;
import java.util.Queue;
import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.TimeUnit;
import org.junit.jupiter.api.Test;
Expand Down Expand Up @@ -73,10 +76,50 @@ void configureLogRecordProcessors_multipleExportersWithLogging() {

assertThat(
LoggerProviderConfiguration.configureLogRecordProcessors(
DefaultConfigProperties.createForTest(Collections.emptyMap()),
ImmutableMap.of("logging", loggingExporter, "otlp", otlpExporter),
MeterProvider.noop()))
.hasSize(2)
.hasAtLeastOneElementOfType(SimpleLogRecordProcessor.class)
.hasAtLeastOneElementOfType(BatchLogRecordProcessor.class);
}

@Test
void configureBatchLogRecordProcessor() {
Map<String, String> properties = new HashMap<>();
properties.put("otel.blrp.schedule.delay", "100000");
properties.put("otel.blrp.max.queue.size", "2");
properties.put("otel.blrp.max.export.batch.size", "3");
properties.put("otel.blrp.export.timeout", "4");

BatchLogRecordProcessor processor =
LoggerProviderConfiguration.configureBatchLogRecordProcessor(
DefaultConfigProperties.createForTest(properties),
SystemOutLogRecordExporter.create(),
MeterProvider.noop());

try {
assertThat(processor)
.extracting("worker")
.satisfies(
worker -> {
assertThat(worker)
.extracting("scheduleDelayNanos")
.isEqualTo(TimeUnit.MILLISECONDS.toNanos(100000));
assertThat(worker)
.extracting("exporterTimeoutNanos")
.isEqualTo(TimeUnit.MILLISECONDS.toNanos(4));
assertThat(worker).extracting("maxExportBatchSize").isEqualTo(3);
assertThat(worker)
.extracting("queue")
.isInstanceOfSatisfying(
Queue.class, queue -> assertThat(JcTools.capacity(queue)).isEqualTo(2));
assertThat(worker)
.extracting("logRecordExporter")
.isInstanceOf(SystemOutLogRecordExporter.class);
});
} finally {
processor.shutdown();
}
}
}