BatchLogRecordProcessorBuilder doesn't force flush logs to export when the maxQueueSize is full.

[tongshushan]

Hello,

I set the 2 parameters of BatchLogRecordProcessorBuilder as below:
setScheduleDelay(Duration.ofMinutes(2)).setMaxQueueSize(10) ,
and I sent more than 100 logs in 1 minutes, but after 2 minutes the exporter only receive 8 logs, many logs lost.
But if I don't modify the maxQueueSize, left it as default, the exporter can receive all logs.

Configuration Code:

@Bean
   public OpenTelemetry openTelemetry() {
       Resource resource = Resource.getDefault()
               .merge(Resource.create(Attributes.builder()
                       .put("test", "xxx")
                       .put(ResourceAttributes.SERVICE_NAME, serviceName)
                       .build()));
       System.setProperty("otel.service.name", serviceName);
       System.setProperty("otel.exporter.otlp.endpoint", oltpLogEndpoint);
       System.setProperty("otel.logs.exporter", "otlp");
       OpenTelemetrySdk openTelemetrySdk = AutoConfiguredOpenTelemetrySdk.initialize().getOpenTelemetrySdk();
       SdkTracerProvider sdkTracerProvider = SdkTracerProvider.builder()
               .addSpanProcessor(BatchSpanProcessor.builder(OtlpGrpcSpanExporter.builder().setEndpoint(oltpLogEndpoint).build()).build())
               .setResource(resource)
               .build();
       SdkMeterProvider sdkMeterProvider = SdkMeterProvider.builder()
               .registerMetricReader(PeriodicMetricReader.builder(OtlpGrpcMetricExporter.builder().setEndpoint(oltpLogEndpoint).build()).build())
               .setResource(resource)
               .build();
       SdkLoggerProvider sdkLoggerProvider = SdkLoggerProvider.builder()
               .setResource(resource)
               .addLogRecordProcessor(BatchLogRecordProcessor.builder(OtlpGrpcLogRecordExporter.builder().setEndpoint(oltpLogEndpoint).build()).setScheduleDelay(Duration.ofMinutes(2)).setMaxQueueSize(10).build())
               .build();
       OpenTelemetrySdk upgradeOpenTelemetrySdk = openTelemetrySdk.builder()
               .setMeterProvider(sdkMeterProvider)
               .setTracerProvider(sdkTracerProvider)
               .setLoggerProvider(sdkLoggerProvider)
               .build();
       // Install OpenTelemetry in logback appender
       io.opentelemetry.instrumentation.logback.appender.v1_0.OpenTelemetryAppender.install(upgradeOpenTelemetrySdk);

       // Route JUL logs to slf4j
       SLF4JBridgeHandler.removeHandlersForRootLogger();
       SLF4JBridgeHandler.install();
       return upgradeOpenTelemetrySdk;
   }

Caller code:

   @Value("${spring.application.name}")
   private String serviceName;

  @Autowired
   OpenTelemetry openTelemetry;

   LoggerProvider loggerProvider;

   @Autowired
   ApiAopHelper(OpenTelemetry openTelemetry) {
       loggerProvider = openTelemetry.getLogsBridge();
   }

//just ignore the bussiness fields here
public void log2Otlp(args ...) {
    Logger logger = loggerProvider.get(serviceName);
       DataloaderLog dataloaderLog = new DataloaderLog();
       dataloaderLog.setCallApiMethodName(caller);
        ...
       AttributeKey<Long> durationKey = AttributeKey.longKey("duration");
       ...
       AttributeKey<String> response = AttributeKey.stringKey("response");

       Gson gson = new Gson();
       dataloaderLog.setDuration(duration);
       dataloaderLog.setResponse(status);
       LogRecordBuilder logRecordBuilder = logger.logRecordBuilder()
               .setSeverity(Severity.INFO)
               .setTimestamp(Instant.now())
              ...
               .setAttribute(durationKey, duration)
               .setAttribute(response, status);
       String body = "callApiMethodName {callApiMethodName} .... duration {duration} ms and response {response}";
       log.info(gson.toJson(dataloaderLog));
       logRecordBuilder.setBody(body);
       logRecordBuilder.emit();
  }

Steps to reproduce
Call method log2Otlp(args ...) more than 100 times to send logs.

Environment
java 8
io.opentelemetry: 1.37.0

[tongshushan]

I find the root cause is maxQueueSize < maxExportBatchSize.
Is it works as designed? If it is ,can the api give some clear message if misconfigued?

[jkwatson]

This is working as designed. I think logging a warning when this mis-configuration is done is a reasonable request. Can you put in a PR to add a log message, @tongshushan ?

[tongshushan]

Yes, I put the PR here:
#6454

Thanks.