BatchLogRecordProcessor's exportCurrentBatch wait for export finish, which limited logData to be exposed.

[qixiaogang]

Is your feature request related to a problem? Please describe.
Our application has a high throughput of logData, using BatchLogRecordProcessor as a processor to batch output, we find that throughput is limited and CPU & memory usage are not high. After some investigation, we find that bottleneck is in BatchLogRecordProcessor.

BatchLogRecordProcessor has only one work and batch export pending on export result, so throughput is limited.

Describe the solution you'd like
Support multiple worker for BatchLogRecordProcessor, and worker number is configurable.

Describe alternatives you've considered
NA

Additional context
NA

[jack-berg]

BatchSpanProcessor and BatchLogRecordProcessor are intended to synchronously wait for the exporter's CompletableResultCode to complete before continuing. Some options:

• Write a custom exporter which returns a CompletableResultCode that resolves immediately, allowing the single worker in BatchLogRecordProcessor to achieve higher throughput.
• Write a custom processor which has multiple workers pulling log records of the queue

[qixiaogang]

@jack-berg Thanks for the options. and one question, can BatchLogRecordProcessor support multiple workers pulling log records of the queue instead of writing a custom processor.

[jack-berg]

can BatchLogRecordProcessor support multiple workers pulling log records of the queue instead of writing a custom processor.

No it can't. And it would be difficult to make this happen. For one, the code itself would be tricky to write - its a pretty complex implementation with only a single thread. But because the batch log record processor reflects the spec, we would almost certainly want to see the spec evolved to describe this multi-worker scenario.