Add concurrent batching to the file
sink
#20394
Labels
domain: performance
Anything related to Vector's performance
sink: file
Anything `file` sink related
type: enhancement
A value-adding code change that enhances its existing functionality.
A note for the community
Problem
I can see 100% utilization on file sink which then applies backpressure and slows-down whole pipeline. I am using tmpfs so disk is not a bottleneck but high cardinal partitioning could be. It seems that file sink is not batching concurrently and therefore applying backpressure quickly (especially with gzip compression).
Configuration
Version
0.37.0
Debug Output
No response
Example Data
No response
Additional Context
No response
References
No response
The text was updated successfully, but these errors were encountered: