Fluentd multiple processors?

[theherodied]

What is a problem?

I'm currently forwarding logs from FortiAnalyzer as syslog to fluentd. Fluentd then passes them as is to Kafka. The issue that I'm seeing is that under a load of about 15,000 messages per second the cpu maxes one core at 100%. This seems to be throttling the input ability.

I've tried enabling multiple workers but only one seems to be used. Is it possible to increase the number of processors for the input?

Describe the configuration of Fluentd

####
## Output descriptions:
##

#Test add multiple workers 20240425-1445
#<system>
# workers 4
#</system>


#<worker 0-3>
<source>
@type tcp
port 10526
protocol_type tcp
tag fortinet.fgt
<parse>
@type none
</parse>
</source>


<match fortinet.fgt.**>
@type kafka2
# brokers kafka1.company.com:9092
brokers kafka1.company.com:9092,kafka2.company.com:9092,kafka3.company.com:9092
default_topic fortinet_fgt
<format>
@type ltsv
</format>
<buffer>
@type file
path /var/log/fluent/buffer/kafka_buffer
flush_interval 1s
flush_thread_count 4
chunk_limit_size 10m # Set the maximum size of each chunk (e.g., 1MB)
</buffer>
# Add retry settings here
retry_forever true
retry_max_interval 60s

</match>
#</worker>

Describe the logs of Fluentd

No response

Environment

- Fluent Package version: fluent-package 5.0.3 fluentd 1.16.5 (95d130aaa44fc09b14a4e9686ee8018253185fc5)

- Operating system: Debian 12
- Kernel version: SMP PREEMPT_DYNAMIC Debian 6.1.76-1 (2024-02-01) x86_64 GNU/Linux