Sticky logs in the same output message

[sergeye-boltinc]

Bug Report

Describe the bug
We're trying to use the Fluent-Bit to ship our logs form Kubernetes to Splunk.
The Splunk output plugin sends several logs combined into one message.
We tried to use HTTP output plugin instead, but we see the same behavior.
At the same time the regular stdout plugin shows the logs correctly separated.
The interesting part is that if we put Splunk_Send_Raw Off, the logs are coming as expected.

To Reproduce
[SERVICE]
Daemon Off
Flush {{ .Values.flush }}
Log_Level {{ .Values.logLevel }}
Parsers_File /fluent-bit/etc/parsers.conf
Parsers_File /fluent-bit/etc/conf/custom_parsers.conf
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_Port {{ .Values.metricsPort }}
Health_Check On

[INPUT]
    Name tail
    Path /var/log/containers/*.log
    Read_from_Head True
    DB /var/log/fluentbit-containers.log.pos.db
    Tag kubernetes.*
    Buffer_Max_Size 50MB
    Mem_Buf_Limit 50MB
    Refresh_Interval  10

[FILTER]
    Name    multiline
    Match   *
    multiline.key_content log
    multiline.parser      cri-stdout, cri-stderr

[FILTER]
    Name    parser
    Match   *
    key_name log
    Parser cri-custom

[FILTER]
    Name                kubernetes
    Match               kubernetes.*
    Kube_Tag_Prefix     kubernetes.var.log.containers
    Merge_Log           On
    Keep_Log            Off
    K8S-Logging.Parser  On
    K8S-Logging.Exclude Off
    Buffer_Size         50MB
    Labels              Off
    Annotations         Off

[FILTER]
    Name                modify
    Match               kubernetes.*
    Condition Key_Does_Not_Exist source
    set source kubernetes-rke2-dev

[FILTER]
    Name                modify
    Match               kubernetes.*
    Condition Key_Does_Not_Exist sourcetype
    set sourcetype iisadv

[FILTER]
    Name                modify
    Match               kubernetes.*
    add host ${HOSTNAME}

[OUTPUT]
    Name stdout
    Match *

[OUTPUT]
    Name splunk
    Match      kubernetes.*
    host       my-splunk
    port       8088
    Retry_Limit 3
    Splunk_Send_Raw On
    splunk_token xxxxx-xxxx-xxxx-xxxxxxxx
    tls true
    tls.verify false

[PARSER]
    Name cri-custom
    Format regex
    Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr)( (.))? (?<log>.*)$
    Time_Key    time
    Time_Format %Y-%m-%dT%H:%M:%S.%L%z
    Time_Keep   On

[MULTILINE_PARSER]
    name cri-stdout
    type regex
    flush_timeout 1000
    rule "start_state" "/^(?<time>[^ ]+) (?<stream>stdout)( (.))? (?<message>.*)\}$/" "start_state"

[MULTILINE_PARSER]
    name cri-stderr
    type regex
    flush_timeout 1000
    rule "start_state" "/^(?<time>[^ ]+) (?<stream>stderr)( (.))? (?<message>.*)$/" "start_state"

• Example log message if applicable:

These are 3 separated logs we can see in stdout:
[0] kubernetes.var.log.containers.redis-ha-natgen-server-2_redis_split-brain-fix-f08b732e310061863286aed9bcf49503c39473397332774141be8b1732ca6032.log: [[1714408170.710990289, {}], {"time"=>"2024-04-29T16:29:30.710990289Z", "stream"=>"stdout", "log"=>"Identifying redis master (get-master-addr-by-name)..", "kubernetes"=>{"pod_name"=>"redis-ha-server-2", "namespace_name"=>"redis", "pod_id"=>"153d6b6a-a156-49e1-8a7d-18e632106e93", "host"=>"rke2-dev-worker1-02712b6a-grl6d", "container_name"=>"split-brain-fix", "docker_id"=>"f08b732e310061863286aed9bcf49503c39473397332774141be8b1732ca6032", "container_hash"=>"docker.io/library/redis@sha256:7635b0bfdd7dd8552b4b31d6541fef07b734614045b45a52fd5cc27c9dada9e2", "container_image"=>"docker.io/library/redis:7.2.4-alpine"}, "source"=>"kubernetes-rke2-dev", "sourcetype"=>"iisadv", "host"=>"fluent-bit-q6skm"}]
[1] kubernetes.var.log.containers.redis-ha-server-2_redis_split-brain-fix-f08b732e310061863286aed9bcf49503c39473397332774141be8b1732ca6032.log: [[1714408170.711020650, {}], {"time"=>"2024-04-29T16:29:30.71102065Z", "stream"=>"stdout", "log"=>"  using sentinel (redis-ha), sentinel group name (redis)", "kubernetes"=>{"pod_name"=>"redis-ha-server-2", "namespace_name"=>"redis", "pod_id"=>"153d6b6a-a156-49e1-8a7d-18e632106e93", "host"=>"rke2-dev-worker1-02712b6a-grl6d", "container_name"=>"split-brain-fix", "docker_id"=>"f08b732e310061863286aed9bcf49503c39473397332774141be8b1732ca6032", "container_hash"=>"docker.io/library/redis@sha256:7635b0bfdd7dd8552b4b31d6541fef07b734614045b45a52fd5cc27c9dada9e2", "container_image"=>"docker.io/library/redis:7.2.4-alpine"}, "source"=>"kubernetes-rke2-dev", "sourcetype"=>"iisadv", "host"=>"fluent-bit-q6skm"}]
[2] kubernetes.var.log.containers.redis-ha-server-2_redis_split-brain-fix-f08b732e310061863286aed9bcf49503c39473397332774141be8b1732ca6032.log: [[1714408170.720546886, {}], {"time"=>"2024-04-29T16:29:30.720546886Z", "stream"=>"stdout", "log"=>"  Mon Apr 29 16:29:30 UTC 2024 Found redis master (10.43.155.109)", "kubernetes"=>{"pod_name"=>"redis-ha-server-2", "namespace_name"=>"redis", "pod_id"=>"153d6b6a-a156-49e1-8a7d-18e632106e93", "host"=>"rke2-dev-worker1-02712b6a-grl6d", "container_name"=>"split-brain-fix", "docker_id"=>"f08b732e310061863286aed9bcf49503c39473397332774141be8b1732ca6032", "container_hash"=>"docker.io/library/redis@sha256:7635b0bfdd7dd8552b4b31d6541fef07b734614045b45a52fd5cc27c9dada9e2", "container_image"=>"docker.io/library/redis:7.2.4-alpine"}, "source"=>"kubernetes-rke2-dev", "sourcetype"=>"iisadv", "host"=>"fluent-bit-q6skm"}]


And  this way, they are coming to Splunk or HTTP output:
[{"time":"2024-04-29T16:14:30.463600411Z","stream":"stdout","log":"Identifying redis master (get-master-addr-by-name)..","kubernetes":{"pod_name":"redis-ha-server-2","namespace_name":"redis","pod_id":"153d6b6a-a156-49e1-8a7d-18e632106e93","host":"rke2-dev-worker1-02
712b6a-grl6d","container_name":"split-brain-fix","docker_id":"f08b732e310061863286aed9bcf49503c39473397332774141be8b1732ca6032","container_hash":"docker.io/library/redis@sha256:7635b0bfdd7dd8552b4b31d6541fef07b734614045b45a52fd5cc27c9dada9e2","container_image":"docker.io/l
ibrary/redis:7.2.4-alpine"},"source":"kubernetes-rke2-dev","sourcetype":"iisadv","host":"fluent-bit-q6skm"},{"time":"2024-04-29T16:14:30.463644183Z","stream":"stdout","log":"  using sentinel (redis-ha), sentinel group name (redis)","kubernetes":{"pod_name":"r
edis-ha-server-2","namespace_name":"redis","pod_id":"153d6b6a-a156-49e1-8a7d-18e632106e93","host":"rke2-dev-worker1-02712b6a-grl6d","container_name":"split-brain-fix","docker_id":"f08b732e310061863286aed9bcf49503c39473397332774141be8b1732ca6032","container_hash":"do
cker.io/library/redis@sha256:7635b0bfdd7dd8552b4b31d6541fef07b734614045b45a52fd5cc27c9dada9e2","container_image":"docker.io/library/redis:7.2.4-alpine"},"source":"kubernetes-rke2-dev","sourcetype":"iisadv","host":"fluent-bit-q6skm"},{"time":"2024-04-29T16:14:30.479640611Z"
,"stream":"stdout","log":"  Mon Apr 29 16:14:30 UTC 2024 Found redis master (10.43.155.109)","kubernetes":{"pod_name":"redis-ha-server-2","namespace_name":"redis","pod_id":"153d6b6a-a156-49e1-8a7d-18e632106e93","host":"rke2-dev-worker1-02712b6a-grl6d","container_nam
e":"split-brain-fix","docker_id":"f08b732e310061863286aed9bcf49503c39473397332774141be8b1732ca6032","container_hash":"docker.io/library/redis@sha256:7635b0bfdd7dd8552b4b31d6541fef07b734614045b45a52fd5cc27c9dada9e2","container_image":"docker.io/library/redis:7.2.4-alpine"},
"source":"kubernetes-rke2-dev","sourcetype":"iisadv","host":"fluent-bit-q6skm"}]

• Steps to reproduce the problem:

Expected behavior
every single log should arrive as a separate message

Your Environment
Rancher RKE2 based on ubuntu 20.04 linux.
Kubernetes v1.26.8
Fluent-Bit helmachart version 0.46.2
Fluent-Bit image 3.0.3