Multiline is processing lines out of order

[sagargp]

Bug Report

My setup consists of a multi-container docker-compose project that's sending logs to journald. The services are mostly Python. I've got multi-line working mostly fine, but in some cases it seems to break in a way that I can't fully understand. I've pasted an example below.

fluent-bit.conf

[INPUT]
    Name            systemd
    Path            /var/log/journal
    Systemd_Filter  _SYSTEMD_UNIT=docker.service
    Tag             untagged


[FILTER]
    Name                  multiline
    Match                 untagged
    Multiline.key_content MESSAGE
    Multiline.parser      multiline-traceback-regex,python
    Buffer                true

[OUTPUT]
    Name stdout
    Match untagged

parses.conf

[MULTILINE_PARSER]
    name          multiline-traceback-regex
    type          regex
    flush_timeout 5000
    rule      "start_state"   "^\d{4}-"    "cont"
    rule      "cont"          "^(?!\d{4}-)" "cont"

Sample logs below. These are from journalctl CONTAINER_NAME=my_container but the actual logs are being ingested by the systemd input, and are JSON formatted.

2024-05-02 19:46:37,444 INFO [root] [base_daemon.py:26] - Starting up Signals daemon
2024-05-02 19:46:37,452 WARNING [root] [signals.py:64] - Couldn't get a login token from App!
2024-05-02 19:46:37,454 WARNING [root] [signals.py:66] - Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/urllib3/connection.py", line 158, in _new_conn
    conn = connection.create_connection(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/urllib3/util/connection.py", line 57, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/socket.py", line 962, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
socket.gaierror: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/urllib3/connectionpool.py", line 597, in urlopen
    httplib_response = self._make_request(conn, method, url,
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/urllib3/connectionpool.py", line 354, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.11/http/client.py", line 1282, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.11/http/client.py", line 1328, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.11/http/client.py", line 1277, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.11/http/client.py", line 1037, in _send_output
    self.send(msg)
  File "/usr/lib/python3.11/http/client.py", line 975, in send
    self.connect()
  File "/usr/local/lib/python3.11/dist-packages/urllib3/connection.py", line 181, in connect
    conn = self._new_conn()
           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/urllib3/connection.py", line 167, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f32c653f410>: Failed to establish a new connection: [Errno -2] Name or service not known

<snip> (this goes on for quite a while)

2024-05-02 19:46:37,461 INFO [root] [signals.py:395] - Fetched 0 active output connectors
2024-05-02 19:46:37,965 INFO [root] [signals.py:395] - suppressing duplicate messages...

Snippet of output from fluent-bit

[0] untagged: [[1714679197.444539000, {}], {"MESSAGE"=>"2024-05-02 19:46:37,444 INFO [root] [base_daemon.py:26] - Starting up Signals daemon", "CONTAINER_NAME"=>"my_container"}]
[1] untagged: [[1714679197.452401000, {}], {"CONTAINER_NAME"=>"my_container", "MESSAGE"=>"2024-05-02 19:46:37,452 WARNING [root] [signals.py:64] - Couldn't get a login token from App!"}]
[2] untagged: [[1714679197.455503000, {}], {"CONTAINER_NAME"=>"my_container", "MESSAGE"=>"Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/urllib3/connectionpool.py", line 597, in urlopen
    httplib_response = self._make_request(conn, method, url,"}]
[3] untagged: [[1714679197.455503000, {}], {"MESSAGE"=>"  File "/usr/local/lib/python3.11/dist-packages/urllib3/connectionpool.py", line 354, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.11/http/client.py", line 1282, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.11/http/client.py", line 1328, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.11/http/client.py", line 1277, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.11/http/client.py", line 1037, in _send_output
    self.send(msg)
  File "/usr/lib/python3.11/http/client.py", line 975, in send
    self.connect()
  File "/usr/local/lib/python3.11/dist-packages/urllib3/connection.py", line 181, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.11/dist-packages/urllib3/connection.py", line 167, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f32c653f410>: Failed to establish a new connection: [Errno -2] Name or service not known"}]
[4] untagged: [[1714679197.456147000, {}], {"CONTAINER_NAME"=>"my_container", "MESSAGE"=>"Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/requests/adapters.py", line 489, in send
    resp = conn.urlopen("}]

<snip>

As you can see, the traceback starting [2] untagged: doesn't match the log output (compare the subsequent lines):

  File "/usr/local/lib/python3.11/dist-packages/urllib3/connection.py", line 158, in _new_conn

vs.

  File "/usr/local/lib/python3.11/dist-packages/urllib3/connectionpool.py", line 597, in urlopen

What am I doing wrong here?

This is on Fluent Bit v3.0.2 (Git commit: 33ce918351cdae2056b9e1e470ee163293f3fb5d)