[Journald] input crashes with "failed to read message field: cannot allocate memory"

[belimawr]

Filebat: 8.13.2
Host OS: Amazon Linux 2
Systemd/Journald version: systemd 252 (252.16-1.amzn2023.0.2)

journalctl --version
systemd 252 (252.16-1.amzn2023.0.2)
+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP -GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 -BZIP2 -LZ4 +XZ -ZLIB -ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified

##How to reproduce

1. Flood jounrald with logs so it it rotates logs every minute or so. Mostly follow [Filebeat] Journald causes Filebeat to crash #34077 (comment)
2. Start Filebeat with the config from the above link
3. Wait until Journald reaches its maximum number of files and starts deleting old entries
4. Filebeat might crash due to [Filebeat] Journald causes Filebeat to crash #34077, it's ok. Ignore it
5. Let the logs flowing for a while (I waited for hours)
6. Start Filebeat again
7. Journald input will fail with:

   {"log.level":"error","@timestamp":"2024-05-01T19:29:01.010Z","log.logger":"input.journald","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/v2/compat.(*runner).Start.func1","file.name":"compat/compat.go","file.line":132},"message":"Input 'journald' failed with: input.go:130: input journald-input failed (id=journald-input)\n\tfailed to read message field: cannot allocate memory","service.name":"filebeat","id":"journald-input","ecs.version":"1.6.0"}
   

Sometimes Filebeat might just crash again. I also saw it failing once or twice with the same message as in #32782.

Both seem to be relates with Filebeat being too far behind reading the journal, probably further behind than what journald has got stored in disk.

On both cases the error is coming from the Journald library we use, github.com/coreos/go-systemd/v22

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)