tracing-journald ignores EMSGSIZE from send() #1698
Comments
Thanks for offering to open a PR to fix this! <3 I think adding a dependency on the |
|
nix is a surprisingly heavyweight, broadly-scoped dependency that is usually avoidable; I would strongly prefer that it not be added, even if that means some unsafe libc, which I think will be fairly limited in scope. IMO one of the big upsides of tracing-journald as it stands is that it is so lightweight. We could also consider logging an error on overflow, but actually passing the data through would be nicer. |
|
Oh, I'm sorry, I wasn't aware that nix is considered heavyweight 🤔 Out of curiosity, and without contesting your words (I am not very familiar with nix after all), may I ask why is that? As far as I can see nix itself has only a few small and trivial dependencies on its own. It is the number of source files and the build time I presume? In any case I'll try and make a pull request with plain libc, but it'll take a bit longer then I'm afraid, for I just noticed that libc doesn't wrap the Also, can I add some integration tests for this crate along the way and have them run on CI? I've written some for libsystemd-rs, which I could mostly just copy; I'd just like to have some safety net when entering unsafe territory 😊 |
please do, if it's not too much trouble! the one concern might be ensuring that appropriate dependencies are present on CI if you need to e.g. use |
|
On review nix isn't as bad as I recalled (1.6s debug, 4.4s release on my machine, not counting the commonplace deps), but it's still orders of magnitude heavier than tracing-journald itself for dubious benefit, unless I'm vastly underestimating the complexity required here. It's also one more thing that has to be kept up to date lest duplicate versions proliferate through everyone's depgraph. Invoking syscalls by hand isn't too difficult, but the barrier to adding new bindings to libc is very low as well. Thanks for working on this! |
|
I think it would be preferable to add a new binding in the |
|
Certainly less work in aggregate but a bit more waiting for them to kick a release out. |
|
Yeah...well, we can always do it ourselves with the intention of removing it once |
|
Oh, this escalated quickly 😔. I can make an implementation without nix for sure, but upstreaming a libc change is an extra mile I was not really prepared for. I'm not sure I'd like to do this, less so since there's already a merge request for this at libc that went nowhere over some Android issues that I'm entirely unfamiliar with. I'd prefer if I could focus my attention on this crate first 🙂 |
|
A raw syscall is fine with me; we can always swap it out. I'll try to nudge libc forward in the mean time. |
|
Thanks 🙏 Just one final question from my side: Would you prefer a separate MR for the integration tests, or an all in one MR with tests and this change? I don't mind either way, so whatever works best for you 🙂 |
I'd prefer separate PRs if that's practical. It would be okay for the EMSGSIZE change to be in a branch that's based on the branch adding integration tests (rather than on the |
|
The libc change at rust-lang/libc#2069 has now been merged. |
|
Cool. |
|
And released in 0.2.107 🎉 |
Per discussion with @hawkw in #1698 I'm adding a few simple integration tests for the journald subscriber, to have some safety net when implementing the actual issue in #1698. These tests send messages of various complexity to the journal, and then use `journalctl`'s JSON output to get them back out, to check whether the message arrives in the systemd journal as it was intended to. ## Motivation Increase test coverage for the journald subscriber and codify a known good state before approaching a fix for #1698.
|
Nice 🙂 Thanks for pushing this. |
|
Turns out though that the real pain is passing the file descriptor over the socket; |
|
cmsgs are wacky for sure, but the special case of sending exactly one message of hardcoded type shouldn't be too bad. https://github.com/quinn-rs/quinn/blob/main/quinn-udp/src/cmsg.rs may be an interesting reference. |
|
I've opened #1744; I'm sorry that it took a bit longer than expected, but I had to fiddle a bit with this cmsg stuff 😇 |
See #1698: Properly write large payloads to journal. I'd appreciate a very careful review; this cmsg stuff is nasty, and while it's well documented in `cmsg(3)` I had to fiddle a bit because the corresponding functions in libc aren't const and thus don't permit a direct allocation of the buffer as most `cmsg` C code around does. Closes #1698 ## Motivation Linux limits the maximum amount of data permitted for a single Unix datagram; sending large payloads directly will fail. ## Solution Follow systemd.io/JOURNAL_NATIVE_PROTOCOL/ and check for `EMSGSIZE` from `send()`; in this case write the payload to a memfd, seal it, and pass it on to journald via a corresponding SCM_RIGHTS control message. Per discussion in #1698 this adds no dependency on `nix`, and instead implements fd forwarding directly with some bits of unsafe `libc` code.
Per discussion with @hawkw in #1698 I'm adding a few simple integration tests for the journald subscriber, to have some safety net when implementing the actual issue in #1698. These tests send messages of various complexity to the journal, and then use `journalctl`'s JSON output to get them back out, to check whether the message arrives in the systemd journal as it was intended to. ## Motivation Increase test coverage for the journald subscriber and codify a known good state before approaching a fix for #1698.
See #1698: Properly write large payloads to journal. I'd appreciate a very careful review; this cmsg stuff is nasty, and while it's well documented in `cmsg(3)` I had to fiddle a bit because the corresponding functions in libc aren't const and thus don't permit a direct allocation of the buffer as most `cmsg` C code around does. Closes #1698 ## Motivation Linux limits the maximum amount of data permitted for a single Unix datagram; sending large payloads directly will fail. ## Solution Follow systemd.io/JOURNAL_NATIVE_PROTOCOL/ and check for `EMSGSIZE` from `send()`; in this case write the payload to a memfd, seal it, and pass it on to journald via a corresponding SCM_RIGHTS control message. Per discussion in #1698 this adds no dependency on `nix`, and instead implements fd forwarding directly with some bits of unsafe `libc` code.
See #1698: Properly write large payloads to journal. I'd appreciate a very careful review; this cmsg stuff is nasty, and while it's well documented in `cmsg(3)` I had to fiddle a bit because the corresponding functions in libc aren't const and thus don't permit a direct allocation of the buffer as most `cmsg` C code around does. Closes #1698 ## Motivation Linux limits the maximum amount of data permitted for a single Unix datagram; sending large payloads directly will fail. ## Solution Follow systemd.io/JOURNAL_NATIVE_PROTOCOL/ and check for `EMSGSIZE` from `send()`; in this case write the payload to a memfd, seal it, and pass it on to journald via a corresponding SCM_RIGHTS control message. Per discussion in #1698 this adds no dependency on `nix`, and instead implements fd forwarding directly with some bits of unsafe `libc` code.
Per discussion with @hawkw in #1698 I'm adding a few simple integration tests for the journald subscriber, to have some safety net when implementing the actual issue in #1698. These tests send messages of various complexity to the journal, and then use `journalctl`'s JSON output to get them back out, to check whether the message arrives in the systemd journal as it was intended to. ## Motivation Increase test coverage for the journald subscriber and codify a known good state before approaching a fix for #1698.
See #1698: Properly write large payloads to journal. I'd appreciate a very careful review; this cmsg stuff is nasty, and while it's well documented in `cmsg(3)` I had to fiddle a bit because the corresponding functions in libc aren't const and thus don't permit a direct allocation of the buffer as most `cmsg` C code around does. Closes #1698 ## Motivation Linux limits the maximum amount of data permitted for a single Unix datagram; sending large payloads directly will fail. ## Solution Follow systemd.io/JOURNAL_NATIVE_PROTOCOL/ and check for `EMSGSIZE` from `send()`; in this case write the payload to a memfd, seal it, and pass it on to journald via a corresponding SCM_RIGHTS control message. Per discussion in #1698 this adds no dependency on `nix`, and instead implements fd forwarding directly with some bits of unsafe `libc` code.
Per discussion with @hawkw in #1698 I'm adding a few simple integration tests for the journald subscriber, to have some safety net when implementing the actual issue in #1698. These tests send messages of various complexity to the journal, and then use `journalctl`'s JSON output to get them back out, to check whether the message arrives in the systemd journal as it was intended to. ## Motivation Increase test coverage for the journald subscriber and codify a known good state before approaching a fix for #1698.
See #1698: Properly write large payloads to journal. I'd appreciate a very careful review; this cmsg stuff is nasty, and while it's well documented in `cmsg(3)` I had to fiddle a bit because the corresponding functions in libc aren't const and thus don't permit a direct allocation of the buffer as most `cmsg` C code around does. Closes #1698 ## Motivation Linux limits the maximum amount of data permitted for a single Unix datagram; sending large payloads directly will fail. ## Solution Follow systemd.io/JOURNAL_NATIVE_PROTOCOL/ and check for `EMSGSIZE` from `send()`; in this case write the payload to a memfd, seal it, and pass it on to journald via a corresponding SCM_RIGHTS control message. Per discussion in #1698 this adds no dependency on `nix`, and instead implements fd forwarding directly with some bits of unsafe `libc` code.
Per discussion with @hawkw in #1698 I'm adding a few simple integration tests for the journald subscriber, to have some safety net when implementing the actual issue in #1698. These tests send messages of various complexity to the journal, and then use `journalctl`'s JSON output to get them back out, to check whether the message arrives in the systemd journal as it was intended to. ## Motivation Increase test coverage for the journald subscriber and codify a known good state before approaching a fix for #1698.
See #1698: Properly write large payloads to journal. I'd appreciate a very careful review; this cmsg stuff is nasty, and while it's well documented in `cmsg(3)` I had to fiddle a bit because the corresponding functions in libc aren't const and thus don't permit a direct allocation of the buffer as most `cmsg` C code around does. Closes #1698 ## Motivation Linux limits the maximum amount of data permitted for a single Unix datagram; sending large payloads directly will fail. ## Solution Follow systemd.io/JOURNAL_NATIVE_PROTOCOL/ and check for `EMSGSIZE` from `send()`; in this case write the payload to a memfd, seal it, and pass it on to journald via a corresponding SCM_RIGHTS control message. Per discussion in #1698 this adds no dependency on `nix`, and instead implements fd forwarding directly with some bits of unsafe `libc` code.
See tokio-rs#1698: Properly write large payloads to journal. I'd appreciate a very careful review; this cmsg stuff is nasty, and while it's well documented in `cmsg(3)` I had to fiddle a bit because the corresponding functions in libc aren't const and thus don't permit a direct allocation of the buffer as most `cmsg` C code around does. Closes tokio-rs#1698 ## Motivation Linux limits the maximum amount of data permitted for a single Unix datagram; sending large payloads directly will fail. ## Solution Follow systemd.io/JOURNAL_NATIVE_PROTOCOL/ and check for `EMSGSIZE` from `send()`; in this case write the payload to a memfd, seal it, and pass it on to journald via a corresponding SCM_RIGHTS control message. Per discussion in tokio-rs#1698 this adds no dependency on `nix`, and instead implements fd forwarding directly with some bits of unsafe `libc` code.
Bug Report
As far as I can see the current implementation of tracing-journal entirely ignores the error returned from
.send():However, according to "Basics" in the systemd protocol documentation socket datagrams are subject to size limits. If a datagram exceeds the message size
socket.sendwill return anEMSGSIZEerror. In this case journal clients should write the payload to a sealed memfd instead and send that memfd to journald.tracing-journald doesn't, so it may silently loose messages. On my system the limit appears to be about 213Kb according to
/proc/sys/net/core/wmem_max; I say "appears" because I'm not entirely sure that this/procfile is definitely relevant. In any case the limit seems to be system-specific so I don't think tracing can generally assume that "reasonably-sized" messages fit into a single datagram. And I don't think tracing should risk loosing messages.I can make a pull request (mostly copying from the working impl in libsystemdrs), but memfds aren't in the standard library and require either a lot of unsafe
libccode or anixdependency, so I'm not sure what the proper course of action is here.Cheers,
Basti
The text was updated successfully, but these errors were encountered: