out_loki: allow sending unquoted strings #8814
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This patch adds a third value to `drop_single_key` - `raw`, which allows sending unquoted strings to Loki when using JSON as the `line_format`. While yes, for the output to be valid JSON, quotes would be expected, Loki does not support reading a plain quoted string with its JSON parser, complaining that it cannot find a `}` character. Instead, you need to use a combination of regexp and line_format expressions to unquote the log before running any other parsers over it. By adding a third value of `raw`, this ensures backwards compatibility for anyone that is already relying on the existing behaviour. Signed-off-by: Andrew Titmuss <iandrewt@icloud.com>
|
I'm undecided on whether |
iandrewt
requested review from
edsiper,
leonardo-albertovich,
fujimotos and
koleini
as code owners
|
Can you link the docs PR once you have it too? |
iandrewt
added a commit
to iandrewt/fluent-bit-docs
that referenced
this pull request
May 10, 2024
Relates to fluent/fluent-bit#8814 Signed-off-by: Andrew Titmuss <iandrewt@icloud.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This patch adds a third value to
drop_single_key-raw, which allows sending unquoted strings to Loki when using JSON as theline_format.While yes, for the output to be valid JSON, quotes would be expected, Loki does not support reading a plain quoted string with its JSON parser, complaining that it cannot find a
}character. Instead, you need to use a combination of regexp and line_format expressions to unquote the log before running any other parsers over it.By adding a third value of
raw, this ensures backwards compatibility for anyone that is already relying on the existing behaviour.An example query before this change for plaintext logs, to remove the quotes:
This is necessary to add before any other parsing for non-JSON logs, such as using Loki's
logfmtparser, as it will otherwise drop the first and last key/value pair due to the extraneous quotes. Given Loki by design does not care about the input format (there's a reason JSON parsing is optional in the first place!), I do think this should be the default behavior some day, but given the breaking nature of the change, having it as an option for now should be fine.Addresses #4353, #3005
Testing
Before we can approve your change; please submit the following in a comment:
Note that I sent this to a fake http server instead of a Loki instance for testing, hence the 200 response. The HTTP body my server received was this:
{"streams":[{"stream":{"job":"fluent-bit"},"values":[["1715344552991602966","value"]]}]}If this is a change to packaging of containers or native binaries then please confirm it works for all targets.
ok-package-testlabel to test for all targets (requires maintainer to do).Documentation
fluent/fluent-bit-docs#1368
Backporting
Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.