Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Impossible to ingest timestamp from nested json #8777

Open
tumd opened this issue Apr 30, 2024 · 0 comments
Open

Impossible to ingest timestamp from nested json #8777

tumd opened this issue Apr 30, 2024 · 0 comments

Comments

@tumd
Copy link
Contributor

tumd commented Apr 30, 2024

Describe the bug

Similar to #1013, I try to parse a timestamp field located under a nested key in a json structured log.
My particular logs are from MongoDB. Please see it's documentation for more example logs.

To Reproduce

Example log that should be possible to parse;

{"t":{"$date":"2020-05-01T15:16:17.180+00:00"},"s":"I", "c":"NETWORK", "id":12345, "ctx":"listener", "msg":"Listening on","attr":{"address":"127.0.0.1"}}
[PARSER]
  Name   mongodb-json
  Format json
  Time_Key t.$date
  Time_Format %Y-%m-%dT%H:%M:%S.%L%z
[SERVICE]
  Log_Level debug
  Parsers_File parsers.conf

[INPUT]
  Name tail
  Path mongos.log
  Parser mongodb-json
  Read_from_Head true
  Exit_On_Eof true

[OUTPUT]
  Name   stdout
  Match  *

The problem is; The field containing the timestamp is impossible to parse as such. I have been unable to find any working workaround to get this field parsed.

Please either add support to use Time_Key with nested fields, or suggest a reasonable & working workaround.

I have tried;

  • to set the Parser Time_Key in various different variations to suit the nested key;
    • Time_Key t.$date
    • Time_Key t.\$date
    • Time_Key "t.$date"
    • Time_Key "t.\$date"
    • Time_Key t.$$date
    • ...
  • to use the "Record Accessor" in the parser to reach the timestamp. Like so; Time_Key $t.["$date"].
  • a whole bunch of different nest-filter variations, including lift $date from t without luck.
  • to parse the t key as json with another parser-filter:
[PARSER]
 Name   json
 Format json

[PARSER]
 Name   mongodb-json
 Format json
 Time_Key $date
 Time_Format %Y-%m-%dT%H:%M:%S.%L%z
[INPUT]
  Name tail
  Path mongos.log
  Parser json
  Read_from_Head true
  Exit_On_Eof true

[FILTER]
  Name parser
  Parser mongodb-json
  Key_name t

Initially I figured that the key-name $date containing a $ was the issue, but it seem not to be.
A manually crafted log-entry where the t-key have been flatted out works great with bellow parser;

{"$date":"2020-05-01T15:16:17.180+00:00","s":"I", "c":"NETWORK", "id":12345, "ctx":"listener", "msg":"Listening on","attr":{"address":"127.0.0.1"}}
[PARSER]
  Name   mongodb-json
  Format json
  Time_Key $date
  Time_Format %Y-%m-%dT%H:%M:%S.%L%z

Expected behavior

That the nested t.$date may be possible to use as a timestamp field.

Screenshots

Your Environment

  • Version used: v3.0.3
  • Configuration: See above
  • Environment name and version (e.g. Kubernetes? What version?): Nah
  • Server type and version: GCP e2-medium
  • Operating System and version: Debian GNU/Linux 11 (bullseye)
  • Filters and plugins: Se above

Additional context

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant