Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fluent-bit 3.0.3 upgrade has broken splunk output plugin when event_sourcetype_key is specified #8781

Closed
mannbiher opened this issue May 1, 2024 · 4 comments · Fixed by #8793
Closed

fluent-bit 3.0.3 upgrade has broken splunk output plugin when event_sourcetype_key is specified #8781

mannbiher opened this issue May 1, 2024 · 4 comments · Fixed by #8793
Labels
status: waiting-for-triage

Comments

@mannbiher
Copy link
Contributor

mannbiher commented May 1, 2024 •
edited

Bug Report

Describe the bug
fluent-bit 3.0.3 upgrade has broken Splunk output plugin when event_sourcetype_key attribute is specified. It sends a wrong Authorization header to Splunk causing it to respond with an HTTP 401 error. It now sends Authorization: hec_token which should be Authorization: Splunk db496524-e7e6-4ae9-b3f0-2287d8e65cd4 where db496524-e7e6-4ae9-b3f0-2287d8e65cd4 is a HEC token.

[2024/05/01 23:11:43] [ warn] [output:splunk:splunk.0] http_status=401:
{"text":"Invalid authorization","code":3}

To Reproduce

  • Steps to reproduce the problem:
    On Linux
  1. Create sample log.
cat <<EOF > sample.log
log line 1
log line 2
log line 3
EOF
  1. Run nc in a different terminal as dummy Splunk server.
echo -e 'HTTP/1.1 200 OK\r\n' | nc -l -p 8081
  1. Start fluent-bit 3.0.3 and wait for [http_client] broken connection to 127.0.0.1:8081 ? error. As we are using nc the broken connection is expected.
docker run -ti -v $PWD/sample.log:/sample.log \
--network="host" fluent/fluent-bit:3.0.3 /fluent-bit/bin/fluent-bit \
-i tail -p path=/sample.log -p read_from_head=true \
-o splunk -p port=8081 -psplunk_token=db496524-e7e6-4ae9-b3f0-2287d8e65cd4 -p 'event_sourcetype_key=sourcetype' \
-f 1
  1. View the request sent to nc.
POST /services/collector/event HTTP/1.1
Host: 127.0.0.1:8081
Content-Length: 242
User-Agent: Fluent-Bit
Authorization: hec_token
Connection: keep-alive

{"time":1714604689.81654,"sourcetype":"sourcetype","event":{"log":"log line 1"}}{"time":1714604689.816549,"sourcetype":"sourcetype","event":{"log":"log line 2"}}{"time":1714604689.816551,"sourcetype":"sourcetype","event":{"log":"log line 3"}}

Expected behavior

Running the same steps with fluent-bit 3.0.2 docker image sends correct authorization header.

docker run -ti -v $PWD/sample.log:/sample.log \
--network="host" fluent/fluent-bit:3.0.2 /fluent-bit/bin/fluent-bit \
-i tail -p path=/sample.log -p read_from_head=true \
-o splunk -p port=8081 -psplunk_token=db496524-e7e6-4ae9-b3f0-2287d8e65cd4 -p 'event_sourcetype_key=sourcetype' \
-f 1
POST /services/collector/event HTTP/1.1
Host: 127.0.0.1:8081
Content-Length: 243
User-Agent: Fluent-Bit
Authorization: Splunk db496524-e7e6-4ae9-b3f0-2287d8e65cd4
Connection: keep-alive

{"time":1714604726.505695,"sourcetype":"sourcetype","event":{"log":"log line 1"}}{"time":1714604726.505703,"sourcetype":"sourcetype","event":{"log":"log line 2"}}{"time":1714604726.505704,"sourcetype":"sourcetype","event":{"log":"log line 3"}}

Your Environment

  • Version used: 3.0.3
  • Configuration:
[INPUT]
    Name             tail
    Path             /sample.log
    Read_From_Head   On
[OUTPUT]
    Name                 splunk
    Match                *
    Port                 8081
    Splunk_Token         db496524-e7e6-4ae9-b3f0-2287d8e65cd4
    event_sourcetype_key sourcetype
  • Environment name and version (e.g. Kubernetes? What version?): EKS 1.27
  • Server type and version: NA
  • Operating System and version: Amazon Linux 2023
  • Filters and plugins: tail, splunk
@nirnanaaa
Copy link

Facing a very similar issue here. After some digging I found #8738 which might have caused this issue. Although I am not sure how to use the newly introduced event format correctly.

@mannbiher mannbiher mentioned this issue May 3, 2024
Merged
3 tasks
@mannbiher
Copy link
Contributor Author

Hi @nirnanaaa I have created a PR to fix the issue. Would you be able to test the PR for your scenario. All inputs are welcome.

@mannbiher mannbiher changed the title fluent-bit 3.0.3 upgrade has broken splunk output plugin fluent-bit 3.0.3 upgrade has broken splunk output plugin when event_sourcetype_key is specified May 4, 2024
@neugeeug
Copy link

neugeeug commented May 7, 2024

We have the same issue, Splunk integration stopped working since 3.0.3

@agup006
Copy link
Member

agup006 commented May 10, 2024

@cosmo0920 fyi looks like an edge case on hec token

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
status: waiting-for-triage
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: splunk output plugin correct record accessor key for hec_token mannbiher/fluent-bit
4 participants
@nirnanaaa @agup006 @mannbiher @neugeeug