Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

I saw "Loki no labels received" in Grafana using Fluent-Bit and Loki. #12843

Open
aiden-91 opened this issue May 1, 2024 · 1 comment
Open

I saw "Loki no labels received" in Grafana using Fluent-Bit and Loki. #12843

aiden-91 opened this issue May 1, 2024 · 1 comment
Labels
component/fluent-bit-plugin type/question

Comments

@aiden-91
Copy link

aiden-91 commented May 1, 2024 •
edited

I'm using Fluent-Bit and Loki and Grafana those versions is under

Fluent-Bit 2.2 Grafana 8.3.4 Loki 3.0.0

I have 2 Virtual Machine (Rockey OS 8)

the first VM is installed Loki and Grafana. The second VM is installed Nginx and Fluent Bit.

I wanna send Nginx access log to Loki using Fluent-Bit and visualize logs in Grafana. But Loki doesn't seem to received any logs from Fluent-Bit of the second VM. so I can't see any labels of Fluent-Bit that I configured in Grafana.

It's not a firewall issue. because when I replaced Fluent-Bit with Promtail, It worked well.

Please check my configuration and let me know that what I made wrong.

This is config of Loki in the first VM.

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  instance_addr: 10.206.21.122
  path_prefix: /tmp/loki
  storage:
    filesystem:
      chunks_directory: /tmp/loki/chunks
      rules_directory: /tmp/loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

query_range:
  results_cache:
    cache:
      embedded_cache:
        enabled: true
        max_size_mb: 100

schema_config:
  configs:
    - from: 2020-10-24
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

ruler:
  alertmanager_url: http://localhost:9093

# By default, Loki will send anonymous, but uniquely-identifiable usage and configuration
# analytics to Grafana Labs. These statistics are sent to https://stats.grafana.org/
#
# Statistics help us better understand how Loki is used, and they show us performance
# levels for most users. This helps us prioritize features and documentation.
# For more information on what's sent, look at
# https://github.com/grafana/loki/blob/main/pkg/analytics/stats.go
# Refer to the buildReport method to see what goes into a report.
#
# If you would like to disable reporting, uncomment the following lines:
#analytics:
#  reporting_enabled: false

This is config of Fluent-Bit in the second VM.

[PARSER]
    Name std_nginx_parser
    Format regex
    Regex ^(?<remote>[^ ]*)(?: (?<x_forwarded_for>[^\[]*))+ \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*) (?<elapsed_time>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
    Time_Key time
    Time_Format %d/%b/%Y:%H:%M:%S %z
    Types code:integer size:integer

[INPUT]
    Name tail
    Path /data01/logs/webLog/ssl_mssgmall/access/*_access.log
    Parser std_nginx_parser
    Tag std_nginx

[OUTPUT]
    name loki
    match *
    host 10.206.21.122
    port 3100
    uri /loki/api/v1/push
    labels job=fluentbit

This is logs of fluent-bit in the second VM.

Fluent Bit v2.2.0
* Copyright (C) 2015-2023 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2024/05/01 21:26:23] [ info] Configuration:
[2024/05/01 21:26:23] [ info]  flush time     | 1.000000 seconds
[2024/05/01 21:26:23] [ info]  grace          | 5 seconds
[2024/05/01 21:26:23] [ info]  daemon         | 0
[2024/05/01 21:26:23] [ info] ___________
[2024/05/01 21:26:23] [ info]  inputs:
[2024/05/01 21:26:23] [ info]      tail
[2024/05/01 21:26:23] [ info] ___________
[2024/05/01 21:26:23] [ info]  filters:
[2024/05/01 21:26:23] [ info] ___________
[2024/05/01 21:26:23] [ info]  outputs:
[2024/05/01 21:26:23] [ info]      loki.0
[2024/05/01 21:26:23] [ info] ___________
[2024/05/01 21:26:23] [ info]  collectors:
[2024/05/01 21:26:23] [ info] [fluent bit] version=2.2.0, commit=, pid=2846609
[2024/05/01 21:26:23] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2024/05/01 21:26:23] [ info] [storage] ver=1.5.1, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/05/01 21:26:23] [ info] [cmetrics] version=0.6.4
[2024/05/01 21:26:23] [ info] [ctraces ] version=0.3.1
[2024/05/01 21:26:23] [ info] [input:tail:tail.0] initializing
[2024/05/01 21:26:23] [ info] [input:tail:tail.0] storage_strategy='memory' (memory only)
[2024/05/01 21:26:23] [debug] [tail:tail.0] created event channels: read=21 write=22
[2024/05/01 21:26:23] [debug] [input:tail:tail.0] flb_tail_fs_inotify_init() initializing inotify tail input
[2024/05/01 21:26:23] [debug] [input:tail:tail.0] inotify watch fd=27
[2024/05/01 21:26:23] [debug] [input:tail:tail.0] scanning path /data01/logs/webLog/ssl_mssgmall/access/*_access.log
[2024/05/01 21:26:23] [debug] [input:tail:tail.0] inode=102994165 with offset=7738 appended as /data01/logs/webLog/ssl_mssgmall/access/2024042921_access.log
[2024/05/01 21:26:23] [debug] [input:tail:tail.0] scan_glob add(): /data01/logs/webLog/ssl_mssgmall/access/2024042921_access.log, inode 102994165
[2024/05/01 21:26:23] [debug] [input:tail:tail.0] inode=103045936 with offset=8471 appended as /data01/logs/webLog/ssl_mssgmall/access/2024042923_access.log
[2024/05/01 21:26:23] [debug] [input:tail:tail.0] scan_glob add(): /data01/logs/webLog/ssl_mssgmall/access/2024042923_access.log, inode 103045936
[2024/05/01 21:26:23] [debug] [input:tail:tail.0] inode=103045945 with offset=2711 appended as /data01/logs/webLog/ssl_mssgmall/access/2024043000_access.log
[2024/05/01 21:26:23] [debug] [input:tail:tail.0] scan_glob add(): /data01/logs/webLog/ssl_mssgmall/access/2024043000_access.log, inode 103045945
[2024/05/01 21:26:23] [debug] [input:tail:tail.0] inode=103045933 with offset=2240 appended as /data01/logs/webLog/ssl_mssgmall/access/2024043001_access.log
[2024/05/01 21:26:23] [debug] [input:tail:tail.0] scan_glob add(): /data01/logs/webLog/ssl_mssgmall/access/2024043001_access.log, inode 103045933
[2024/05/01 21:26:23] [debug] [input:tail:tail.0] inode=103045934 with offset=1788 appended as /data01/logs/webLog/ssl_mssgmall/access/2024043020_access.log
[2024/05/01 21:26:23] [debug] [input:tail:tail.0] scan_glob add(): /data01/logs/webLog/ssl_mssgmall/access/2024043020_access.log, inode 103045934
[2024/05/01 21:26:23] [debug] [input:tail:tail.0] inode=103045950 with offset=3568 appended as /data01/logs/webLog/ssl_mssgmall/access/2024050120_access.log
[2024/05/01 21:26:23] [debug] [input:tail:tail.0] scan_glob add(): /data01/logs/webLog/ssl_mssgmall/access/2024050120_access.log, inode 103045950
[2024/05/01 21:26:23] [debug] [input:tail:tail.0] inode=103181441 with offset=1344 appended as /data01/logs/webLog/ssl_mssgmall/access/2024050121_access.log
[2024/05/01 21:26:23] [debug] [input:tail:tail.0] scan_glob add(): /data01/logs/webLog/ssl_mssgmall/access/2024050121_access.log, inode 103181441
[2024/05/01 21:26:23] [debug] [input:tail:tail.0] 7 new files found on path '/data01/logs/webLog/ssl_mssgmall/access/*_access.log'
[2024/05/01 21:26:23] [debug] [loki:loki.0] created event channels: read=35 write=36
[2024/05/01 21:26:23] [ info] [output:loki:loki.0] configured, hostname=10.206.21.122:3100
[2024/05/01 21:26:23] [ info] [sp] stream processor started
[2024/05/01 21:26:23] [debug] [input:tail:tail.0] inode=102994165 file=/data01/logs/webLog/ssl_mssgmall/access/2024042921_access.log promote to TAIL_EVENT
[2024/05/01 21:26:23] [ info] [input:tail:tail.0] inotify_fs_add(): inode=102994165 watch_fd=1 name=/data01/logs/webLog/ssl_mssgmall/access/2024042921_access.log
[2024/05/01 21:26:23] [debug] [input:tail:tail.0] inode=103045936 file=/data01/logs/webLog/ssl_mssgmall/access/2024042923_access.log promote to TAIL_EVENT
[2024/05/01 21:26:23] [ info] [input:tail:tail.0] inotify_fs_add(): inode=103045936 watch_fd=2 name=/data01/logs/webLog/ssl_mssgmall/access/2024042923_access.log
[2024/05/01 21:26:23] [debug] [input:tail:tail.0] inode=103045945 file=/data01/logs/webLog/ssl_mssgmall/access/2024043000_access.log promote to TAIL_EVENT
[2024/05/01 21:26:23] [ info] [input:tail:tail.0] inotify_fs_add(): inode=103045945 watch_fd=3 name=/data01/logs/webLog/ssl_mssgmall/access/2024043000_access.log
[2024/05/01 21:26:23] [debug] [input:tail:tail.0] inode=103045933 file=/data01/logs/webLog/ssl_mssgmall/access/2024043001_access.log promote to TAIL_EVENT
[2024/05/01 21:26:23] [ info] [input:tail:tail.0] inotify_fs_add(): inode=103045933 watch_fd=4 name=/data01/logs/webLog/ssl_mssgmall/access/2024043001_access.log
[2024/05/01 21:26:23] [debug] [input:tail:tail.0] inode=103045934 file=/data01/logs/webLog/ssl_mssgmall/access/2024043020_access.log promote to TAIL_EVENT
[2024/05/01 21:26:23] [ info] [input:tail:tail.0] inotify_fs_add(): inode=103045934 watch_fd=5 name=/data01/logs/webLog/ssl_mssgmall/access/2024043020_access.log
[2024/05/01 21:26:23] [debug] [input:tail:tail.0] inode=103045950 file=/data01/logs/webLog/ssl_mssgmall/access/2024050120_access.log promote to TAIL_EVENT
[2024/05/01 21:26:23] [ info] [input:tail:tail.0] inotify_fs_add(): inode=103045950 watch_fd=6 name=/data01/logs/webLog/ssl_mssgmall/access/2024050120_access.log
[2024/05/01 21:26:23] [debug] [input:tail:tail.0] inode=103181441 file=/data01/logs/webLog/ssl_mssgmall/access/2024050121_access.log promote to TAIL_EVENT
[2024/05/01 21:26:23] [ info] [input:tail:tail.0] inotify_fs_add(): inode=103181441 watch_fd=7 name=/data01/logs/webLog/ssl_mssgmall/access/2024050121_access.log
[2024/05/01 21:26:23] [debug] [input:tail:tail.0] [static files] processed 0b, done
[2024/05/01 21:26:37] [debug] [input:tail:tail.0] inode=103181441, /data01/logs/webLog/ssl_mssgmall/access/2024050121_access.log, events: IN_MODIFY 
[2024/05/01 21:26:37] [debug] [input chunk] update output instances with new chunk size diff=484, records=1, input=tail.0
[2024/05/01 21:26:37] [trace] [task 0x7fe61e436a00] created (id=0)
[2024/05/01 21:26:37] [debug] [task] created task=0x7fe61e436a00 id=0 OK
[2024/05/01 21:26:37] [trace] [upstream] get new connection for 10.206.21.122:3100, net setup:
net.connect_timeout        = 10 seconds
net.source_address         = any
net.keepalive              = enabled
net.keepalive_idle_timeout = 30 seconds
net.max_worker_connections = 0
[2024/05/01 21:26:37] [trace] [net] connection #47 in process to 10.206.21.122:3100
[2024/05/01 21:26:37] [trace] [engine] resuming coroutine=0x7fe61e40b248
[2024/05/01 21:26:37] [trace] [io] connection OK
[2024/05/01 21:26:37] [debug] [upstream] KA connection #47 to 10.206.21.122:3100 is connected
[2024/05/01 21:26:37] [debug] [http_client] not using http_proxy for header
[2024/05/01 21:26:37] [trace] [io coro=0x7fe61e40b248] [net_write] trying 161 bytes
[2024/05/01 21:26:37] [trace] [io coro=0x7fe61e40b248] [fd 47] write_async(2)=161 (161/161)
[2024/05/01 21:26:37] [trace] [io coro=0x7fe61e40b248] [net_write] ret=161 total=161/161
[2024/05/01 21:26:37] [trace] [io coro=0x7fe61e40b248] [net_write] trying 56 bytes
[2024/05/01 21:26:37] [trace] [io coro=0x7fe61e40b248] [fd 47] write_async(2)=56 (56/56)
[2024/05/01 21:26:37] [trace] [io coro=0x7fe61e40b248] [net_write] ret=56 total=56/56
[2024/05/01 21:26:37] [trace] [io coro=0x7fe61e40b248] [net_read] try up to 4095 bytes
[2024/05/01 21:26:37] [trace] [engine] resuming coroutine=0x7fe61e40b248
[2024/05/01 21:26:37] [trace] [io coro=0x7fe61e40b248] [net_read] ret=64
[2024/05/01 21:26:37] [debug] [output:loki:loki.0] 10.206.21.122:3100, HTTP status=204
[2024/05/01 21:26:37] [debug] [upstream] KA connection #47 to 10.206.21.122:3100 is now available
[2024/05/01 21:26:37] [trace] [engine] [task event] task_id=0 out_id=0 return=OK
[2024/05/01 21:26:37] [debug] [out flush] cb_destroy coro_id=0
[2024/05/01 21:26:37] [trace] [coro] destroy coroutine=0x7fe61e40b248 data=0x7fe61e40b260
[2024/05/01 21:26:37] [debug] [task] destroy task=0x7fe61e436a00 (task_id=0)

image

I think Fluent-Bit sent nginx access logs to Loki well. because I can see "[2024/05/01 21:26:37] [debug] [output:loki:loki.0] 10.206.21.122:3100, HTTP status=204". but I don't know why loki hasn't received any labels.
@JStickler
Copy link
Contributor

Fluent-bit is a community supported client. Configuration questions have a better chance of being answered if you ask them on the community forums.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
component/fluent-bit-plugin type/question
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@JStickler @aiden-91