Unable to configure logging

[abgarg2022]

What did you do?

Set up /etc/prometheus/prometheus.yml and configured /etc/systemd/system/prometheus.service

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Restart=always
Type=simple
ExecStart=/usr/local/bin/prometheus
--config.file=/etc/prometheus/prometheus.yml
--storage.tsdb.path=/var/lib/prometheus/
--storage.tsdb.retention.time=7d
--web.console.templates=/etc/prometheus/consoles
--web.console.libraries=/etc/prometheus/console_libraries
--web.listen-address=0.0.0.0:8095
--log.level=debug \

[Install]
WantedBy=multi-user.target

What did you expect to see?

Want the Debug/Error/Info logs to be saved to separate files.

What did you see instead? Under which circumstances?

As of now I can see all logs going to /var/log/syslog

System information

Linux 5.15.0-1031-aws x86_64

Prometheus version

prometheus, version 2.27.1 (branch: HEAD, revision: db7f0bcec27bd8aeebad6b08ac849516efa9ae02)
  build user:       root@fd804fbd4f25
  build date:       20210518-14:17:54
  go version:       go1.16.4
  platform:         linux/amd64

Prometheus configuration file

root@ip-X.Y.A.B# cat /etc/prometheus/prometheus.yml
# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets: ['X.Y.A.B:3000']
      # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  #- job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    #    static_configs:
    #- targets: ['localhost:9090']
  - job_name: 'postgres_exporter'
    scrape_interval: 5s
    metrics_path: '/pgmetrics'
    static_configs:
      - targets: ['X.Y.A.B:9113']

root@ip-X.Y.A.B:/etc/systemd/system#

Alertmanager version

No response

Alertmanager configuration file

No response

Logs

May 19 11:36:56 ip-172-31-19-162 prometheus[186698]: level=info ts=2023-05-19T11:36:56.882Z caller=main.go:706 msg="Scrape manager stopped"
May 19 11:36:56 ip-172-31-19-162 prometheus[186698]: level=info ts=2023-05-19T11:36:56.889Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..."
May 19 11:36:56 ip-172-31-19-162 prometheus[186698]: level=info ts=2023-05-19T11:36:56.889Z caller=main.go:885 msg="Notifier manager stopped"
May 19 11:36:56 ip-172-31-19-162 prometheus[186698]: level=info ts=2023-05-19T11:36:56.890Z caller=main.go:897 msg="See you next time!"
May 19 11:36:56 ip-172-31-19-162 systemd[1]: prometheus.service: Deactivated successfully.
May 19 11:36:56 ip-172-31-19-162 systemd[1]: prometheus.service: Consumed 1.043s CPU time.
May 19 11:36:56 ip-172-31-19-162 prometheus[186869]: Error parsing commandline arguments: unknown long flag '--log.fmt'
May 19 11:36:56 ip-172-31-19-162 prometheus[186869]: prometheus: error: unknown long flag '--log.fmt'
May 19 11:36:56 ip-172-31-19-162 systemd[1]: prometheus.service: Main process exited, code=exited, status=1/FAILURE
May 19 11:36:56 ip-172-31-19-162 systemd[1]: prometheus.service: Failed with result 'exit-code'.
May 19 11:36:57 ip-172-31-19-162 systemd[1]: prometheus.service: Scheduled restart job, restart counter is at 1.
May 19 11:36:57 ip-172-31-19-162 prometheus[186874]: Error parsing commandline arguments: unknown long flag '--log.fmt'
May 19 11:36:57 ip-172-31-19-162 prometheus[186874]: prometheus: error: unknown long flag '--log.fmt'
May 19 11:36:57 ip-172-31-19-162 systemd[1]: prometheus.service: Main process exited, code=exited, status=1/FAILURE
May 19 11:36:57 ip-172-31-19-162 systemd[1]: prometheus.service: Failed with result 'exit-code'.
May 19 11:36:57 ip-172-31-19-162 systemd[1]: prometheus.service: Scheduled restart job, restart counter is at 2.

[roidelapluie]

Prometheus logs everything into one stream, you are correct. In general, it's best to have the context of a log message and I don't think splitting them into multiple streams would be beneficial.

[abgarg2022]

Prometheus logs everything into one stream, you are correct. In general, it's best to have the context of a log message and I don't think splitting them into multiple streams would be beneficial.

Hi Julien,
Since all the logs get clubbed into syslog, so for our Support team it becomes difficult . And they have raised a request where prometheus logs can be separatedly logged into any other file?

[abgarg2022]

Prometheus logs everything into one stream, you are correct. In general, it's best to have the context of a log message and I don't think splitting them into multiple streams would be beneficial.

Hi Julien, Since all the logs get clubbed into syslog, so for our Support team it becomes difficult . And they have raised a request where prometheus logs can be separatedly logged into any other file?

@roidelapluie
Any updates on this?

[roidelapluie]

In general you could do the splitting with a wrapper script:

 ./prometheus --log.level=debug 2>&1 | tee >(cat|grep -v level=error) >(cat|grep level=error &>/dev/stderr)