Prometheus WAL files are not deleted and its size is 261GB now

[praveen5kgit]

Prometheus WAL files are not deleted since the Prometheus setup was done. I am using Prometheus version 2.26.0 and I have gone through the same issue in forums. It has been mentioned that it was fixed in version 2.11 but for me, the WAL files are not deleting even though I use version 2.26.0. In the prometheus.yml file, I have not set --storage.tsdb.retention.time and --storage.tsdb.wal-compression parameters. Because of these huge WAL files, whenever I restart Prometheus, it takes more than 2 hours to load all the segments. Please help me to find out if I am missing anything or it is a bug. Currently, the size of the WAL folder is 261GB. I have verified this below expression in Prometheus URL
(time() - prometheus_tsdb_lowest_timestamp_seconds) / 86400

Output:

{instance="xxx.xx.xx:9090", job="prometheus"} 15.376041666666667

*** Prometheus version:**

prometheus, version 2.26.0 (branch: HEAD, revision: 3cafc58)
build user: root@a67cafebe6d0
build date: 20210331-11:56:23
go version: go1.16.2
platform: linux/amd64

*** Prometheus configuration file:**

Below is my prometheus.yml file. I have removed targets for security reasons. We have 50 plus targets configured.

# my global config
global:
  scrape_interval:     60s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 60s # Evaluate rules every 15 seconds. The default is every 1 minute.
  query_log_file: /etc/prometheus/query.log
  # scrape_timeout is set to the global default (10s).


# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
       -  xx.xxx.xx:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  - /etc/prometheus/applicationrules/*.yml
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'
    static_configs:
    - targets: [' xx.xxx.xx:9090',' xx.xxx.xx:9115']

  - job_name: 'redis'
    scheme: https
    tls_config:
      insecure_skip_verify: true
    static_configs:
  

  - job_name: 'rabbitmq'
    static_configs:
  

  - job_name: 'elasticsearch'
    static_configs:
   
  - job_name: 'postgresql'
    static_configs:
 

  - job_name: 'kafka'
    static_configs:
  
  - job_name: 'zookeeper'
    static_configs:
  

  - job_name: 'jboss'
    static_configs:
 

  - job_name: 'blackbox'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
    
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: xx.xxx.xx:9115  # The blackbox exporter's real hostname:port.

  - job_name: 'blackboxtcp'
    metrics_path: /probe
    params:
      module: [tcp_connect]
    static_configs:
      - targets:
        
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement:  xx.xxx.xx:9115  # The blackbox exporter's real hostname:port.

[praveen5kgit]

[roidelapluie]

Can you provide us the starting logs of prometheus? If possible with --log.level=debug.

[praveen5kgit]

startup_logs.txt

I have attached starting logsof prometheus. WAL files are available with the date from July 2021 which starts from 00001476. I have attached screenshot of checkpoint as well here.

Please check and let me know

[praveen5kgit]

latest_logs.txt

I have attached latest logs of prometheus here.

[praveen5kgit]

@roidelapluie Did you get a chance to look at the logs? Pls help.

[praveen5kgit]

@codesome Can you please help me on this issue? This issue is occurring in production. So wanted to fix it sooner.

[codesome]

Log has WAL truncation in Compact: create checkpoint: create segment reader: open segment:00002392 in dir:/data/prometheus/wal: open /data/prometheus/wal/00002392: too many open files

too many open files

You might have some lower limit set at some level. A quick search gives me https://www.baeldung.com/linux/error-too-many-open-files/. Can you try increasing the limit to some high value and see if that fixes it?

[praveen5kgit]

@codesome Yes, i saw this error and I have increased ulimit from 1024 to 65536 for root user as we are running prometheus with root user. But after this change when I tried to restart prometheus it was taking more than 7 hours to load all the wal segments and it was not starting. Usually it will take 2 hours to load all the wal segments which are present under /data/prometheus/wal and start prometheus. But after ulimit changes service did not started. So we have reverted that ulimit value to 1024 and then started prometheus service. So if we resolve this wal segments truncation issue, then it will delete old wal segments and prometheus will be started quickly. Am I right?

[codesome]

Since you have piled up so much of WAL, it is going to take a ton of time yes. You can delete some % of older files to speed this up, and if you are lucky, there will be no data loss, but expect some data loss nevertheless.

I would not recommend reducing back the ulimit because it all started with the lower ulimit from the looks of it.

[praveen5kgit]

@codesome We have stopped prometheus service and moved some of the wal segments like dated from july2021 to Oct2021 to backup folder. Then tried to start the prometheus service. But it did not start and gave exit code 1 error. Can you pls give me the steps to delete old wal segments and start prometheus?
And one more question, as I pasted in one of the screenshot checkpoint value shows as checkpoint.00001475 . is that correct? or it should be updated based on the latest wal segment? why old wal files are not deleting? Will it give any issue if I delete the segment 00001475 manually? Because this 00001475 segment was the very old one.

[praveen5kgit]

@codesome Can you please help me?

[codesome]

But it did not start and gave exit code 1 error

Do you have the logs that I can see?

wal segments like dated from july2021 to Oct2021

That is like too old. It makes sense to keep the WAL only for last 4-6hrs max. If blocks have already been cut (you can check the meta.json in the last block to know the time range of the block), you can delete the WAL upto the last day without issues.

And you should not change the names of the files manually. It is an old number for checkpoint because new checkpoint creation had failed according to your log message.

[vini-intenseye]

@codesome is there a possibility to limit the WAL to 4-6 hours? I have the following as the flags

--storage.tsdb.retention | 0s
-- | --
--storage.tsdb.retention.size | 0B
--storage.tsdb.retention.time | 1w
--storage.tsdb.samples-per-chunk | 120
--storage.tsdb.wal-compression | true
--storage.tsdb.wal-compression-type | snappy
--storage.tsdb.wal-segment-size | 0B

I have an evergrowing disk, and a WAL size of 600GB with data retained from 19 days. I manually delete the wal folder to startup the prometheus. Is there anything else i could chck? I m using prom 2.46 (I will perform an upgrade as well today to see if it would get better)
Thanks

[praveen5kgit]

@codesome Sorry for the delayed response. I have set LimitNOFILE=65536 in prometheus.service file and moved all WAL files to the backup folder. Then started Prometheus and not is running fine. So now prometheus is taking 65536 value and also WAL truncation is happening properly.
Thanks for your inputs.

[macsmarte]

Vou escrever em português pois sou do Brasil, Estou tendo um problema, tem como eu reduzir o tamanho wal de 1 em 1 hora? de forma automatizada com uma flag?
A minha pergunta vem porque chega uma altura que ele consome toda RAM do meu serviço, cerca de 7GB e aí ele começa a reiniciar.
estou usando a v2.51.2 do prometheus.
Estou usando AWS ECS Fargate , só falta isto para o meu serviço ficar altamente estável.
Obrigado .