restart prometheus, server unavailable，loading data long time

[bizy01]

What did you do?
i modify retention time

Environment
k8s cluster

• Prometheus version:

  v2.30.0

• Prometheus configuration file:

- '--web.listen-address=0.0.0.0:{{ apps.geminiStatusMonitor.prometheusPort}}'
 - '--config.file=/etc/prometheus/prometheus.yml'
 - '--query.lookback-delta=30s'
 - '--storage.tsdb.path=/prometheus-data'  
- '--storage.tsdb.retention.time=14d'    

[codesome]

This is because of WAL replay which can be slow depending on how many active series you have.

If you can afford some additional disk space, could you please upgrade to v2.30.3 and enable memory snapshots on shutdown via --enable-feature=memory-snapshot-on-shutdown and see if it helps? Any graceful restarts after enabling snapshots should bring down the restart time.