Prometheus remote write desired shards calculation wants to run more than configured max shards.

[kwenzh]

prometheus alert: Prometheus remote write desired shards calculation wants to run more than configured max shards.

and restart prometheus , i can be recovery

max_over_time(prometheus_remote_storage_shards_desired{job="prometheus-k8s",namespace="monitoring"}[5m])

remote config

remote_write:
- url: http://xxxxx.mimir.svc:80/api/v1/push
  remote_timeout: 30s
  follow_redirects: true
  queue_config:
    capacity: 5000
    max_shards: 500
    min_shards: 1
    max_samples_per_send: 100
    batch_send_deadline: 5s
    min_backoff: 30ms
    max_backoff: 100ms
  metadata_config:
    send: true
    send_interval: 1m

[LeviHarrison]

The desired shards metric represents how many shards Prometheus thinks it should run based on calculations, but not necessarily how much it will run because of the configured max/min shards. As can be seen in the following code snippet, the number of shards that are actually run is bounded by the min and max.

prometheus/storage/remote/queue_manager.go

Lines 975 to 987 in 432ea87

	numShards := int(desiredShards)
	// Do not downshard if we are more than ten seconds back.
	if numShards < t.numShards && delay > 10.0 {
	level.Debug(t.logger).Log("msg", "Not downsharding due to being too far behind")
	return t.numShards
	}
	if numShards > t.cfg.MaxShards {
	numShards = t.cfg.MaxShards
	} else if numShards < t.cfg.MinShards {
	numShards = t.cfg.MinShards
	}
	return numShards

I think this distinction is made reasonably clear in the metric name, but I suppose a note could be added in the docs, although I'm not sure if that's necessary.