Prometheus is crashign with Mimir push errors

[kunalmehta-eve]

Describe the bug

Getting below errors in Prometheus

        e limit, set to 40000 items/s with a maximum allowed burst of 80000.
        This limit is applied on the total number of samples, exemplars and
        metadata received across all distributors
        (err-mimir-tenant-max-ingestion-rate). To adjust the related
        per-tenant limits, configure -distributor.ingestion-rate-limit and
        -distributor.ingestion-burst-size, or contact your service
        administrator."

        ts=2024-05-08T09:20:14.366Z caller=dedupe.go:112 component=remote
        level=error remote_name=00251d url=http://mimir-nginx/api/v1/push
        msg="non-recoverable error" count=2000 exemplarCount=0 err="server
        returned HTTP status 400 Bad Request: failed pushing to ingester:
        user=anonymous: per-user series limit of 990000 exceeded
        (err-mimir-max-series-per-user). To adjust the related per-tenant
        limit, configure -ingester.max-global-series-per-user, or contact
        your service administrator."

        ts=2024-05-08T09:20:52.173Z caller=dedupe.go:112 component=remote
        level=error remote_name=00251d url=http://mimir-nginx/api/v1/push
        msg="non-recoverable error" count=2000 exemplarCount=0 err="server
        returned HTTP status 429 Too Many Requests: the request has been
        rejected because the tenant exceeded the ingestion rate limit, set
        to 40000 items/s with a maximum allowed burst of 80000. This limit
        is applied on the total number of samples, exemplars and metadata
        received across all distributors
        (err-mimir-tenant-max-ingestion-rate). To adjust the related
        per-tenant limits, configure -distributor.ingestion-rate-limit and
        -distributor.ingestion-burst-size, or contact your service
        administrator."

        ts=2024-05-08T09:21:14.411Z caller=dedupe.go:112 component=remote
        level=error remote_name=00251d url=http://mimir-nginx/api/v1/push
        msg="non-recoverable error" count=2000 exemplarCount=0 err="server
        returned HTTP status 400 Bad Request: failed pushing to ingester:
        user=anonymous: per-user series limit of 990000 exceeded
        (err-mimir-max-series-per-user). To adjust the related per-tenant
        limit, configure -ingester.max-global-series-per-user, or contact
        your service administrator."

current Limits:

** "distributor.ingestion-rate-limit": 40000
"distributor.ingestion-burst-size": 80000
ingester.max-global-series-per-user": 990000
**

To Reproduce

Steps to reproduce the behavior:

1. chart version: 4.4.2
2. write

Expected behavior

A clear and concise description of what you expected to happen.

Environment

• Infrastructure: [Kubernetes]
• Deployment tool: [helm]

Additional Context

[kunalmehta-eve]

@jpkrohling

[aknuds1]

I don't see any evidence of a bug, so converting to a discussion.

[56quarters]

The error messages include the information you need to resolve this. Please take a look at the documentation here: https://grafana.com/docs/mimir/latest/manage/mimir-runbooks/