Discrepancy between Mimir and the actual data

[girishms-sentient]

The Mimir is configured and running, but the data I'm seeing in the Mimir Grafana dashboard does not match the actual Grafana for the cluster. Sometimes the displayed data is twice/thrice as accurate.

Actual Grafana dashboard:

Mimir Grafana:

Here's the mimir config:

        ################ MIMIR CONFIGURATION #####################
        mimir:
          structuredConfig:
            ingester:
              ring:
                final_sleep: 0s
                num_tokens: 512
                tokens_file_path: /data/tokens
                heartbeat_period: 20s
                heartbeat_timeout: 60s
                unregister_on_shutdown: true
                kvstore:
                  store: memberlist
                replication_factor: 3
                zone_awareness_enabled: true

            memberlist:
              abort_if_cluster_join_fails: false
              compression_enabled: false
              join_members:
                - dns+{{ include "mimir.fullname" . }}-gossip-ring.{{ .Release.Namespace }}.svc.{{ .Values.global.clusterDomain }}:{{ include "mimir.memberlistBindPort" . }}
              advertise_addr: ${MY_POD_IP}

            limits:
              compactor_blocks_retention_period: 604800s
              ingestion_rate: 500000
              max_global_series_per_metric: 9000000
              max_global_series_per_user: 9000000
              max_label_names_per_series: 60

            distributor:
              ring:
                kvstore:
                  store: memberlist

            compactor:
              data_dir: /data/compactor
              sharding_ring:
                heartbeat_period: 20s
                heartbeat_timeout: 60s  
                kvstore:
                  store: memberlist

            store_gateway:
              sharding_ring:
                heartbeat_period: 20s
                heartbeat_timeout: 60s
                zone_awareness_enabled: true 
                kvstore:
                  store: memberlist  

            ruler:
              rule_path: /data/ruler
              poll_interval: 2s
              ring:
                heartbeat_period: 20s
                heartbeat_timeout: 60s
                kvstore:
                  store: memberlist
          
          ############## MIMIR STORAGE ################
            blocks_storage:
              backend: s3
              bucket_store:
                max_chunk_pool_bytes: 12884901888 # 12GiB
              s3:
                endpoint: s3.us-west-2.amazonaws.com
                bucket_name: central-logging-mimir-bucket-block-storage        
                insecure: true
              tsdb:
                dir: /data/tsdb


            alertmanager_storage:
              backend: s3
              s3:
                endpoint: s3.us-west-2.amazonaws.com
                bucket_name: central-logging-mimir-bucket-alertmanager-storage 


            ruler_storage:
              backend: s3
              s3:
                endpoint: s3.us-west-2.amazonaws.com
                bucket_name: central-logging-mimir-bucket-ruler-storage         

        #############################################################

        global:
          extraEnv:
            - name: MY_POD_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP

        serviceAccount:
          create: true
          name: mimir
          annotations:
            eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxxxxxxx:role/xxxxxxxxxx
                                        
        alertmanager:
          persistentVolume:
            enabled: true
            storageClass: ebs-sc
          replicas: 2
          resources:
            limits:
              memory: 1.4Gi
            requests:
              cpu: 1
              memory: 1Gi
          statefulSet:
            enabled: true
          podAnnotations:
            eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxxxxxxx:role/xxxxxxxxxx
          extraArgs:
            memberlist.bind-addr: ${MY_POD_IP}

        compactor:
          persistentVolume:
            size: 5Gi
            storageClass: ebs-sc
          podAnnotations:
            eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxxxxxxx:role/xxxxxxxxxx
          extraArgs:
            memberlist.bind-addr: ${MY_POD_IP}

        distributor:
          replicas: 2
          podAnnotations:
            eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxxxxxxx:role/xxxxxxxxxx
          extraArgs:
            memberlist.bind-addr: ${MY_POD_IP}

        ingester:
          persistentVolume:
            size: 50Gi
            storageClass: ebs-sc
          replicas: 3
          podAnnotations:
            eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxxxxxxx:role/xxxxxxxxxx
          extraArgs:
            memberlist.bind-addr: ${MY_POD_IP}

        ruler:
          replicas: 1
          podAnnotations:
            eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxxxxxxx:role/xxxxxxxxxx
          extraArgs:
            memberlist.bind-addr: ${MY_POD_IP}

        store_gateway:
          persistentVolume:
            size: 10Gi
            storageClass: ebs-sc
          replicas: 3
          extraArgs:
            memberlist.bind-addr: ${MY_POD_IP}

        querier:
          replicas: 1
          extraArgs:
            memberlist.bind-addr: ${MY_POD_IP}

        admin-cache:
          enabled: true
          replicas: 2

        chunks-cache:
          enabled: true
          replicas: 2

        index-cache:
          enabled: true
          replicas: 1

        metadata-cache:
          enabled: true

        results-cache:
          enabled: true
          replicas: 2

        minio:
          enabled: false

        overrides_exporter:
          replicas: 1
          resources:
            limits:
              memory: 128Mi
            requests:
              cpu: 100m
              memory: 128Mi

        query_frontend:
          replicas: 1

        nginx:
          service:
            type: LoadBalancer
            annotations:
              service.beta.kubernetes.io/aws-load-balancer-ip-address-type: ipv4
              service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance
              service.beta.kubernetes.io/aws-load-balancer-subnets: "xxxxxxxxx"
              service.beta.kubernetes.io/aws-load-balancer-type: external
          replicas: 1
          resources:
            limits:
              memory: 731Mi
            requests:
              cpu: 1
              memory: 512Mi

        # Grafana Enterprise Metrics feature related
        admin_api:
          replicas: 1
          resources:
            limits:
              memory: 128Mi
            requests:
              cpu: 100m
              memory: 64Mi

        gateway:
          replicas: 1
          resources:
            limits:
              memory: 731Mi
            requests:
              cpu: 1
              memory: 512Mi

Prometheus remote write:

prometheus:
  prometheusSpec:
    remoteWrite:
      - url: http://xxxxxxxxxx.elb.us-west-2.amazonaws.com/api/v1/push

Is there anything I'm missing or is there a way to solve this issue?

[girishms-sentient]

@pstibrany @pracucci @DylanGuedes I would appreciate your assistance.

[dimitarvdimitrov]

Is it possible that you are scraping the same targets multiple times? you can check which targets are being scraped by checking the prometheus UI on the prometheis. One way this could be happening is that There are multiple Prometheus custom resources that are configured to scrape the same Service/PodMonitors

[girishms-sentient]

It appears that the response is the same as the previous screenshot, the data is being replicated three times.

-bash-4.4$ curl -s "http://k8s-xxxxxxxxxxxxxxxxxxxxx.elb.us-west-2.amazonaws.com/prometheus/api/v1/query" -d query="(node_memory_MemFree_bytes / 1024^3)"
{"status":"success","data":{"resultType":"vector","result":[
{"metric":{"container":"node-exporter","endpoint":"http-metrics","environment":"dev-aws1","instance":"10.253.48.104:9100","job":"node-exporter","namespace":"monitoring","pod":"monitoring-prometheus-node-exporter-7mbwz","prometheus":"monitoring/monitoring-kube-prometheus-prometheus","prometheus_replica":"prometheus-monitoring-kube-prometheus-prometheus-0","service":"monitoring-prometheus-node-exporter"},"value":[1696345390,"2.893810272216797"]},

{"metric":{"container":"node-exporter","endpoint":"http-metrics","environment":"dev-aws1","instance":"10.253.48.104:9100","job":"node-exporter","namespace":"monitoring","pod":"monitoring-prometheus-node-exporter-7mbwz","prometheus":"monitoring/monitoring-kube-prometheus-prometheus","prometheus_replica":"prometheus-monitoring-kube-prometheus-prometheus-1","service":"monitoring-prometheus-node-exporter"},"value":[1696345390,"2.8940505981445312"]},

{"metric":{"container":"node-exporter","endpoint":"http-metrics","environment":"dev-aws1","instance":"10.253.48.104:9100","job":"node-exporter","namespace":"monitoring","pod":"monitoring-prometheus-node-exporter-7mbwz","prometheus":"monitoring/monitoring-kube-prometheus-prometheus","prometheus_replica":"prometheus-monitoring-kube-prometheus-prometheus-2","service":"monitoring-prometheus-node-exporter"},"value":[1696345390,"2.8892440795898438"]},

{"metric":{"container":"node-exporter","endpoint":"http-metrics","environment":"dev-aws1","instance":"10.253.48.132:9100","job":"node-exporter","namespace":"monitoring","pod":"monitoring-prometheus-node-exporter-vdfbg","prometheus":"monitoring/monitoring-kube-prometheus-prometheus","prometheus_replica":"prometheus-monitoring-kube-prometheus-prometheus-0","service":"monitoring-prometheus-node-exporter"},"value":[1696345390,"4.596828460693359"]},

{"metric":{"container":"node-exporter","endpoint":"http-metrics","environment":"dev-aws1","instance":"10.253.48.132:9100","job":"node-exporter","namespace":"monitoring","pod":"monitoring-prometheus-node-exporter-vdfbg","prometheus":"monitoring/monitoring-kube-prometheus-prometheus","prometheus_replica":"prometheus-monitoring-kube-prometheus-prometheus-1","service":"monitoring-prometheus-node-exporter"},"value":[1696345390,"4.594913482666016"]},

{"metric":{"container":"node-exporter","endpoint":"http-metrics","environment":"dev-aws1","instance":"10.253.48.132:9100","job":"node-exporter","namespace":"monitoring","pod":"monitoring-prometheus-node-exporter-vdfbg","prometheus":"monitoring/monitoring-kube-prometheus-prometheus","prometheus_replica":"prometheus-monitoring-kube-prometheus-prometheus-2","service":"monitoring-prometheus-node-exporter"},"value":[1696345390,"4.596595764160156"]},

{"metric":{"container":"node-exporter","endpoint":"http-metrics","environment":"dev-aws1","instance":"10.253.48.135:9100","job":"node-exporter","namespace":"monitoring","pod":"monitoring-prometheus-node-exporter-5t6gq","prometheus":"monitoring/monitoring-kube-prometheus-prometheus","prometheus_replica":"prometheus-monitoring-kube-prometheus-prometheus-0","service":"monitoring-prometheus-node-exporter"},"value":[1696345390,"4.596412658691406"]},{"metric":{"container":"node-exporter","endpoint":"http-metrics","environment":"dev-aws1","instance":"10.253.48.135:9100","job":"node-exporter","namespace":"monitoring","pod":"monitoring-prometheus-node-exporter-5t6gq","prometheus":"monitoring/monitoring-kube-prometheus-prometheus","prometheus_replica":"prometheus-monitoring-kube-prometheus-prometheus-1","service":"monitoring-prometheus-node-exporter"},"value":[1696345390,"4.595954895019531"]},{"metric":{"container":"node-exporter","endpoint":"http-metrics","environment":"dev-aws1","instance":"10.253.48.135:9100","job":"node-exporter","namespace":"monitoring","pod":"monitoring-prometheus-node-exporter-5t6gq","prometheus":"monitoring/monitoring-kube-prometheus-prometheus","prometheus_replica":"prometheus-monitoring-kube-prometheus-prometheus-2","service":"monitoring-prometheus-node-exporter"},"value":[1696345390,"4.592529296875"]},{"metric":{"container":"node-exporter","endpoint":"http-metrics","environment":"dev-aws1","instance":"10.253.48.168:9100","job":"node-exporter","namespace":"monitoring","pod":"monitoring-prometheus-node-exporter-pwkqh","prometheus":"monitoring/monitoring-kube-prometheus-prometheus","prometheus_replica":"prometheus-monitoring-kube-prometheus-prometheus-0","service":"monitoring-prometheus-node-exporter"},"value":[1696345390,"5.476314544677734"]},{"metric":{"container":"node-exporter","endpoint":"http-metrics","environment":"dev-aws1","instance":"10.253.48.168:9100","job":"node-exporter","namespace":"monitoring","pod":"monitoring-prometheus-node-exporter-pwkqh","prometheus":"monitoring/monitoring-kube-prometheus-prometheus","prometheus_replica":"prometheus-monitoring-kube-prometheus-prometheus-1","service":"monitoring-prometheus-node-exporter"},"value":[1696345390,"5.468334197998047"]},{"metric":{"container":"node-exporter","endpoint":"http-metrics","environment":"dev-aws1","instance":"10.253.48.168:9100","job":"node-exporter","namespace":"monitoring","pod":"monitoring-prometheus-node-exporter-pwkqh","prometheus":"monitoring/monitoring-kube-prometheus-prometheus","prometheus_replica":"prometheus-monitoring-kube-prometheus-prometheus-2","service":"monitoring-prometheus-node-exporter"},"value":[1696345390,"5.468265533447266"]},{"metric":{"container":"node-exporter","endpoint":"http-metrics","environment":"dev-aws1","instance":"10.253.48.176:9100","job":"node-exporter","namespace":"monitoring","pod":"monitoring-prometheus-node-exporter-pgd9m","prometheus":"monitoring/monitoring-kube-prometheus-prometheus","prometheus_replica":"prometheus-monitoring-kube-prometheus-prometheus-0","service":"monitoring-prometheus-node-exporter"},"value":[1696345390,"0.3936424255371094"]},{"metric":{"container":"node-exporter","endpoint":"http-metrics","environment":"dev-aws1","instance":"10.253.48.176:9100","job":"node-exporter","namespace":"monitoring","pod":"monitoring-prometheus-node-exporter-pgd9m","prometheus":"monitoring/monitoring-kube-prometheus-prometheus","prometheus_replica":"prometheus-monitoring-kube-prometheus-prometheus-1","service":"monitoring-prometheus-node-exporter"},"value":[1696345390,"0.39427947998046875"]},{"metric":{"container":"node-exporter","endpoint":"http-metrics","environment":"dev-aws1","instance":"10.253.48.176:9100","job":"node-exporter","namespace":"monitoring","pod":"monitoring-prometheus-node-exporter-pgd9m","prometheus":"monitoring/monitoring-kube-prometheus-prometheus","prometheus_replica":"prometheus-monitoring-kube-prometheus-prometheus-2","service":"monitoring-prometheus-node-exporter"},"value":[1696345390,"0.39470672607421875"]},{"metric":{"container":"node-exporter","endpoint":"http-metrics","environment":"dev-aws1","instance":"10.253.48.192:9100","job":"node-exporter","namespace":"monitoring","pod":"monitoring-prometheus-node-exporter-5tnsp","prometheus":"monitoring/monitoring-kube-prometheus-prometheus","prometheus_replica":"prometheus-monitoring-kube-prometheus-prometheus-0","service":"monitoring-prometheus-node-exporter"},"value":[1696345390,"9.731731414794922"]},{"metric":{"container":"node-exporter","endpoint":"http-metrics","environment":"dev-aws1","instance":"10.253.48.192:9100","job":"node-exporter","namespace":"monitoring","pod":"monitoring-prometheus-node-exporter-5tnsp","prometheus":"monitoring/monitoring-kube-prometheus-prometheus","prometheus_replica":"prometheus-monitoring-kube-prometheus-prometheus-1","service":"monitoring-prometheus-node-exporter"},"value":[1696345390,"9.741405487060547"]},{"metric":{"container":"node-exporter","endpoint":"http-metrics","environment":"dev-aws1","instance":"10.253.48.192:9100","job":"node-exporter","namespace":"monitoring","pod":"monitoring-prometheus-node-exporter-5tnsp","prometheus":"monitoring/monitoring-kube-prometheus-prometheus","prometheus_replica":"prometheus-monitoring-kube-prometheus-prometheus-2","service":"monitoring-prometheus-node-exporter"},"value":[1696345390,"9.739631652832031"]},{"metric":{"container":"node-exporter","endpoint":"http-metrics","environment":"dev-aws1","instance":"10.253.48.233:9100","job":"node-exporter","namespace":"monitoring","pod":"monitoring-prometheus-node-exporter-rwpvd","prometheus":"monitoring/monitoring-kube-prometheus-prometheus","prometheus_replica":"prometheus-monitoring-kube-prometheus-prometheus-0","service":"monitoring-prometheus-node-exporter"},"value":[1696345390,"4.618564605712891"]},{"metric":{"container":"node-exporter","endpoint":"http-metrics","environment":"dev-aws1","instance":"10.253.48.233:9100","job":"node-exporter","namespace":"monitoring","pod":"monitoring-prometheus-node-exporter-rwpvd","prometheus":"monitoring/monitoring-kube-prometheus-prometheus","prometheus_replica":"prometheus-monitoring-kube-prometheus-prometheus-1","service":"monitoring-prometheus-node-exporter"},"value":[1696345390,"4.617603302001953"]},{"metric":{"container":"node-exporter","endpoint":"http-metrics","environment":"dev-aws1","instance":"10.253.48.233:9100","job":"node-exporter","namespace":"monitoring","pod":"monitoring-prometheus-node-exporter-rwpvd","prometheus":"monitoring/monitoring-kube-prometheus-prometheus","prometheus_replica":"prometheus-monitoring-kube-prometheus-prometheus-2","service":"monitoring-prometheus-node-exporter"},"value":[1696345390,"4.615650177001953"]},{"metric":{"container":"node-exporter","endpoint":"http-metrics","environment":"dev-aws1","instance":"10.253.48.248:9100","job":"node-exporter","namespace":"monitoring","pod":"monitoring-prometheus-node-exporter-6nsl6","prometheus":"monitoring/monitoring-kube-prometheus-prometheus","prometheus_replica":"prometheus-monitoring-kube-prometheus-prometheus-0","service":"monitoring-prometheus-node-exporter"},"value":[1696345390,"5.744964599609375"]},{"metric":{"container":"node-exporter","endpoint":"http-metrics","environment":"dev-aws1","instance":"10.253.48.248:9100","job":"node-exporter","namespace":"monitoring","pod":"monitoring-prometheus-node-exporter-6nsl6","prometheus":"monitoring/monitoring-kube-prometheus-prometheus","prometheus_replica":"prometheus-monitoring-kube-prometheus-prometheus-1","service":"monitoring-prometheus-node-exporter"},"value":[1696345390,"5.743633270263672"]},{"metric":{"container":"node-exporter","endpoint":"http-metrics","environment":"dev-aws1","instance":"10.253.48.248:9100","job":"node-exporter","namespace":"monitoring","pod":"monitoring-prometheus-node-exporter-6nsl6","prometheus":"monitoring/monitoring-kube-prometheus-prometheus","prometheus_replica":"prometheus-monitoring-kube-prometheus-prometheus-2","service":"monitoring-prometheus-node-exporter"},"value":[1696345390,"5.743400573730469"]},{"metric":{"container":"node-exporter","endpoint":"http-metrics","environment":"dev-aws1","instance":"10.253.48.29:9100","job":"node-exporter","namespace":"monitoring","pod":"monitoring-prometheus-node-exporter-s7sxs","prometheus":"monitoring/monitoring-kube-prometheus-prometheus","prometheus_replica":"prometheus-monitoring-kube-prometheus-prometheus-0","service":"monitoring-prometheus-node-exporter"},"value":[1696345390,"3.6807785034179688"]},{"metric":{"container":"node-exporter","endpoint":"http-metrics","environment":"dev-aws1","instance":"10.253.48.29:9100","job":"node-exporter","namespace":"monitoring","pod":"monitoring-prometheus-node-exporter-s7sxs","prometheus":"monitoring/monitoring-kube-prometheus-prometheus","prometheus_replica":"prometheus-monitoring-kube-prometheus-prometheus-1","service":"monitoring-prometheus-node-exporter"},"value":[1696345390,"3.6807785034179688"]},{"metric":{"container":"node-exporter","endpoint":"http-metrics","environment":"dev-aws1","instance":"10.253.48.29:9100","job":"node-exporter","namespace":"monitoring","pod":"monitoring-prometheus-node-exporter-s7sxs","prometheus":"monitoring/monitoring-kube-prometheus-prometheus","prometheus_replica":"prometheus-monitoring-kube-prometheus-prometheus-2","service":"monitoring-prometheus-node-exporter"},"value":[1696345390,"3.681011199951172"]},{"metric":{"container":"node-exporter","endpoint":"http-metrics","environment":"dev-aws1","instance":"10.253.48.31:9100","job":"node-exporter","namespace":"monitoring","pod":"monitoring-prometheus-node-exporter-vk4h8","prometheus":"monitoring/monitoring-kube-prometheus-prometheus","prometheus_replica":"prometheus-monitoring-kube-prometheus-prometheus-0","service":"monitoring-prometheus-node-exporter"}

[pstibrany]

It's replicated 3 times, because it's coming from 3 different prometheus_replicas.

It seems like your HA deduplication is either not set up properly (eg. I can't see any cluster label, only prometheus_replica label), or it's not working.

[pstibrany]

Thank you for sharing raw JSON output, that's very useful in this troubleshooting.

[pstibrany]

To read more about HA deduplication feature: https://grafana.com/docs/mimir/latest/configure/configure-high-availability-deduplication/

In your case, perhaps configuring "cluster" label to environment and "replica" label to prometheus_replicas would work?

[girishms-sentient]

Thank you @pstibrany, After adding external labels on Prometheus and making the required changes to the ha-tracker on Mimir, I am able to get accurate data now.

Mimir:

            limits:
              ingestion_rate: 200000
              max_global_series_per_metric: 3000000
              max_global_series_per_user: 3000000                
              max_label_names_per_series: 60
              compactor_blocks_retention_period: 0s
              compactor_split_and_merge_shards: 0
              compactor_split_groups: 0
              accept_ha_samples: true
              ha_cluster_label: "environment"
              ha_replica_label: "__replica__"

            distributor:
              ring:
                kvstore:
                  store: memberlist
              instance_limits:
                max_inflight_push_requests: 0
              ha_tracker:
                enable_ha_tracker: true
                kvstore:
                  store: inmemory
                  prefix: ha-tracker

Prometheus:

prometheus:
  prometheusSpec:
    externalLabels:
      environment: {{ environment_name }}
      __replica__: prometheus_replica_{{ environment_name }}
    replicaExternalLabelNameClear: true

Now I could see some exceptions on Prometheus logs, "the sample has been rejected because another sample with the same timestamp, but a different value, has already been ingested". Btw it's not impacting anything, data in the mimir is fine and accurate.

ts=2023-10-06T12:33:46.907Z caller=dedupe.go:112 component=remote level=error remote_name=e64587 url=http://k8s-centralm-mimirngi-de1e8b696f-d9a0b2a1383dbad7.elb.us-west-2.amazonaws.com/api/v1/push msg="non-recoverable error" count=500 exemplarCount=0 err="server returned HTTP status 400 Bad Request: failed pushing to ingester: user=anonymous: the sample has been rejected because another sample with the same timestamp, but a different value, has already been ingested (err-mimir-sample-duplicate-timestamp). The affected sample has timestamp 2023-10-06T12:33:46.848Z and is from series {__name__=\"instance:node_cpu_utilisation:rate5m\", container=\"node-exporter\", endpoint=\"http-metrics\", environment=\"dev-aws1\", instance=\"10.253.48.104:9100\", job=\"node-exporter\", namespace=\"monitoring\", pod=\"monitoring-prometheus-node-exporter-l65f9\", prometheus=\"monitoring/monitoring-kube-prometheus-prometheus\", service=\"monitoring-prometheus-node-exporter\"}"
ts=2023-10-06T12:33:47.006Z caller=dedupe.go:112 component=remote level=error remote_name=e64587 url=http://k8s-centralm-mimirngi-de1e8b696f-d9a0b2a1383dbad7.elb.us-west-2.amazonaws.com/api/v1/push msg="non-recoverable error" count=500 exemplarCount=0 err="server returned HTTP status 400 Bad Request: failed pushing to ingester: user=anonymous: the sample has been rejected because another sample with the same timestamp, but a different value, has already been ingested (err-mimir-sample-duplicate-timestamp). The affected sample has timestamp 2023-10-06T12:33:46.848Z and is from series {__name__=\"instance_device:node_disk_io_time_seconds:rate5m\", container=\"node-exporter\", device=\"nvme0n1\", endpoint=\"http-metrics\", environment=\"dev-aws1\", instance=\"10.253.48.233:9100\", job=\"node-exporter\", namespace=\"monitoring\", pod=\"monitoring-prometheus-node-exporter-r5c7l\", prometheus=\"monitoring/monitoring-kube-prometheus-prometheus\", service=\"monitoring-prometheus-node-exporter\"}"
ts=2023-10-06T12:33:49.304Z caller=dedupe.go:112 component=remote level=error remote_name=e64587 url=http://k8s-centralm-mimirngi-de1e8b696f-d9a0b2a1383dbad7.elb.us-west-2.amazonaws.com/api/v1/push msg="non-recoverable error" count=500 exemplarCount=0 err="server returned HTTP status 400 Bad Request: failed pushing to ingester: user=anonymous: the sample has been rejected because another sample with the same timestamp, but a different value, has already been ingested (err-mimir-sample-duplicate-timestamp). The affected sample has timestamp 2023-10-06T12:33:48.681Z and is from series {__name__=\"ALERTS_FOR_STATE\", alertname=\"PrometheusRemoteStorageFailures\", container=\"prometheus\", endpoint=\"http-web\", environment=\"dev-aws1\", instance=\"7.0.253.144:9090\", job=\"monitoring-kube-prometheus-prometheus\", namespace=\"monitoring\", pod=\"prometheus-monitoring-kube-prometheus-prometheus-1\", prometheus=\"monitoring/monitoring-kube-prometheus-prometheus\", remote_name=\"e64587\", service=\"monitoring-kube-prometheus-prometheus\", severity=\"critical\", url=\"http://k8s-centralm-mimirngi-de1e8b696f-d9a0b2a1383dbad7.elb.us-west-2.amazonaws.com/api/v1/push\"}"

Do you have any suggestions for resolving this issue?

[sivamalla42]

Hi @girishms-sentient ,

Even I had the same issue, we have 3 prometheus instances in prod, so made the below changes as suggested in your above solution. But it failed with below error
ts=2024-05-03T05:52:55.838623605Z caller=push.go:171 level=error user=anonymous msg="push error" err="at least 2 live replicas required, could only find 1 - unhealthy instances: 10.200.26.190:9095,10.200.26.92:9095" ts=2024-05-03T05:52:55.838668903Z caller=logging.go:126 level=warn trace_id=7047d376b6513325 msg="POST /api/v1/push (500) 1.579785ms Response: \"at least 2 live replicas required, could only find 1 - unhealthy instances: 10.200.26.190:9095,10.200.26.92:9095\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 57717; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.40.5; X-Forwarded-For: 10.200.26.161; X-Forwarded-Host: mimir.tools.digital.jcrew.com; X-Forwarded-Port: 80; X-Forwarded-Proto: http; X-Forwarded-Scheme: http; X-Prometheus-Remote-Write-Version: 0.1.0; X-Real-Ip: 10.200.26.161; X-Request-Id: 388c19995730cc0fd43dda2d1870306d; X-Scheme: http; X-Scope-Orgid: anonymous; " ts=2024-05-03T05:52:56.047864955Z caller=push.go:171 level=error user=anonymous msg="push error" err="at least 2 live replicas required, could only find 1 - unhealthy instances: 10.200.26.132:9095,10.200.26.92:9095" ts=2024-05-03T05:52:56.047904408Z caller=logging.go:126 level=warn trace_id=30feeb624e441d6a msg="POST /api/v1/push (500) 1.210953ms Response: \"at least 2 live replicas required, could only find 1 - unhealthy instances: 10.200.26.132:9095,10.200.26.92:9095\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 22199; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.40.5; X-Forwarded-For: 10.200.26.83; X-Forwarded-Host: mimir.tools.digital.jcrew.com; X-Forwarded-Port: 80; X-Forwarded-Proto: http; X-Forwarded-Scheme: http; X-Prometheus-Remote-Write-Version: 0.1.0; X-Real-Ip: 10.200.26.83; X-Request-Id: d791c64d470aa82cdfa6f82d9ca0f34f; X-Scheme: http; X-Scope-Orgid: anonymous; "

can you please guide me in what mistake i have made in my below yam file below
I have attached my [files]
mimir.zip