Object Storage Capacity Optimization #6135

JayChanggithub · 2023-02-13T05:04:50Z

JayChanggithub
Feb 13, 2023

Hi Team,
I observed our object storage which we adopted Azure blob storage account data capacity already grow up to 1.3TB so far. Also we adopted thanos receiver approach to collect about 15 kubernetes cluster. Have anything possible to optimize object storage consumed. any experiences can reference, regarding those topic. In additional, The thanos had been operated about 9 days.

Promethues version: v2.40.5
Thanos version: v0.30.2

compact args:

      - args:
        - compact
        - --wait
        - --objstore.config-file=/etc/thanos/objectstorage.yaml
        - --data-dir=/var/thanos/compact
        - --debug.accept-malformed-index
        - --log.level=debug
        - --retention.resolution-raw=90d
        - --retention.resolution-5m=180d
        - --retention.resolution-1h=360d

store args:

      - args:
        - store
        - --log.level=debug
        - --data-dir=/var/thanos/store
        - --grpc-address=0.0.0.0:10901
        - --http-address=0.0.0.0:10902
        - --objstore.config-file=/etc/thanos/objectstorage.yaml

Azure blob storage:

fpetkovski · 2023-02-13T09:50:15Z

fpetkovski
Feb 13, 2023
Maintainer

Hi @JayChanggithub, is the Thanos Compactor running against the bucket? You can check the thanos_compact_todo_compactions metric to make sure it is not falling behind.

0 replies

JayChanggithub · 2023-02-14T17:37:39Z

JayChanggithub
Feb 14, 2023
Author

Hi @fpetkovski ,
Thanks for your replying. I saw the compact log as below. I not sure whether exactly result. Also i confirmed that Thanos Compactor component is running the same with store specify Azure blob bucket. The object storage file is /etc/thanos/objectstorage.yaml via a configMap object in kubernetes.

k logs -f thanos-compact-0  -n thanos

level=debug ts=2023-02-14T16:07:47.71954925Z caller=azure.go:304 msg="checking if blob exists" blob=01GRHP1V64T5RDHAMJGD1NY38A/deletion-mark.json
level=info ts=2023-02-14T16:07:47.722790093Z caller=clean.go:59 msg="deleted aborted partial upload" block=01GRHP1V64T5RDHAMJGD1NY38A thresholdAge=48h0m0s
level=info ts=2023-02-14T16:07:47.722820195Z caller=clean.go:49 msg="found partially uploaded block; marking for deletion" block=01GRQX5XVVMFBZV584RR9CANAM
level=debug ts=2023-02-14T16:07:47.722832196Z caller=azure.go:304 msg="checking if blob exists" blob=01GRQX5XVVMFBZV584RR9CANAM/meta.json
level=debug ts=2023-02-14T16:07:47.734805692Z caller=azure.go:304 msg="checking if blob exists" blob=01GRQX5XVVMFBZV584RR9CANAM/deletion-mark.json
level=info ts=2023-02-14T16:07:47.738078436Z caller=clean.go:59 msg="deleted aborted partial upload" block=01GRQX5XVVMFBZV584RR9CANAM thresholdAge=48h0m0s
level=info ts=2023-02-14T16:07:47.738114139Z caller=clean.go:49 msg="found partially uploaded block; marking for deletion" block=01GRRZGHNV13P7A8D6WD1094ZT
level=debug ts=2023-02-14T16:07:47.73812794Z caller=azure.go:304 msg="checking if blob exists" blob=01GRRZGHNV13P7A8D6WD1094ZT/meta.json
level=debug ts=2023-02-14T16:07:47.750331353Z caller=azure.go:304 msg="checking if blob exists" blob=01GRRZGHNV13P7A8D6WD1094ZT/deletion-mark.json
level=info ts=2023-02-14T16:07:47.753703005Z caller=clean.go:59 msg="deleted aborted partial upload" block=01GRRZGHNV13P7A8D6WD1094ZT thresholdAge=48h0m0s
level=info ts=2023-02-14T16:07:47.753739708Z caller=clean.go:49 msg="found partially uploaded block; marking for deletion" block=01GRXRDYJDCT37ZBAP1Q5NMZTQ
level=debug ts=2023-02-14T16:07:47.753756309Z caller=azure.go:304 msg="checking if blob exists" blob=01GRXRDYJDCT37ZBAP1Q5NMZTQ/meta.json
level=debug ts=2023-02-14T16:07:47.766631173Z caller=azure.go:304 msg="checking if blob exists" blob=01GRXRDYJDCT37ZBAP1Q5NMZTQ/deletion-mark.json
level=info ts=2023-02-14T16:07:47.770045528Z caller=clean.go:59 msg="deleted aborted partial upload" block=01GRXRDYJDCT37ZBAP1Q5NMZTQ thresholdAge=48h0m0s
level=info ts=2023-02-14T16:07:47.770081231Z caller=clean.go:49 msg="found partially uploaded block; marking for deletion" block=01GRD5VJXSHKNZNBYW51PGHA8T
level=debug ts=2023-02-14T16:07:47.770095332Z caller=azure.go:304 msg="checking if blob exists" blob=01GRD5VJXSHKNZNBYW51PGHA8T/meta.json
level=debug ts=2023-02-14T16:07:47.782284444Z caller=azure.go:304 msg="checking if blob exists" blob=01GRD5VJXSHKNZNBYW51PGHA8T/deletion-mark.json
level=info ts=2023-02-14T16:07:47.785555088Z caller=clean.go:59 msg="deleted aborted partial upload" block=01GRD5VJXSHKNZNBYW51PGHA8T thresholdAge=48h0m0s
level=info ts=2023-02-14T16:07:47.785588491Z caller=clean.go:49 msg="found partially uploaded block; marking for deletion" block=01GRJHGR60PXGP5ZYMEARFGT74
level=debug ts=2023-02-14T16:07:47.785602092Z caller=azure.go:304 msg="checking if blob exists" blob=01GRJHGR60PXGP5ZYMEARFGT74/meta.json
level=debug ts=2023-02-14T16:07:47.798116228Z caller=azure.go:304 msg="checking if blob exists" blob=01GRJHGR60PXGP5ZYMEARFGT74/deletion-mark.json
level=info ts=2023-02-14T16:07:47.801147155Z caller=clean.go:59 msg="deleted aborted partial upload" block=01GRJHGR60PXGP5ZYMEARFGT74 thresholdAge=48h0m0s
level=info ts=2023-02-14T16:07:47.801180557Z caller=clean.go:49 msg="found partially uploaded block; marking for deletion" block=01GRMP615RTCJ7YJW129CE18F7
level=debug ts=2023-02-14T16:07:47.801192358Z caller=azure.go:304 msg="checking if blob exists" blob=01GRMP615RTCJ7YJW129CE18F7/meta.json
level=debug ts=2023-02-14T16:07:47.81324576Z caller=azure.go:304 msg="checking if blob exists" blob=01GRMP615RTCJ7YJW129CE18F7/deletion-mark.json
level=info ts=2023-02-14T16:07:47.816504204Z caller=clean.go:59 msg="deleted aborted partial upload" block=01GRMP615RTCJ7YJW129CE18F7 thresholdAge=48h0m0s
level=info ts=2023-02-14T16:07:47.816540907Z caller=clean.go:49 msg="found partially uploaded block; marking for deletion" block=01GRVB1HDCBM1X6H3FMMP3PAGP
level=debug ts=2023-02-14T16:07:47.816553808Z caller=azure.go:304 msg="checking if blob exists" blob=01GRVB1HDCBM1X6H3FMMP3PAGP/meta.json
level=debug ts=2023-02-14T16:07:47.834048116Z caller=azure.go:304 msg="checking if blob exists" blob=01GRVB1HDCBM1X6H3FMMP3PAGP/deletion-mark.json
level=info ts=2023-02-14T16:07:47.837112446Z caller=clean.go:59 msg="deleted aborted partial upload" block=01GRVB1HDCBM1X6H3FMMP3PAGP thresholdAge=48h0m0s
level=info ts=2023-02-14T16:07:47.837146448Z caller=clean.go:49 msg="found partially uploaded block; marking for deletion" block=01GRE867NFK6CA4N99HQ6RN7TP
level=debug ts=2023-02-14T16:07:47.837159649Z caller=azure.go:304 msg="checking if blob exists" blob=01GRE867NFK6CA4N99HQ6RN7TP/meta.json
level=debug ts=2023-02-14T16:07:47.849792194Z caller=azure.go:304 msg="checking if blob exists" blob=01GRE867NFK6CA4N99HQ6RN7TP/deletion-mark.json
level=info ts=2023-02-14T16:07:47.853029737Z caller=clean.go:59 msg="deleted aborted partial upload" block=01GRE867NFK6CA4N99HQ6RN7TP thresholdAge=48h0m0s
level=info ts=2023-02-14T16:07:47.853062239Z caller=clean.go:49 msg="found partially uploaded block; marking for deletion" block=01GRF3N45PC8S1BSBNTK6DQ89D
level=debug ts=2023-02-14T16:07:47.85307434Z caller=azure.go:304 msg="checking if blob exists" blob=01GRF3N45PC8S1BSBNTK6DQ89D/meta.json
level=debug ts=2023-02-14T16:07:47.865736787Z caller=azure.go:304 msg="checking if blob exists" blob=01GRF3N45PC8S1BSBNTK6DQ89D/deletion-mark.json
level=info ts=2023-02-14T16:07:47.869167144Z caller=clean.go:59 msg="deleted aborted partial upload" block=01GRF3N45PC8S1BSBNTK6DQ89D thresholdAge=48h0m0s
level=info ts=2023-02-14T16:07:47.869204647Z caller=clean.go:49 msg="found partially uploaded block; marking for deletion" block=01GRYDMBXZX96DM3W47R6SBXDE
level=debug ts=2023-02-14T16:07:47.869218348Z caller=azure.go:304 msg="checking if blob exists" blob=01GRYDMBXZX96DM3W47R6SBXDE/meta.json
level=debug ts=2023-02-14T16:07:47.882152015Z caller=azure.go:304 msg="checking if blob exists" blob=01GRYDMBXZX96DM3W47R6SBXDE/deletion-mark.json
level=info ts=2023-02-14T16:07:47.885381757Z caller=clean.go:59 msg="deleted aborted partial upload" block=01GRYDMBXZX96DM3W47R6SBXDE thresholdAge=48h0m0s
level=info ts=2023-02-14T16:07:47.88541666Z caller=clean.go:49 msg="found partially uploaded block; marking for deletion" block=01GRZ6PPHZ376ZXSDY94PJ7YGD
level=debug ts=2023-02-14T16:07:47.885429261Z caller=azure.go:304 msg="checking if blob exists" blob=01GRZ6PPHZ376ZXSDY94PJ7YGD/meta.json
level=debug ts=2023-02-14T16:07:47.897509664Z caller=azure.go:304 msg="checking if blob exists" blob=01GRZ6PPHZ376ZXSDY94PJ7YGD/deletion-mark.json
level=info ts=2023-02-14T16:07:47.901024627Z caller=clean.go:59 msg="deleted aborted partial upload" block=01GRZ6PPHZ376ZXSDY94PJ7YGD thresholdAge=48h0m0s
level=info ts=2023-02-14T16:07:47.901089732Z caller=clean.go:49 msg="found partially uploaded block; marking for deletion" block=01GS26XGWX2HJT3D7NEEQHTN90
level=debug ts=2023-02-14T16:07:47.901116434Z caller=azure.go:304 msg="checking if blob exists" blob=01GS26XGWX2HJT3D7NEEQHTN90/meta.json

0 replies

fpetkovski · 2023-02-15T15:27:54Z

fpetkovski
Feb 15, 2023
Maintainer

Got it, thanks for the logs. Looking at your compactor configuration, I don't think you've set a deduplication label, so you are probably not compacting blocks properly. You can open the compactor UI on port 10902 and see what your blocks and compaction groups look like. Ideally you want to deduplicate blocks from all receivers with something like

- --deduplication.replica-label=replica

or whatever the configured replica label on the receiver is.

0 replies

JayChanggithub · 2023-02-16T05:50:28Z

JayChanggithub
Feb 16, 2023
Author

@fpetkovski Thanks for your clarifying. Also the below as our arguments of thanos components. Shall you help to confirm any miss configuration? We using the Azure blob for object storage. I refer here to update my configuration. In additional i also view the compact UI via 10902 port of Compact as screenshot.

compact

      - args:
        - compact
        - --wait
        - --wait-interval=5m
        - --consistency-delay=30m
        - --compact.enable-vertical-compaction
        - --deduplication.func=penalty
        - --delete-delay=30m
        - --compact.concurrency=2
        - --deduplication.replica-label=receive_replica  # without ""
        - --objstore.config-file=/etc/thanos/objectstorage.yaml
        - --data-dir=/var/thanos/compact
        - --debug.accept-malformed-index
        - --log.level=debug
        - --retention.resolution-raw=90d
        - --retention.resolution-5m=180d
        - --retention.resolution-1h=360d

Query

      - args:
        - query
        - --log.level=debug
        - --query.auto-downsampling
        - --grpc-address=0.0.0.0:10901
        - --http-address=0.0.0.0:9090
        - --query.partial-response
        - --query.replica-label=prometheus_replica
        - --query.replica-label=rule_replica  #### rulers
        - --query.replica-label=receive_replica #### Receive
        # - --store=dnssrv+_grpc._tcp.prometheus-headless.thanos.svc.cluster.local
        - --store=dnssrv+_grpc._tcp.thanos-rule.thanos.svc.cluster.local
        - --store=dnssrv+_grpc._tcp.thanos-store.thanos.svc.cluster.local
        - --store=dnssrv+_grpc._tcp.thanos-receive.thanos.svc.cluster.local

Receive

      - args:
        - receive
        - --grpc-address=0.0.0.0:10901
        - --http-address=0.0.0.0:10902
        - --remote-write.address=0.0.0.0:19291
        - --receive.replication-factor=1
        - --objstore.config-file=/etc/thanos/objectstorage.yaml
        - --tsdb.path=/var/thanos/receive
        - --tsdb.retention=12h
        - --receive-forward-timeout=120s
        - --label=receive_replica="$(NAME)"
        - --label=receive="true"
        - --receive.hashrings-file=/etc/thanos/thanos-receive-hashrings.json
        - --receive.local-endpoint=$(NAME).thanos-receive.thanos.svc.cluster.local:10901

Rules

      - args:
        - rule
        - --grpc-address=0.0.0.0:10901
        - --http-address=0.0.0.0:10902
        - --rule-file=/etc/thanos/rules/*rules.yaml
        - --objstore.config-file=/etc/thanos/objectstorage.yaml
        - --data-dir=/var/thanos/rule
        - --label=rule_replica="$(NAME)"
        - --alert.label-drop=rule_replica
        - --query=dnssrv+_http._tcp.thanos-query.thanos.svc.cluster.local
        - --alertmanagers.url=http://alertmanager:9093

Stores

      - args:
        - store
        - --log.level=debug
        - --data-dir=/var/thanos/store
        - --grpc-address=0.0.0.0:10901
        - --http-address=0.0.0.0:10902
        - --objstore.config-file=/etc/thanos/objectstorage.yaml

$ k -n thanos logs -f thanos-compact-0   | grep -i "removed" 
level=debug ts=2023-02-16T06:26:56.964460079Z caller=fetcher.go:720 msg="replica label removed" label=receive_replica
level=debug ts=2023-02-16T06:26:56.964523881Z caller=fetcher.go:720 msg="replica label removed" label=receive_replica
level=debug ts=2023-02-16T06:26:56.964546281Z caller=fetcher.go:720 msg="replica label removed" label=receive_replica
level=debug ts=2023-02-16T06:26:56.964573982Z caller=fetcher.go:720 msg="replica label removed" label=receive_replica
level=debug ts=2023-02-16T06:26:56.964651384Z caller=fetcher.go:720 msg="replica label removed" label=receive_replica
level=debug ts=2023-02-16T06:26:56.964687084Z caller=fetcher.go:720 msg="replica label removed" label=receive_replica
level=debug ts=2023-02-16T06:26:56.964715785Z caller=fetcher.go:720 msg="replica label removed" label=receive_replica
level=debug ts=2023-02-16T06:26:56.964743186Z caller=fetcher.go:720 msg="replica label removed" label=receive_replica
level=debug ts=2023-02-16T06:26:56.964768886Z caller=fetcher.go:720 msg="replica label removed" label=receive_replica
level=debug ts=2023-02-16T06:26:56.964792687Z caller=fetcher.go:720 msg="replica label removed" label=receive_replica
level=debug ts=2023-02-16T06:26:56.964816687Z caller=fetcher.go:720 msg="replica label removed" label=receive_replica
level=debug ts=2023-02-16T06:26:56.964834087Z caller=fetcher.go:720 msg="replica label removed" label=receive_replica
level=debug ts=2023-02-16T06:26:56.964859988Z caller=fetcher.go:720 msg="replica label removed" label=receive_replica
level=debug ts=2023-02-16T06:26:56.964885389Z caller=fetcher.go:720 msg="replica label removed" label=receive_replica
level=debug ts=2023-02-16T06:26:56.964913189Z caller=fetcher.go:720 msg="replica label removed" label=receive_replica
level=debug ts=2023-02-16T06:26:56.96495429Z caller=fetcher.go:720 msg="replica label removed" label=receive_replica
level=debug ts=2023-02-16T06:26:56.96496549Z caller=fetcher.go:720 msg="replica label removed" label=receive_replica
level=debug ts=2023-02-16T06:26:56.964978891Z caller=fetcher.go:720 msg="replica label removed" label=receive_replica
level=debug ts=2023-02-16T06:26:56.964991391Z caller=fetcher.go:720 msg="replica label removed" label=receive_replica
level=debug ts=2023-02-16T06:26:56.965000191Z caller=fetcher.go:720 msg="replica label removed" label=receive_replica
level=debug ts=2023-02-16T06:26:56.965012491Z caller=fetcher.go:720 msg="replica label removed" label=receive_replica
level=debug ts=2023-02-16T06:26:56.965021092Z caller=fetcher.go:720 msg="replica label removed" label=receive_replica
level=debug ts=2023-02-16T06:26:56.965030892Z caller=fetcher.go:720 msg="replica label removed" label=receive_replica
level=debug ts=2023-02-16T06:26:56.965044592Z caller=fetcher.go:720 msg="replica label removed" label=receive_replica
level=debug ts=2023-02-16T06:26:56.965053692Z caller=fetcher.go:720 msg="replica label removed" label=receive_replica
level=debug ts=2023-02-16T06:26:56.965063292Z caller=fetcher.go:720 msg="replica label removed" label=receive_replica
level=debug ts=2023-02-16T06:26:56.965072993Z caller=fetcher.go:720 msg="replica label removed" label=receive_replica
level=debug ts=2023-02-16T06:26:56.965086293Z caller=fetcher.go:720 msg="replica label removed" label=receive_replica
level=debug ts=2023-02-16T06:26:56.965095193Z caller=fetcher.go:720 msg="replica label removed" label=receive_replica
level=debug ts=2023-02-16T06:26:56.965109693Z caller=fetcher.go:720 msg="replica label removed" label=receive_replica
level=debug ts=2023-02-16T06:26:56.965124194Z caller=fetcher.go:720 msg="replica label removed" label=receive_replica
level=debug ts=2023-02-16T06:26:56.965134294Z caller=fetcher.go:720 msg="replica label removed" label=receive_replica
level=debug ts=2023-02-16T06:26:56.965142894Z caller=fetcher.go:720 msg="replica label removed" label=receive_replica
level=debug ts=2023-02-16T06:26:56.965155394Z caller=fetcher.go:720 msg="replica label removed" label=receive_replica
level=debug ts=2023-02-16T06:26:56.965163395Z caller=fetcher.go:720 msg="replica label removed" label=receive_replica
level=debug ts=2023-02-16T06:26:56.965172895Z caller=fetcher.go:720 msg="replica label removed" label=receive_replica
level=debug ts=2023-02-16T06:26:56.965185295Z caller=fetcher.go:720 msg="replica label removed" label=receive_replica
level=debug ts=2023-02-16T06:26:56.965193595Z caller=fetcher.go:720 msg="replica label removed" label=receive_replica
level=debug ts=2023-02-16T06:26:56.965203395Z caller=fetcher.go:720 msg="replica label removed" label=receive_replica
level=debug ts=2023-02-16T06:26:56.965230196Z caller=fetcher.go:720 msg="replica label removed" label=receive_replica
level=debug ts=2023-02-16T06:26:56.965243196Z caller=fetcher.go:720 msg="replica label removed" label=receive_replica
level=debug ts=2023-02-16T06:26:56.965256097Z caller=fetcher.go:720 msg="replica label removed" label=receive_replica
level=debug ts=2023-02-16T06:26:56.965265097Z caller=fetcher.go:720 msg="replica label removed" label=receive_replica

0 replies

fpetkovski · 2023-02-16T11:50:17Z

fpetkovski
Feb 16, 2023
Maintainer

Thanks for sharing more information. I wonder why you see only blocks from receiver replica 0 in that compaction group 🤔 Are blocks from other receivers in different groups?

0 replies

JayChanggithub · 2023-02-16T12:29:31Z

JayChanggithub
Feb 16, 2023
Author

Hi @fpetkovski,
How should i confirmed that what's compacted group. I not sure how to clarify. However just now i take into compact UI again. The screenshot as following. Whether become expected result(3 groups? right?).

0 replies

fpetkovski · 2023-02-16T12:33:50Z

fpetkovski
Feb 16, 2023
Maintainer

This looks good, it seems that you have 3 receivers in the same compaction group and the compactor is properly configured. I don't think there is much else you can do to reduce the amount of data in object storage. It's also not that big of a problem since object storage is usually billed per API call and the storage itself is relatively cheaper.

0 replies

JayChanggithub · 2023-02-16T12:36:27Z

JayChanggithub
Feb 16, 2023
Author

@fpetkovski Thank you your confirmation.

0 replies

yeya24 · 2023-02-17T00:52:21Z

yeya24
Feb 17, 2023
Maintainer

Converted to a discussion instead.

0 replies

JayChanggithub · 2023-02-18T14:42:04Z

JayChanggithub
Feb 18, 2023
Author

@yeya24 @fpetkovski
We Intend to use receiver in formal environments such that collected all of kubernetes promethues metrics no matter production or none productions around 150+ totally probably. In addition few questions regarding receiver.

Through experiment we founds when i adjust the argument which --receive.replication-factor from 1 to 3. The pod of receiver memory consumed would be increased. does it make reasonable? If we have 5 replicas of receiver pods what's --receive.replication-factor is properly or make sense?
We founds when receiver pods after rolling restart it' will interrupt metrics for a while. No matters the --receive.replication-factor default or be increased. Any suggestion?
What's limitation of resources it's enough for us when we will adopted 150+ promethues via each kubernetes cluster. Also we have 5 replicas of receiver pod.

If any best practices. I'd like to reference. Thank you.

apiVersion: v1
kind: ConfigMap
metadata:
  name: thanos-receive-hashrings
  namespace: thanos
data:
  thanos-receive-hashrings.json: |
    [
      {
        "hashring": "soft-tenants",
        "endpoints":
        [
            "thanos-receive-0.thanos-receive.thanos.svc.cluster.local:10901",
            "thanos-receive-1.thanos-receive.thanos.svc.cluster.local:10901",
            "thanos-receive-2.thanos-receive.thanos.svc.cluster.local:10901",
            "thanos-receive-3.thanos-receive.thanos.svc.cluster.local:10901",
            "thanos-receive-4.thanos-receive.thanos.svc.cluster.local:10901"
        ]
      }
    ]

spec:
  replicas: 5
  selector:
    matchLabels:
      kubernetes.io/name: thanos-receive
  serviceName: thanos-receive
  template:
    metadata:
      labels:
        kubernetes.io/name: thanos-receive
        thanos-store-api: "true"
    spec:
      containers:
      - args:
        - receive
        - --grpc-address=0.0.0.0:10901
        - --http-address=0.0.0.0:10902
        - --remote-write.address=0.0.0.0:19291
        - --receive.replication-factor=2
        - --objstore.config-file=/etc/thanos/objectstorage.yaml
        - --tsdb.path=/var/thanos/receive
        - --tsdb.retention=12h
        - --receive-forward-timeout=120s
        - --label=receive_replica="$(NAME)"
        - --label=receive="true"
        - --receive.hashrings-file=/etc/thanos/thanos-receive-hashrings.json
        - --receive.local-endpoint=$(NAME).thanos-receive.thanos.svc.cluster.local:10901

        resources:
          limits:
            cpu: "16"
            memory: 128Gi
          requests:
            cpu: "4"
            memory: 16Gi

1 reply

fpetkovski Feb 20, 2023
Maintainer

I would suggest to set replication factor to 3, and I strongly recommend using the ketama hashing algorithm by setting --receive.hashrings-algorithm=ketama. This will allow you to add more receivers later without creating memory spikes. With this setup your receivers will use 3 times more memory, but this you will have very good resilience and will also be able to roll out new versions without disruptions.

One thing to note is that 150 clusters can produce a lot of data, so I would suggest you use a different tenant for each cluster by setting the THANOS-TENANT header to the cluster name in the remote write config. This will allow you to run compaction in parallel for each cluster.

JayChanggithub · 2023-02-20T11:01:32Z

JayChanggithub
Feb 20, 2023
Author

@fpetkovski Very clearly. Also i will do that. Thank you.

0 replies

JayChanggithub · 2023-02-23T12:32:13Z

JayChanggithub
Feb 23, 2023
Author

Hi @fpetkovski,
In other hands, we would like to adopt thanos-rules to implement alerting to someone, Anything possible to management alering rules by like CRD of prometheus-rules object to do? If so shall you share link? And did you recommended us to use thaos-rules to manage all of clusters alerting? Thanks you.

0 replies

JayChanggithub · 2023-02-24T08:37:52Z

JayChanggithub
Feb 24, 2023
Author

Hi @fpetkovski @yeya24
Would you please help me confirm that i meet OOM issue again today. I just only receiver around 27 cluster's promethues to receiver. The below pods status. You can see the receiver-0 always restart.

$ k get po -n thanos                                                                                                                                                                                               [16:23:41]
NAME                            READY   STATUS    RESTARTS        AGE
alertmanager-0                  2/2     Running   0               15d
alertmanager-1                  2/2     Running   0               15d
alertmanager-2                  2/2     Running   0               15d
thanos-compact-0                1/1     Running   0               8d
thanos-query-7bd9bfb7cf-29647   1/1     Running   0               15d
thanos-query-7bd9bfb7cf-59jvg   1/1     Running   0               15d
thanos-query-7bd9bfb7cf-qj689   1/1     Running   0               15d
thanos-receive-0                0/1     Running   6 (2m27s ago)   58m
thanos-receive-1                1/1     Running   0               64m
thanos-receive-2                1/1     Running   0               70m
thanos-receive-3                1/1     Running   0               76m
thanos-receive-4                1/1     Running   0               83m
thanos-rule-0                   1/1     Running   0               15d
thanos-rule-1                   1/1     Running   0               15d
thanos-store-0                  1/1     Running   0               20d
thanos-store-1                  1/1     Running   0               20d

In additional logs as following.

level=error ts=2023-02-24T08:30:11.78168462Z caller=grpc.go:70 component=receive service=gRPC/server component=receive msg="recovered from panic" panic="runtime error: invalid memory address or nil pointer dereference" stack="goroutine 108086 [running]:\nruntime/debug.Stack()\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x65\ngithub.com/thanos-io/thanos/pkg/server/grpc.New.func1({0x22909c0?, 0x4109510})\n\t/app/pkg/server/grpc/grpc.go:70 +0xda\ngithub.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors/recovery.WithRecoveryHandler.func1.1({0x7f6a923979a8?, 0x8?}, {0x22909c0?, 0x4109510?})\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/interceptors/recovery/options.go:33 +0x2d\ngithub.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors/recovery.recoverFrom({0x2bf41e8?, 0xc0a84d9c20?}, {0x22909c0?, 0x4109510?}, 0x0?)\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/interceptors/recovery/interceptors.go:53 +0x36\ngithub.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors/recovery.UnaryServerInterceptor.func1.1()\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/interceptors/recovery/interceptors.go:27 +0x68\npanic({0x22909c0, 0x4109510})\n\t/usr/local/go/src/runtime/panic.go:884 +0x212\ngithub.com/thanos-io/thanos/pkg/store.(*TSDBStore).LabelSet(0xc0a6621720?)\n\t/app/pkg/store/tsdb.go:104 +0x18\ngithub.com/thanos-io/thanos/pkg/receive.(*localClient).LabelSets(0xc0008be000?)\n\t/app/pkg/receive/multitsdb.go:109 +0x2f\ngithub.com/thanos-io/thanos/pkg/store.(*ProxyStore).LabelSet(0xc00063f3b0)\n\t/app/pkg/store/proxy.go:196 +0x112\nmain.runReceive.func3()\n\t/app/cmd/thanos/receive.go:323 +0x1d\ngithub.com/thanos-io/thanos/pkg/info.(*InfoServer).Info(0xc000a8d4a0, {0x26b4c38?, 0x2184f60?}, 0xc0a6621700?)\n\t/app/pkg/info/info.go:174 +0x26\ngithub.com/thanos-io/thanos/pkg/info/infopb._Info_Info_Handler.func1({0x2bf41e8, 0xc0a84fc0c0}, {0x2579f20?, 0x417d880})\n\t/app/pkg/info/infopb/rpc.pb.go:460 +0x78\ngithub.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors.UnaryServerInterceptor.func1({0x2bf41e8, 0xc0a84fc0c0}, {0x2579f20, 0x417d880}, 0x44cd0f954?, 0xc0a84e89d8)\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/interceptors/server.go:22 +0x21e\ngithub.com/grpc-ecosystem/go-grpc-middleware/v2.ChainUnaryServer.func1.1.1({0x2bf41e8?, 0xc0a84fc0c0?}, {0x2579f20?, 0x417d880?})\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/chain.go:27 +0x3a\ngithub.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors.UnaryServerInterceptor.func1({0x2bf41e8, 0xc0a84fc060}, {0x2579f20, 0x417d880}, 0x22f5640?, 0xc0a84e7a60)\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/interceptors/server.go:22 +0x21e\ngithub.com/thanos-io/thanos/pkg/tracing.UnaryServerInterceptor.func1({0x2bf41e8?, 0xc0a84d9f80?}, {0x2579f20, 0x417d880}, 0x0?, 0xffffffffffffffff?)\n\t/app/pkg/tracing/grpc.go:30 +0x88\ngithub.com/grpc-ecosystem/go-grpc-middleware/v2.ChainUnaryServer.func1.1.1({0x2bf41e8?, 0xc0a84d9f80?}, {0x2579f20?, 0x417d880?})\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/chain.go:27 +0x3a\ngithub.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors.UnaryServerInterceptor.func1({0x2bf41e8, 0xc0a84d9c20}, {0x2579f20, 0x417d880}, 0x0?, 0xc0a84e7a80)\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/interceptors/server.go:22 +0x21e\ngithub.com/grpc-ecosystem/go-grpc-middleware/v2.ChainUnaryServer.func1.1.1({0x2bf41e8?, 0xc0a84d9c20?}, {0x2579f20?, 0x417d880?})\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/chain.go:27 +0x3a\ngithub.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).UnaryServerInterceptor.func1({0x2bf41e8, 0xc0a84d9c20}, {0x2579f20, 0x417d880}, 0x7f6a8a07bec8?, 0xc0a84e7aa0)\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-prometheus@v1.2.0/server_metrics.go:107 +0x87\ngithub.com/grpc-ecosystem/go-grpc-middleware/v2.ChainUnaryServer.func1.1.1({0x2bf41e8?, 0xc0a84d9c20?}, {0x2579f20?, 0x417d880?})\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/chain.go:27 +0x3a\ngithub.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors/recovery.UnaryServerInterceptor.func1({0x2bf41e8?, 0xc0a84d9c20?}, {0x2579f20?, 0x417d880?}, 0x7f6a8ae8f5e8?, 0xc0a84e7a40?)\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/interceptors/recovery/interceptors.go:31 +0xa7\ngithub.com/grpc-ecosystem/go-grpc-middleware/v2.ChainUnaryServer.func1.1.1({0x2bf41e8?, 0xc0a84d9c20?}, {0x2579f20?, 0x417d880?})\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/chain.go:27 +0x3a\ngithub.com/grpc-ecosystem/go-grpc-middleware/v2.ChainUnaryServer.func1({0x2bf41e8, 0xc0a84d9c20}, {0x2579f20, 0x417d880}, 0x228f400?, 0x26056e0?)\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/chain.go:36 +0xbe\ngithub.com/thanos-io/thanos/pkg/info/infopb._Info_Info_Handler({0x223fee0?, 0xc000a8d4a0}, {0x2bf41e8, 0xc0a84d9c20}, 0x26056e0?, 0xc0007dae40)\n\t/app/pkg/info/infopb/rpc.pb.go:462 +0x126\ngoogle.golang.org/grpc.(*Server).processUnaryRPC(0xc0005a9500, {0x2c06b00, 0xc04f72bd40}, 0xc0712aeea0, 0xc0007db1d0, 0x410cb60, 0x0)\n\t/go/pkg/mod/google.golang.org/grpc@v1.45.0/server.go:1282 +0xccf\ngoogle.golang.org/grpc.(*Server).handleStream(0xc0005a9500, {0x2c06b00, 0xc04f72bd40}, 0xc0712aeea0, 0x0)\n\t/go/pkg/mod/google.golang.org/grpc@v1.45.0/server.go:1619 +0xa2f\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2()\n\t/go/pkg/mod/google.golang.org/grpc@v1.45.0/server.go:921 +0x98\ncreated by google.golang.org/grpc.(*Server).serveStreams.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.45.0/server.go:919 +0x28a\n"
level=error ts=2023-02-24T08:30:11.783911128Z caller=grpc.go:70 component=receive service=gRPC/server component=receive msg="recovered from panic" panic="runtime error: invalid memory address or nil pointer dereference" stack="goroutine 108075 [running]:\nruntime/debug.Stack()\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x65\ngithub.com/thanos-io/thanos/pkg/server/grpc.New.func1({0x22909c0?, 0x4109510})\n\t/app/pkg/server/grpc/grpc.go:70 +0xda\ngithub.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors/recovery.WithRecoveryHandler.func1.1({0x40d95f?, 0x4f3ba7?}, {0x22909c0?, 0x4109510?})\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/interceptors/recovery/options.go:33 +0x2d\ngithub.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors/recovery.recoverFrom({0x2bf41e8?, 0xc0a3914c60?}, {0x22909c0?, 0x4109510?}, 0xc06ede8f40?)\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/interceptors/recovery/interceptors.go:53 +0x36\ngithub.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors/recovery.UnaryServerInterceptor.func1.1()\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/interceptors/recovery/interceptors.go:27 +0x68\npanic({0x22909c0, 0x4109510})\n\t/usr/local/go/src/runtime/panic.go:884 +0x212\ngithub.com/thanos-io/thanos/pkg/store.(*TSDBStore).TimeRange(0xc06ede90e8?)\n\t/app/pkg/store/tsdb.go:117 +0x14\ngithub.com/thanos-io/thanos/pkg/receive.(*localClient).TimeRange(0x40?)\n\t/app/pkg/receive/multitsdb.go:113 +0x1b\ngithub.com/thanos-io/thanos/pkg/store.(*ProxyStore).Info(0xc00063f3b0, {0x232f1a0?, 0x2504820?}, 0x8000000000000000?)\n\t/app/pkg/store/proxy.go:152 +0x178\ngithub.com/thanos-io/thanos/pkg/store/storepb._Store_Info_Handler.func1({0x2bf41e8, 0xc0a3914f90}, {0x257be60?, 0x417d880})\n\t/app/pkg/store/storepb/rpc.pb.go:1071 +0x78\ngithub.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors.UnaryServerInterceptor.func1({0x2bf41e8, 0xc0a3914f90}, {0x257be60, 0x417d880}, 0x44cf07e80?, 0xc0a334dd10)\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/interceptors/server.go:22 +0x21e\ngithub.com/grpc-ecosystem/go-grpc-middleware/v2.ChainUnaryServer.func1.1.1({0x2bf41e8?, 0xc0a3914f90?}, {0x257be60?, 0x417d880?})\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/chain.go:27 +0x3a\ngithub.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors.UnaryServerInterceptor.func1({0x2bf41e8, 0xc0a3914f30}, {0x257be60, 0x417d880}, 0x22f5640?, 0xc0a3343540)\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/interceptors/server.go:22 +0x21e\ngithub.com/thanos-io/thanos/pkg/tracing.UnaryServerInterceptor.func1({0x2bf41e8?, 0xc0a3914e70?}, {0x257be60, 0x417d880}, 0x0?, 0xffffffffffffffff?)\n\t/app/pkg/tracing/grpc.go:30 +0x88\ngithub.com/grpc-ecosystem/go-grpc-middleware/v2.ChainUnaryServer.func1.1.1({0x2bf41e8?, 0xc0a3914e70?}, {0x257be60?, 0x417d880?})\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/chain.go:27 +0x3a\ngithub.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors.UnaryServerInterceptor.func1({0x2bf41e8, 0xc0a3914c60}, {0x257be60, 0x417d880}, 0x0?, 0xc0a3343560)\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/interceptors/server.go:22 +0x21e\ngithub.com/grpc-ecosystem/go-grpc-middleware/v2.ChainUnaryServer.func1.1.1({0x2bf41e8?, 0xc0a3914c60?}, {0x257be60?, 0x417d880?})\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/chain.go:27 +0x3a\ngithub.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).UnaryServerInterceptor.func1({0x2bf41e8, 0xc0a3914c60}, {0x257be60, 0x417d880}, 0x7f6a89564ff8?, 0xc0a3343580)\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-prometheus@v1.2.0/server_metrics.go:107 +0x87\ngithub.com/grpc-ecosystem/go-grpc-middleware/v2.ChainUnaryServer.func1.1.1({0x2bf41e8?, 0xc0a3914c60?}, {0x257be60?, 0x417d880?})\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/chain.go:27 +0x3a\ngithub.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors/recovery.UnaryServerInterceptor.func1({0x2bf41e8?, 0xc0a3914c60?}, {0x257be60?, 0x417d880?}, 0x7f6a8b791758?, 0xc0a3343520?)\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/interceptors/recovery/interceptors.go:31 +0xa7\ngithub.com/grpc-ecosystem/go-grpc-middleware/v2.ChainUnaryServer.func1.1.1({0x2bf41e8?, 0xc0a3914c60?}, {0x257be60?, 0x417d880?})\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/chain.go:27 +0x3a\ngithub.com/grpc-ecosystem/go-grpc-middleware/v2.ChainUnaryServer.func1({0x2bf41e8, 0xc0a3914c60}, {0x257be60, 0x417d880}, 0x228f400?, 0x26056e0?)\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.2.0.20201207153454-9f6bf00c00a7/chain.go:36 +0xbe\ngithub.com/thanos-io/thanos/pkg/store/storepb._Store_Info_Handler({0x2504820?, 0xc0002cb000}, {0x2bf41e8, 0xc0a3914c60}, 0x26056e0?, 0xc0007dae40)\n\t/app/pkg/store/storepb/rpc.pb.go:1073 +0x126\ngoogle.golang.org/grpc.(*Server).processUnaryRPC(0xc0005a9500, {0x2c06b00, 0xc04f72bd40}, 0xc0bb030480, 0xc0007db020, 0x411a120, 0x0)\n\t/go/pkg/mod/google.golang.org/grpc@v1.45.0/server.go:1282 +0xccf\ngoogle.golang.org/grpc.(*Server).handleStream(0xc0005a9500, {0x2c06b00, 0xc04f72bd40}, 0xc0bb030480, 0x0)\n\t/go/pkg/mod/google.golang.org/grpc@v1.45.0/server.go:1619 +0xa2f\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2()\n\t/go/pkg/mod/google.golang.org/grpc@v1.45.0/server.go:921 +0x98\ncreated by google.golang.org/grpc.(*Server).serveStreams.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.45.0/server.go:919 +0x28a\n"

apiVersion: v1
kind: ConfigMap
metadata:
  name: thanos-receive-hashrings
  namespace: thanos
data:
  thanos-receive-hashrings.json: |
    [
      {
        "hashring": "soft-tenants",
        "endpoints":
        [
            "thanos-receive-0.thanos-receive.thanos.svc.cluster.local:10901",
            "thanos-receive-1.thanos-receive.thanos.svc.cluster.local:10901",
            "thanos-receive-2.thanos-receive.thanos.svc.cluster.local:10901",
            "thanos-receive-3.thanos-receive.thanos.svc.cluster.local:10901",
            "thanos-receive-4.thanos-receive.thanos.svc.cluster.local:10901"
        ]
      }
    ]

---

apiVersion: v1
kind: Service
metadata:
  name: thanos-receive
  namespace: thanos
  labels:
    kubernetes.io/name: thanos-receive
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/path: /metrics
    prometheus.io/port: "10902"
spec:
  ports:
  - name: http
    port: 10902
    protocol: TCP
    targetPort: 10902
  - name: remote-write
    port: 19291
    protocol: TCP
    targetPort: 19291
  - name: grpc
    port: 10901
    protocol: TCP
    targetPort: 10901
  selector:
    kubernetes.io/name: thanos-receive
  clusterIP: None

---

apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    kubernetes.io/name: thanos-receive
  name: thanos-receive
  namespace: thanos
spec:
  replicas: 5
  selector:
    matchLabels:
      kubernetes.io/name: thanos-receive
  serviceName: thanos-receive
  template:
    metadata:
      labels:
        kubernetes.io/name: thanos-receive
        thanos-store-api: "true"
    spec:
      containers:
      - args:
        - receive
        - --grpc-address=0.0.0.0:10901
        - --http-address=0.0.0.0:10902
        - --remote-write.address=0.0.0.0:19291
        - --receive.replication-factor=3
        - --receive.hashrings-algorithm=ketama
        - --objstore.config-file=/etc/thanos/objectstorage.yaml
        - --tsdb.path=/var/thanos/receive
        - --tsdb.retention=12h
        - --receive-forward-timeout=120s
        - --label=receive_replica="$(NAME)"
        - --label=receive="true"
        - --receive.hashrings-file=/etc/thanos/thanos-receive-hashrings.json
        - --receive.local-endpoint=$(NAME).thanos-receive.thanos.svc.cluster.local:10901
        env:
        - name: NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        image: thanosio/thanos:v0.30.2
        livenessProbe:
          failureThreshold: 4
          httpGet:
            path: /-/healthy
            port: 10902
            scheme: HTTP
          periodSeconds: 30
        name: thanos-receive
        ports:
        - containerPort: 10901
          name: grpc
        - containerPort: 10902
          name: http
        - containerPort: 19291
          name: remote-write
        readinessProbe:
          httpGet:
            path: /-/ready
            port: 10902
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 30
        resources:
          limits:
            cpu: "48"
            memory: 200Gi
          requests:
            cpu: "4"
            memory: 16Gi
        volumeMounts:
        - mountPath: /var/thanos/receive
          name: data
          readOnly: false
        - mountPath: /etc/thanos/thanos-receive-hashrings.json
          name: thanos-receive-hashrings
          subPath: thanos-receive-hashrings.json
        - mountPath: /etc/thanos/objectstorage.yaml
          name: thanos-objectstorage
          subPath: objectstorage.yaml
      terminationGracePeriodSeconds: 120
      volumes:
      - configMap:
          defaultMode: 420
          name: thanos-receive-hashrings
        name: thanos-receive-hashrings
      - name: thanos-objectstorage
        secret:
          secretName: thanos-objectstorage
  volumeClaimTemplates:
  - metadata:
      labels:
        app.kubernetes.io/name: thanos-receive
      annotations:
        volume.beta.kubernetes.io/storage-class: "thanos-object-sc"
      name: data
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 200Gi
---

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    dns.gardener.cloud/dnsnames: '*'
    cert.gardener.cloud/purpose: managed
    dns.gardener.cloud/class: garden
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/service-upstream: "true"
    dns.gardener.cloud/ttl: "60"
  name: thanos-receive-ingress
  namespace: thanos
spec:
  rules:
  - host: thanos-receive.thanos.xxxxx.com
    http:
      paths:
      - backend:
          service:
            name: thanos-receive
            port:
              number: 19291
        path: /
        pathType: Prefix

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Object Storage Capacity Optimization #6135

{{title}}

Replies: 13 comments 1 reply

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

Object Storage Capacity Optimization #6135

JayChanggithub Feb 13, 2023

Replies: 13 comments · 1 reply

fpetkovski Feb 13, 2023 Maintainer

JayChanggithub Feb 14, 2023 Author

fpetkovski Feb 15, 2023 Maintainer

JayChanggithub Feb 16, 2023 Author

compact

Query

Receive

Rules

Stores

fpetkovski Feb 16, 2023 Maintainer

JayChanggithub Feb 16, 2023 Author

fpetkovski Feb 16, 2023 Maintainer

JayChanggithub Feb 16, 2023 Author

yeya24 Feb 17, 2023 Maintainer

JayChanggithub Feb 18, 2023 Author

fpetkovski Feb 20, 2023 Maintainer

JayChanggithub Feb 20, 2023 Author

JayChanggithub Feb 23, 2023 Author

JayChanggithub Feb 24, 2023 Author

JayChanggithub
Feb 13, 2023

Replies: 13 comments 1 reply

fpetkovski
Feb 13, 2023
Maintainer

JayChanggithub
Feb 14, 2023
Author

fpetkovski
Feb 15, 2023
Maintainer

JayChanggithub
Feb 16, 2023
Author

fpetkovski
Feb 16, 2023
Maintainer

JayChanggithub
Feb 16, 2023
Author

fpetkovski
Feb 16, 2023
Maintainer

JayChanggithub
Feb 16, 2023
Author

yeya24
Feb 17, 2023
Maintainer

JayChanggithub
Feb 18, 2023
Author

fpetkovski Feb 20, 2023
Maintainer

JayChanggithub
Feb 20, 2023
Author

JayChanggithub
Feb 23, 2023
Author

JayChanggithub
Feb 24, 2023
Author