Why thanos query is much slower than query prometheus directly?

[WystanW]

I'm using a thanos cluster with 3 machines(in k8s)? But query the thanos cluster is much slower than query single prometheus instance and each thanos query install take much more memory than prometheus

thanos version: 0.11.0

[GiedriusS]

Why are you using such an old version? There has been an uncountable number of improvements since then. Please try with a newer version.

[WystanW]

Why are you using such an old version? There has been an uncountable number of improvements since then. Please try with a newer version.

I'm using thanos query frontend to accelerate range query, but I'm confused in caching result in memory. I use K8s Pod to start thanos query frontend, yaml is as follows

apiVersion: v1
kind: Service
metadata:
  name: thanos-query-frontend-nodeport
  namespace: default
  labels:
    app.kubernetes.io/name: thanos-query-frontend
spec:
  type: NodePort
  externalTrafficPolicy: Cluster
  selector:
    app.kubernetes.io/name: thanos-query-frontend
  ports:
    - name: http
      protocol: TCP
      port: 9090
      targetPort: 9090
      nodePort: 30302
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: thanos-query-frontend
  namespace: default
  labels:
    app.kubernetes.io/name: thanos-query-frontend
spec:
  replicas: 3
  selector:
    matchLabels:
      app.kubernetes.io/name: thanos-query-frontend
  template:
    metadata:
      labels:
        app.kubernetes.io/name: thanos-query-frontend
    spec:
      tolerations:
      - key: "node-role.kubernetes.io/master"
        operator: "Exists"
        effect: "NoSchedule"
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app.kubernetes.io/name
                  operator: In
                  values:
                  - thanos-query-frontend
              topologyKey: kubernetes.io/hostname
            weight: 100
      containers:
      - args:
        - query-frontend
        - --log.level=debug
        - --http-address=0.0.0.0:9090
        - --labels.partial-response
        - --query-range.partial-response
        - --query-range.split-interval=1h
        - --query-frontend.downstream-url=http://thanos-query-headless.default.svc.cluster.local:9090
        - |-
          --query-range.response-cache-config="config":
              max_size: "10737418240"
              validity: 6h
          "type": "IN-MEMORY"
        image: mirrors.tencent.com/yottadb_release/thanos:v0.22.0
        livenessProbe:
          failureThreshold: 4
          httpGet:
            path: /-/healthy
            port: 9090
            scheme: HTTP
          periodSeconds: 30
        name: thanos-query-frontend
        ports:
        - containerPort: 9090
          name: http
        readinessProbe:
          failureThreshold: 20
          httpGet:
            path: /-/ready
            port: 9090
            scheme: HTTP
          periodSeconds: 5
        terminationMessagePolicy: FallbackToLogsOnError
      terminationGracePeriodSeconds: 120
      restartPolicy: Always

Is the filed --query-range.response-cache-config right in this yaml? I cannot found any memory growth in query frontend pod

[WystanW]

Why are you using such an old version? There has been an uncountable number of improvements since then. Please try with a newer version.

I have update thanos to v0.22.0 for all service

[yeya24]

Ideally it is slower than Prometheus and it is expected. As you can see, querying Prometheus only requires one hop but querying Thanos requires at least 2 hops. (If you are using query frontend then it is 3 hops).

[GiedriusS]

The answer here is probably that we don't have query push down yet: #305. So, Thanos has to transfer all of the data over the wire if the components are not on the system, and calculate the result.