thanos-receive ingress #193
Labels
Comments
kakkoyun
added
help wanted
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
{{ message }}
kakkoyun
added
help wanted
Hello everyone, can somebody help me with
ingressconfiguration for thanos-receive?I have pretty good working thanos stack running on minikube, but when I deploy it to testing k8s env., I have a lot errors in logs:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/auth-realm: Authentication Required
# nginx.ingress.kubernetes.io/backend-protocol: GRPC
nginx.ingress.kubernetes.io/proxy-buffer-size: "128k"
nginx.ingress.kubernetes.io/proxy-connect-timeout: "40"
nginx.ingress.kubernetes.io/proxy-buffers-number: "8"
nginx.ingress.kubernetes.io/proxy-buffering: "on"
nginx.ingress.kubernetes.io/proxy-max-temp-file-size: "1024m"
nginx.ingress.kubernetes.io/auth-secret: prometheus-auth
nginx.ingress.kubernetes.io/auth-type: basic
nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
nginx.ingress.kubernetes.io/proxy-body-size: "20m"
name: thanos-receive
namespace: monitoring
spec:
rules:
http:
paths:
serviceName: thanos-receive
servicePort: 19291
path: /
tls:
secretName: thanos-tls-certs
logs of ingress pod:
2021/02/24 09:19:32 [error] 89#89: *1227 upstream timed out (110: Connection timed out) while connecting to upstream, client: 1.2.3.4, server: receive-thanos.sandbox.k8s.sandbox.example.com, request: "POST /api/v1/receive HTTP/1.1", upstream: "http://100.96.24.19:19291/api/v1/receive", host: "receive-thanos.sandbox.k8s.sandbox.example.com"logs of receive:
level=error ts=2021-02-24T09:37:28.552040714Z caller=handler.go:330 component=receive component=receive-handler err="context deadline exceeded" msg="internal server error" level=debug ts=2021-02-24T09:37:20.829019499Z caller=handler.go:315 component=receive component=receive-handler msg="failed to handle request" err="context deadline exceeded"logs of prometheus
ts=2021-02-24T10:06:20.725Z caller=dedupe.go:112 component=remote level=warn remote_name=b29d86 url=https://receive-thanos.sandbox.k8s.sandbox.example.com/api/v1/receive msg="Failed to send batch, retrying" err="server returned HTTP status 502 Bad Gateway: <html>"The text was updated successfully, but these errors were encountered: