Migrated to newer prometheus operator

[Zelldon]

Description

Our current gke clusters using an outdated and deprecated prometheus operator. We should migrate to the newer maintained version here. As part of this we should update our setup docs

Checking the status of the current installation shows the deprecation status as well.

$ helm status metrics
NAME: metrics
LAST DEPLOYED: Mon Jun 28 08:50:37 2021
NAMESPACE: default
STATUS: deployed
REVISION: 24
NOTES:

---

*** DEPRECATED ****

---

• stable/prometheus-operator chart is deprecated.
• Further development has moved to https://github.com/prometheus-community/helm-charts
• The chart has been renamed kube-prometheus-stack to more clearly reflect
• that it installs the kube-prometheus project stack, within which Prometheus
• Operator is only one component.

The Prometheus Operator has been installed. Check its status by running:
kubectl --namespace default get pods -l "release=metrics"

Visit https://github.com/coreos/prometheus-operator for instructions on how
to create & configure Alertmanager and Prometheus instances using the Operator.

$ helm list
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
metrics default 24 2021-06-28 08:50:37.787649576 +0200 CEST deployed prometheus-operator-9.3.2 0.38.1

[npepinpe]

I would propose to migrate to the managed Prometheus service from GCP instead to reduce maintenance. This means installing Grafana as a Helm chart alone, but that's hopefully less work to migrate to.

I will for now leave it in the backlog however and we can look into it for Q3, when (hopefully) SREs are mostly through migrating the prod monitoring stack.

[Zelldon]

I guess would be nice to tackle this sooner than later. Today we run again into an issue with Prometheus, since it was not able to boot because the WAL was too big and the readiness probe caused an crashloop.

Found a good guide for migrating, seem to be not that hard
https://cloud.google.com/stackdriver/docs/managed-prometheus/setup-managed#config-mgd-collection

Example for migrating the service monitor https://cloud.google.com/stackdriver/docs/managed-prometheus/setup-managed#gmp-servicemonitor

One thing we need to keep in mind that this will increase our costs, we need to verify that I guess first https://cloud.google.com/stackdriver/pricing#mgd-prometheus-pricing-summary