Atlas general alerting review and migration to mimir #3318

QuentinBisson · 2024-03-11T20:48:33Z

Towards #3312

Atlas is planning to migrate our monitoring setup to mimir targetting CAPI only.
This will result in all data being in a single database, instead of the current one-prometheus-per-cluster setup.
Current alerts have to be updated as queries will see all data for all clusters, MC and WC alike, instead of data for one specific cluster at a time.

We already did a lot of work towards this on the current alerts (removed a lot of deprecated alerts and providers, fixed alerts that clearly were not working an so on).

By doing so, we discovered a few things about Mimir itself but also that a chunk of our alert currently do not work on CAPI (e.g. based on vintage only components, deprecated and missing metrics an so on).

To ensure proper monitoring in CAPI and with Mimir, Atlas needs your help!

We would kindly ask all teams to help us out for the following use-cases, ordered in terms of priorities if they can't be performed all at once.

1. Test and fix your teams alerts and dashboards on CAPI clusters.

A lot of the alerts we have do not work on CAPI (e.g. cluster-autoscaler, ebs-csi and external-dns) simply because they are flagged behind the "aws" provider only, or because they rely on metrics of vintage components (cluster_created|upgraded inhibitions).
The specific alerts issue that were identified will be added to the team issues.

Atlas finished reviewing their alerts in prometheus-rules and sloth-rules on CAPI - please report it in the umbrella issue

2. Test and fix your teams alerts and dashboards on Mimir.

We currently have Mimir deployed on Golem for testing of alerts accessible as a datasource in grafana.

Current known/unknown with Mimir are behing written here by @giantswarm/team-atlas but feel free to add what you found.

We request a second round of testing for Mimir because Mimir in inherently different from our vintage monitoring setup.
First,all metrics will be stored in one central place (we are not enabling multi-tenancy yet). This means that:

No two alerts should have the same name
All aggregations (sum, count, ...) must at the very least have cluster_id, provider and pipeline in the by clause
All joins must at the very least have cluster_id, provider and pipeline in the on clause
the promql absent function should be use carefully because this function renders an empty vector so having it empty for all clusters in a MC seems relatively impossible. If you target 1 cluster in particular, this could work (cluster_type="management_cluster" for example but we think it's best to rely on other mechanisms)

Second, for grafana cloud, we rely a lot on external labels (labels added by prometheus when metrics leave the cluster like installation, provider and so on) but data sent from mimir to grafana cloud will not have those external labels anymore so recording rules aggregations and join must contain all eternal labels in the on and by clauses (that was mostly done by atlas but please review)

Third, we know that the alerting link (prometheus query) in opsgenie and slack will not work directly because Mimir does not have a UI per se (hint: it's grafana). The only way to have this source link back is to migrate to mimir's alertmanager but that's a whole over beast that we cannot tacke right now so we advise you, for each alert, to try to find a dashboard can be linked to the alert to help with oncall.

Atlas finished reviewing their alerts in prometheus-rules and sloth-rules and dashboards on Mimir - please report it in the umbrella issue

3. Move away from the old slo framework towards sloth

We deprecated the old slo dashboard a while ago in favor of sloth but teams are not really using it. We would love if you could replace the old slo alerts with sloth-based ones.

Atlas migrated away from the old slo framework using sloth (tested in both CAPI and Mimir) - please report it in the umbrella issue Remove label for old slo framework kube-prometheus-stack-app#393 https://github.com/giantswarm/prometheus-rules/pull/1182/files and https://github.com/giantswarm/sloth-rules/pull/186

4. Test Grafana Cloud dashboards with golem data

As mimir data will be sent to grafana cloud by a single prometheus with no external labels, we would like you to ensure the grafana cloud dashboard that your team owns work on golem.

This is currently blocked by #3159

Atlas fixed their grafana cloud dashboards - please report it in the umbrella issue

5. Move all apps (latest versions) to service monitors

Towards closing this https://github.com/giantswarm/giantswarm/issues/27145

There are still some leftovers (although not a lot) that still need to use a service monitor. Without this, we will not be able to tear down our Prometheus stack.

This is not that much of a priority but the effort should be rather small and easy to finish so feel free to pick this up

To easily find out what is not monitored via service monitors, you can connect to a MC and WC prometheus using opsctl open -i -a prometheus --workload-cluster=<cluster_id> and check out the targets page.
If they are there (be careful to also check out the workload section), they need a servicemonitor :)

Atlas added their missing service monitors - please report it in the umbrella issue

We will of course be here to help you for the migration :)

Further info:

To help you, you can always add alert tests in prometheus-rules, those are great :)

The text was updated successfully, but these errors were encountered:

marieroque · 2024-03-20T14:53:51Z

I'm working on reviewing all rules on CAPI and Mimir installations ONLY.
I'm comparing rules evaluated on grizzly between Mimir datasource and Prometheus datasource.

RECORDING RULES:

Managed Prometheus
Managed Alertmanager

ALERTING RULES:

SLOTH:

AtlasOperatorsReconciliationError

QuentinBisson · 2024-03-20T15:49:57Z

You're missing there servicelevelburnratetoohigh alert

marieroque · 2024-03-21T11:26:03Z

Using mimirtool on grizzly:

Extract metrics used in Mimir ruler:

kubectl port-forward -n mimir svc/mimir-gateway 8080
mimirtool analyze ruler --address=http://localhost:8080 --id=anonymous

and got this file:
metrics-in-ruler.json

Analyze those rules on mimir:

[~]$ mimirtool analyze prometheus --address=http://localhost:8080/prometheus --id=anonymous --ruler-metrics-file=metrics-in-ruler.json
INFO[0002] 155801 active series are being used in dashboards 
INFO[0002] Found 3749 metric names                      
INFO[0017] 331303 active series are NOT being used in dashboards 
INFO[0017] 271 in use active series metric count        
INFO[0017] 3473 not in use active series metric count

and got this file:
prometheus-metrics.json

Analyze those rules on prometheus-grizzly:

[~]$ kubectl port-forward -n grizzly-prometheus prometheus-grizzly-0 9090
[~]$ mimirtool analyze prometheus --address=http://localhost:9090/grizzly --ruler-metrics-file=metrics-in-ruler.json --output=prom-grizzly-output.json
INFO[0004] 149485 active series are being used in dashboards 
INFO[0004] Found 3271 metric names                      
INFO[0017] 287311 active series are NOT being used in dashboards 
INFO[0017] 249 in use active series metric count        
INFO[0017] 3021 not in use active series metric count

and got this file:
prom-grizzly-output.json

marieroque · 2024-03-22T16:37:32Z

I sorted 2 above outputs to be able to compare them:

cat prometheus-metrics.json | jq -S > prometheus-metrics-sort.json
cat prom-grizzly-output.json | jq -S > prom-grizzly-output-sort.json

prometheus-metrics-sort.json
prom-grizzly-output-sort.json

QuentinBisson · 2024-03-26T15:11:57Z

##Old Slo framework

The old slo framework is used here:

https://github.com/giantswarm/prometheus-rules/blob/7f42fafab7b02df4a44e663841dfbd870653bb55/helm/prometheus-rules/templates/recording-rules/service-level.rules.yml#L329
for the prometheus-operator via this label label_giantswarm_io_monitoring_basic_sli

marieroque · 2024-03-26T16:09:22Z

Even if those files give information about metrics used in rules, it's not useful to check if the rules have the same behavior on Mimir and Prometheus.

For now, I didn't find a way to do it "smartly".
I need to edit rules to have results and be able to compare whatever.
And for each rules, it's different. Sometimes, I have to remove the operator > 0 for instance.
Sometimes I have to remove some labels like status_code=~"5..|failed" for instance.
Because If I didn't, the rules execution has no data and I can't compare anything.
😞
I think I'll have to check them one by one manually...

marieroque · 2024-03-27T14:19:05Z

count by (cluster_id, installation, provider, pipeline) (label_replace(up{app="kube-state-metrics", instance=~".*:8080"}, "ip", "$1.$2.$3.$4", "node", "ip-(\\d+)-(\\d+)-(\\d+)-(\\d+).*")) == 0
  or (
        label_replace(
            capi_cluster_status_condition{type="ControlPlaneReady", status="True"},
            "cluster_id",
            "$1",
            "name",
            "(.*)"
          ) == 1
        ) unless on (cluster_id, customer, installation, pipeline, provider, region) (
          count(up{app="kube-state-metrics", instance=~".*:8080"} == 1) by (cluster_id, customer, installation, pipeline, provider, region)
    )

MatchingNumberOfPrometheusAndCluster need to add labels installation, provider, pipeline + rewrite rules to fix it because not working

 (
          sum by(cluster_id) (
            {__name__=~"cluster_service_cluster_info|cluster_operator_cluster_status", status!="Deleting"}
          ) unless sum by(cluster_id) (
            label_replace(
              kube_pod_container_status_running{container="prometheus", namespace!="{{ .Values.managementCluster.name }}-prometheus", namespace=~".*-prometheus"},
              "cluster_id", "$2", "pod", "(prometheus-)(.+)(-.+)"
            )
          )
        ) or (
          sum by (name) (
            capi_cluster_status_phase{phase!="Deleting"}
          ) unless sum by (name) (
            label_replace(kube_pod_container_status_running{container="prometheus",namespace=~".*-prometheus"}, 
            "name", "$2", "pod", "(prometheus-)(.+)(-.+)"
            )
          )
        )
        > 0

Managed Prometheus rewrite recording rules:

(up{app="prometheus-operator-app-prometheus",container="prometheus"}*0)+1
(up{app="prometheus-operator-app-prometheus",container="prometheus"}*-1)+1 == 1

=>

(up{app=~"kube-prometheus-stack-prometheus-operator|prometheus-operator-app-prometheus",container=~"kube-prometheus-stack|prometheus"}*0)+1
(up{app=~"kube-prometheus-stack-prometheus-operator|prometheus-operator-app-prometheus",container=~"kube-prometheus-stack|prometheus"}*-1)+1 == 1

Managed AlertManager rewrite recording rules:

(up{app="prometheus-operator-app-alertmanager", container="alertmanager"}*0)+1
(up{app="prometheus-operator-app-alertmanager",container="alertmanager"}*-1)+1 == 1

=>

(up{app=~"alertmanager|prometheus-operator-app-alertmanager",container="alertmanager"}*0)+1
(up{app=~"alertmanager|prometheus-operator-app-alertmanager",container="alertmanager"}*-1)+1 == 1

marieroque · 2024-04-09T07:46:22Z

Status:

Atlas finished reviewing their alerts in prometheus-rules and sloth-rules on CAPI
Atlas finished reviewing their dashboards on CAPI
Atlas finished reviewing their alerts in prometheus-rules and sloth-rules on Mimir
Atlas finished reviewing their dashboards on Mimir
Atlas finished reviewing their dashboards on Grafana Cloud
Atlas finished migrating from the old SLO framework wip Remove label for old slo framework kube-prometheus-stack-app#393 https://github.com/giantswarm/prometheus-rules/pull/1182/files and https://github.com/giantswarm/sloth-rules/pull/186

QuentinBisson · 2024-05-30T14:11:42Z

All dashboards have been tested on both CAPI and Mimir and they all work.
The angular plugins were mostly removed.

The only remaining issues are:

https://github.com/giantswarm/giantswarm/issues/30857
and the 2 mixin issues for loki and mimir

As we have separate issues for those, I'm closing here

QuentinBisson mentioned this issue Mar 11, 2024

Mimir Migration Readiness for CAPA: Alerting and Dashboard Reviews #3312

Open

41 tasks

QuentinBisson added topic/observability team/atlas Team Atlas component/mimir Planning labels Mar 11, 2024

QuentinBisson removed the Planning label Mar 19, 2024

marieroque self-assigned this Mar 20, 2024

marieroque mentioned this issue Apr 4, 2024

Make Atlas rules compatible with Mimir giantswarm/prometheus-rules#1102

Merged

5 tasks

marieroque removed their assignment Apr 30, 2024

This was referenced May 3, 2024

fix: mimir/loki and tempo recording rules to be able to work with Mimir giantswarm/prometheus-rules#1158

Merged

fix: set proper tags for atlas dashboards giantswarm/dashboards#529

Merged

Add basic linting to dashboards giantswarm/dashboards#531

Merged

QuentinBisson self-assigned this May 13, 2024

This was referenced May 13, 2024

Fix atlas alerts with pint giantswarm/prometheus-rules#1173

Merged

Remove label for old slo framework giantswarm/kube-prometheus-stack-app#393

Merged

Remove atlas slo alerts giantswarm/prometheus-rules#1182

Merged

QuentinBisson mentioned this issue May 29, 2024

Add data source to some atlas dashboards giantswarm/dashboards#552

Merged

1 task

QuentinBisson closed this as completed May 30, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Atlas general alerting review and migration to mimir #3318

Atlas general alerting review and migration to mimir #3318

QuentinBisson commented Mar 11, 2024 •

edited

marieroque commented Mar 20, 2024 •

edited

QuentinBisson commented Mar 20, 2024

marieroque commented Mar 21, 2024 •

edited

marieroque commented Mar 22, 2024

QuentinBisson commented Mar 26, 2024

marieroque commented Mar 26, 2024

marieroque commented Mar 27, 2024 •

edited

marieroque commented Apr 9, 2024 •

edited by QuentinBisson

QuentinBisson commented May 30, 2024

Atlas general alerting review and migration to mimir #3318

Atlas general alerting review and migration to mimir #3318

Comments

QuentinBisson commented Mar 11, 2024 • edited

1. Test and fix your teams alerts and dashboards on CAPI clusters.

2. Test and fix your teams alerts and dashboards on Mimir.

3. Move away from the old slo framework towards sloth

4. Test Grafana Cloud dashboards with golem data

5. Move all apps (latest versions) to service monitors

marieroque commented Mar 20, 2024 • edited

QuentinBisson commented Mar 20, 2024

marieroque commented Mar 21, 2024 • edited

marieroque commented Mar 22, 2024

QuentinBisson commented Mar 26, 2024

marieroque commented Mar 26, 2024

marieroque commented Mar 27, 2024 • edited

marieroque commented Apr 9, 2024 • edited by QuentinBisson

QuentinBisson commented May 30, 2024

QuentinBisson commented Mar 11, 2024 •

edited

marieroque commented Mar 20, 2024 •

edited

marieroque commented Mar 21, 2024 •

edited

marieroque commented Mar 27, 2024 •

edited

marieroque commented Apr 9, 2024 •

edited by QuentinBisson