MLA - Partial statistics

[clickersmudge]

Hi,
We installed the MLA stack on the Master/ Seed cluster, and MLA for the user clusters.

Master / Seed

1. Securing System Services

2. IAP

3. Grafana
   https://grafana.kubermatic.example.com
   
   Working

User cluster

1. Enabled MLA

kubectl edit kubermaticconfiguration --namespace kubermatic kubermatic
  spec.featureGates.UserClusterMLA: true
nano seed.yaml
  spec.mla.userClusterMLAEnabled: true
kubectl apply --filename seed.yaml

2. We set up external repositories on Wasabi S3.
   

3. Installation

./kubermatic-installer deploy usercluster-mla --config kubermatic.yaml --helm-values values.mla.yaml \
  --mla-force-secrets \
  --mla-skip-minio \
  --mla-skip-minio-lifecycle-mgr

By the way, Minio, despite the "--mla-skip-minio" flag, installs anyway.

4. Grafana
   https://grafana.kubermatic.kubermatic.example.com
   General / Kubernetes Overview Dashboard
   

Problem

Other dashboards do not work.

All pods in namespace "mla" work.
Not sure where to debug?

[wurbanski]

Hi,

By the way, Minio, despite the "--mla-skip-minio" flag, installs anyway.

Thank you, we will verify this and fix the problem.

All pods in namespace "mla" work.
Not sure where to debug?

Have you tried scheduling some more pods on the cluster? system pods should be excluded from the dashboards by design.

[clickersmudge]

Hi Wojtek we will check this. / Cześć Wojtek, sprawdzimy.

The strange thing is that the default dashboard displays "node and namespace" correctly.

But "General / Nodes Overview" not working.

[wurbanski]

Cześć Wojtek, sprawdzimy.

👋 :)

But "General / Nodes Overview" not working.

Do you have node-exporter add-on enabled for your user cluster? It is required for for the MLA dashboards to work correctly.

[clickersmudge]

Yes. We added node-exporter and kube-state-metrics to the clusters.

[clickersmudge]

Hi,
We have been debugging for several days and cannot understand the cause of the error.

Dubugging scenario

Uninstalling old MLA

helm list --namespace mla
helm uninstall --namespace mla alertmanager-proxy consul cortex grafana loki-distributed minio minio-lifecycle-mgr mla-secrets
kubectl delete namespace mla

values.mla.yaml

Default values

mlaSecrets:
  minio:
    ## Set enabled to false if you want to re-use an existing secret for minio.
    enabled: true
    ## Set default accesskey, secretkey
    ## AccessKey and secretKey is generated when not set
    accessKey: "accesskey"
    secretKey: "secretkey"
  grafana:
    ## Set enabled to false if you want to re-use an existing secret for grafana.
    enabled: true
    adminUser: admin
    adminPassword: strongpassword

Install

With default Minio. Without external S3.

./kubermatic-installer deploy usercluster-mla --config kubermatic.yaml --helm-values values.mla.yaml \
  --mla-force-secrets

Results

Problem

Dashboard is empty.

Cannot switching between Grafana Organizations as in the documentation (https://docs.kubermatic.com/kubermatic/v2.22/tutorials-howtos/monitoring-logging-alerting/user-cluster/user-guide/). Switching button missing.
https://docs.kubermatic.com/kubermatic/v2.22/tutorials-howtos/monitoring-logging-alerting/user-cluster/user-guide/
Log in via the KKP button.

[clickersmudge]

Hi,
@wurbanski
I wanted to ask if you are sure the dashboards in the "charts/mla/grafana/files" folder are working out of the box.

Dashboards debugging

kkp-kubernetes-nodes.json (not working)
kkp-kubernetes-nodes-overview.json (not working)
kkp-kubernetes-pods.json (not working)
kkp-kubernetes-statefulsets.json (not working)
kubernetes-overview.json (working, but not all)

kubernetes-overview.json (working, but not all)

Once installed

Pods CPU usage does not work.

We did a bit of searching on the web and in reference to this prometheus-operator/kube-prometheus#1695 topic we have changed:

sum (rate (container_cpu_usage_seconds_total{image!="",name=~"^k8s_.*",kubernetes_io_hostname=~"^$Node$",namespace=~"^$namespace$"}[1m])) by (pod)

to

sum (rate (container_cpu_usage_seconds_total{image!="",kubernetes_io_hostname=~"^$Node$",namespace=~"^$namespace$"}[1m])) by (pod)

And now it works.

kkp-kubernetes-pods.json (not working)

We noticed that the working dashboard (kubernetes-overview.json) has in the variables.

label_values(namespace)

Other e.g. kkp-kubernetes-pods.json

label_values(kube_pod_info, namespace)

This query does not work.

[clickersmudge]

Solution

We uninstalled node-exporter and kube-state-metrics installed by KKP.

We installed charts on the test cluster from the "charts/monitoring/node-exporter" and "charts/monitoring/kube-state-metrics"folder.
It seems to work, and those installed by KKP cause some problems. We'll final test tomorrow....

[wurbanski]

Hi,

Sorry for responding late. Thanks for your input, this solution already brings a lot of information as to what exactly is wrong - I'll verify it on our end as well and get back with more info.

[clickersmudge]

Hi,
Thanks for your interest.
We discovered that the problem is with 'kube-state-metrics' and 'node-exporter'. Those installed from the panel do not work.

We manually installed the version from the KKP folder "charts/monitoring/kube-state-metrics/" and "charts/monitoring/node-exporter/". The stats from "/kube-state-metrics" are working. Those from "node-exporter" do not. We think they are installing too old versions of charts.