Update Grafana agent down alert

[mshahzeb]

Fixes: https://github.com/grafana/support-escalations/issues/10404

The alert does not fire if the up metric is not scraped which can happen if the agent goes down.

[Dasomeone]

LGTM, thanks for addressing this!

[v-zhuravlev]

think it will fire only there is NO SINGLE metric up with label integrations/agent. Is this what expected?

[mshahzeb]

@v-zhuravlev

This is supposted to fire if there is an up metric and it is showing 0 or if the up metric is missing (If the agent service is killed)

expr: |||
              up{
                job="integrations/agent",
              } == 0
              or absent(
                up{
                  job="integrations/agent",
                }
              )
            |||,
            'for': '5m',

Related issue: grafana/support-escalations#10404

[mshahzeb]

After internal discussion, we have discussed not to change this behavior because:

• The service can be down for any number of reasons (e.g maintenance, planned shutdown) or issues with metric ingestion (There might be an issue on Grafana side due to which the metric is not being ingested)
• Therefore it is safe to trigger this only if the up metric is being reported and it is reporting 0

This alert is very prone to false positives and we are not comfortable in making this change globally for all everyone.