Expose paused and retired workers separately in prometheus

[phofl]

Closes #xxxx

• Tests added / passed
• Passes pre-commit run --all-files

cc @ntabris for the grafana dashboards

[ntabris]

Having paused and retiring and paused_or_retiring makes things more complicated for me, because various things would need to include that iff paused and retiring are not present (and not include if they are, otherwise we'd be double counting).

Thoughts about removing paused_or_retiring? I know this would be a breaking change in some sense, but it's also kinda a breaking change to have more non-exclusive states.

[fjetter]

Thoughts about removing paused_or_retiring? I know this would be a breaking change in some sense, but it's also kinda a breaking change to have more non-exclusive states.

I think we don't have any strong preferences about keeping/removing the paused_or_retiring metric. Can you elaborate how adding those would be a breaking change?

[ntabris]

I think we don't have any strong preferences about keeping/removing the paused_or_retiring metric. Can you elaborate how adding those would be a breaking change?

I said "kinda". It messes up anything that assumes states other than "connected" are exclusive, or that (eg) a chart of all states other than "connected" would make sense.

Instead, one would need logic that includes paused_or_retiring or [paused and retiring] but not both... which isn't very straightforward in Prometheus (I'm still thinking about how to do this).

[fjetter]

Instead, one would need logic that includes paused_or_retiring or [paused and retiring] but not both... which isn't very straightforward in Prometheus (I'm still thinking about how to do this).

Right, that makes sense. It's a shame we didn't introduce the split metrics earlier. For now, we're mostly interested in the paused metric because this may highlight a problematic cluster behavior (too small in size) while the retiring signal is a little bit of noise and is usually harmless.

So I don't have a solution for the "nice chart" problem but when using this as a tag, the retired metric as a standalone thing already makes sense.

[github-actions]

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

    29 files      29 suites   11h 4m 38s ⏱️
 4 060 tests  3 948 ✅   109 💤 3 ❌
54 958 runs  52 546 ✅ 2 409 💤 3 ❌

For more details on these failures, see this check.

Results for commit 97675e9.

♻️ This comment has been updated with latest results.

[phofl]

I'd be fine with removing the paused_or_retiring metric, my main concern was backward compatibility.

We mostly care about paused as @fjetter said