[broker][monitoring] add message ack rate metric for consumer

[tjiuming]

Motivation

Currently, we have dispatch rate metrics but don't have consumer acknowledgment-related metrics.
Which not easy to troubleshoot issues like some consumers stop acknowledging messages.

Having an ack rate metric would be very useful to determine if our consumers are processing messages properly.

Verifying this change

• Make sure that the change passes the CI checks.

Documentation

Check the box below or label this PR directly.

Need to update docs?

• doc-required
  (Your PR needs to update docs and you will update later)

• no-need-doc
  (Please explain why)

• doc
  (Your PR contains doc changes)

• doc-added
  (Docs have been already added)

• doc-complete

[github-actions]

@tjiuming:Thanks for your contribution. For this PR, do we need to update docs?
(The PR template contains info about doc, which helps others know more about the changes. Can you provide doc-related info in this and future PR descriptions? Thanks)

[github-actions]

@tjiuming:Thanks for providing doc info!

[tjiuming]

Tests to be completed.

[tjiuming]

/pulsarbot run-failure-checks

[codelipenghui]

/pulsarbot run-failure-checks

[mattisonchao]

Hi @tjiuming
It looks like many conflicts in branch-2.9. I'm not sure if I missing some dependency PR need to merge before this.
Could you please push a new PR for branch-2.9?

[dave2wave]

Why is something that is NOT A BUG FIX being cherry picked to 2.9 and 2.8?

If there is any difficulty this is a risky change that might break an old branch.

[michaeljmarshall]

add message ack rate metric for consumer

@tjiuming - can you please provide more motivation for this addition?

[codelipenghui]

@michaeljmarshall @dave2wave I have updated the motivation of this PR

Currently, we don't have message acknowledgment metrics, which is hard to find the ack-related issues.
With this improvement, users can add alerts and dashboards to the acknowledgment behavior of consumers.
So that they can know if consumers are processing messages properly.

It's not a new feature, just to improve the observability of the acknowledgment.
The metrics and logs can help to find the problem.

And I have mentioned this part on the mailing list https://lists.apache.org/thread/wkm1slrg341kbq7m83nms97df28kl4of
But looks like only @michaeljmarshall and I join the discussion. thanks @michaeljmarshall You have started a valuable discussion. We're a little fuzzy on the cherry-picking, I think we can get a clear cherry-pick criteria on the mailing list.
If the metrics and logs cherry-pick are not acceptable, I think we can discuss the mailing list first. Just based on my previous experience, without the logs and metrics, users are hard to detect problems and collect more context about a BUG. The logs and metrics can help pulsar's developers understand the problem.