Commits on Apr 27, 2022

1. Thread saturation metrics …

   Adds metrics suggested in #488, to record the percentage of time the
   main and FSM goroutines are busy with work vs available to accept new
   work, to give operators an idea of how close they are to hitting
   capacity limits.
   
   We keep 256 samples in memory for each metric, and update gauges (at
   most) once a second, possibly less if the goroutines are idle. This
   should be ok because it's unlikely that a goroutine would go from very
   high saturation to being completely idle (so at worst we'll leave the
   gauge on the previous low value).

   

   boxofrad committed Apr 27, 2022
   Copy the full SHA

   8ac5ab2 View commit details

   Browse the repository at this point in the history

2. Incorporate PR feedback …

   - Much simpler implementation based on an accumulator of sleep time.
     We no longer drop samples when the buffer is full.
   - We now keep 5 previous measurements to smooth out spikes.
   - Rename metrics to `raft.thread.fsm.saturation` and
     `raft.thread.main.saturation`.
   - Remove FSM saturation metric from the `Raft` struct.

   

   boxofrad committed Apr 27, 2022
   Copy the full SHA

   24fc8b5 View commit details

   Browse the repository at this point in the history

3. Round saturation to 2dp to report integer percentages

   

   boxofrad committed Apr 27, 2022
   Copy the full SHA

   5c3d87e View commit details

   Browse the repository at this point in the history

4. Remove smoothing/bucketing in favor of using a sample metric

   

   boxofrad committed Apr 27, 2022
   Copy the full SHA

   1f8b1ad View commit details

   Browse the repository at this point in the history

5. Incorporate @kisunji's comment improvements

   

   boxofrad committed Apr 27, 2022
   Copy the full SHA

   0e9a7c5 View commit details

   Browse the repository at this point in the history