feat: prioritizing incoming messages based on the priority level of channel IDs

[staheri14]

Part of #1192
This PR targets the testground/big-blonks-base branch and mirrors the changes introduced in PR #1228. The goal of this PR is to conduct an experiment within testground, aligned with current testing efforts regarding configuration settings, to assess potential improvements in performance metrics such as block time, and throughput. If the results indicate a positive impact on these metrics, we will consider merging PR #1228, which is open against the main branch

[evan-forbes]

ran branch yesterday and got some interesting results. In a separate experiment, we also tried a version of send prioritization, which got similar results. Both decreased the total observed vote latency to similar levels! however, in both cases throughput was not increased.

[staheri14]

Thank you, Evan, for conducting the experiment and sharing the results. I have a couple of questions:

1. What mechanism is implemented for the send-priority plot? Is it the utilization of multiple connections per peer?
2. When inspecting the results, the end-to-end latency of votes in the receive-priority plot is capped at 7 seconds, whereas in the send-priority plot, it extends up to 15 seconds (observed up to a height of around 60 in both plots). Would it be accurate to infer that prioritizing received messages had a greater impact on reducing latency compared to prioritizing sends?
3. In both experiments, the throughput decreased compared to the baseline. So, not only did it fail to improve, but it also worsened. I had expected to observe an improvement in the throughput. It would be insightful to examine a plot for the block propagation latency. If latency has decreased for the block parts as well, then an improvement in the throughput seems inevitable, unless there has been an increase in block propagation delay.

[evan-forbes]

the send prio was done last second as a hacky version of absolute priority. This is where specific channels were deemed "critical", and always favored

46d7d46

Would it be accurate to infer that prioritizing received messages had a greater impact on reducing latency compared to prioritizing sends?

more data is needed, but yeah it definitely looks like that, which is really useful information!

So, not only did it fail to improve, but it also worsened. I had expected to observe an improvement in the throughput. It would be insightful to examine a plot for the block propagation latency. If latency has decreased for the block parts as well, then an improvement in the throughput seems inevitable, unless there has been an increase in block propagation delay.

Same! and yeah, definitely! It's very interesting. We should query the trace data and find out.

[staheri14]

I conducted a further analysis of the collected trace data to evaluate the impact of message prioritization on specific metrics, aiming to understand the reasons behind the observed lack of improvement in throughput when comparing control-1 and prior-1 experiments. Despite a decrease in average throughput, the throughput across different block sizes showed inconsistent trends compared to the overall average. Below are the plots visualizing throughput for varying block sizes:

In these experiments, I normalized the datasets by trimming them to match the minimum of the last recorded height from both experiments, which was 66.

Control-1 Experiment Observations:

Throughput begins to decline beyond a 31 MB block size. This trend suggests a possible performance bottleneck under standard conditions or could merely be a random fluctuation.

Prior-1 Experiment Observations:

In contrast to the control-1 experiment, there is an improvement in throughput as block sizes increase, indicating that message prioritization MAY help mitigate performance issues at higher loads.

Final Observation:
At a 71 MB block size, the performance of both experiments converged, suggesting that the benefits of message prioritization MAY become more evident at higher load levels.

Nevertheless, further profiling would be required to come to a more definitive conclusion.