fix(dispatcher): prevent possible dispatcher bad behavior

[sauyon]

Previously, it was possible that if request load was fast enough and max latency set low enough, the dispatcher would enter a loop of waiting for new requests while cancelling the oldest ones while doing no work, this change seeks to fix that.

• The optimizer values for slope and intercept are now bounded below at 0; we may want to eventually try to collect cases where the linear optimizer does this, but for now we do this to avoid the most pathological cases.
• If the first request is in danger of being cancelled, we now stop waiting and begin processing immediately.

[codecov]

Codecov Report

Merging #3646 (06bcaab) into main (f503a68) will increase coverage by 0.14%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main    #3646      +/-   ##
==========================================
+ Coverage   31.74%   31.89%   +0.14%     
==========================================
  Files         149      149              
  Lines       12149    12094      -55     
  Branches     2001     1984      -17     
==========================================
  Hits         3857     3857              
+ Misses       8008     7953      -55     
  Partials      284      284              

Impacted Files	Coverage Δ
src/bentoml/_internal/marshal/dispatcher.py	0.00% <ø> (ø)

[bojiang]

@sauyon Is this ready? Maybe let's discuss tomorrow so that we can merge this and another related PR, shall we?

[sauyon]

@sauyon Is this ready? Maybe let's discuss tomorrow so that we can merge this and another related PR, shall we?

Sounds good, when do you want to chat?