Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[NOT FOR MERGING] Simulator for inference autoscaling #107954

Draft
wants to merge 18 commits into
base: main
Choose a base branch
from
Draft

[NOT FOR MERGING] Simulator for inference autoscaling #107954

wants to merge 18 commits into from
+414 −0

Commits on May 24, 2024

  1. Simulator for inference autoscaling.

    @jan-elastic
    jan-elastic committed May 24, 2024
    Configuration menu
    Copy the full SHA
    938d61e View commit details
    Browse the repository at this point in the history

  2. Account for virtual cores / hyperthreading

    @jan-elastic
    jan-elastic committed May 24, 2024
    Configuration menu
    Copy the full SHA
    72744eb View commit details
    Browse the repository at this point in the history

  3. Improvements

    @jan-elastic
    jan-elastic committed May 24, 2024
    Configuration menu
    Copy the full SHA
    e50c794 View commit details
    Browse the repository at this point in the history

  4. Kalman filter for simple state model for average inference duration a… 

    …s a function of time and allocation count
    @tveasey @jan-elastic
    tveasey authored and jan-elastic committed May 24, 2024
    Configuration menu
    Copy the full SHA
    1d3a30b View commit details
    Browse the repository at this point in the history

  5. Simplify and estimate average duration rather than rate

    @tveasey @jan-elastic
    tveasey authored and jan-elastic committed May 24, 2024
    Configuration menu
    Copy the full SHA
    ee4ec77 View commit details
    Browse the repository at this point in the history

  6. Allow extrapolation

    @tveasey @jan-elastic
    tveasey authored and jan-elastic committed May 24, 2024
    Configuration menu
    Copy the full SHA
    3ebf920 View commit details
    Browse the repository at this point in the history

  7. Correction

    @tveasey @jan-elastic
    tveasey authored and jan-elastic committed May 24, 2024
    Configuration menu
    Copy the full SHA
    bb07bc7 View commit details
    Browse the repository at this point in the history

  8. Tweak

    @tveasey @jan-elastic
    tveasey authored and jan-elastic committed May 24, 2024
    Configuration menu
    Copy the full SHA
    ed9eb0b View commit details
    Browse the repository at this point in the history

  9. Correct the dependency on allocations

    @tveasey @jan-elastic
    tveasey authored and jan-elastic committed May 24, 2024
    Configuration menu
    Copy the full SHA
    29a9e51 View commit details
    Browse the repository at this point in the history

  10. Improved autoscaling simulation

    @jan-elastic
    jan-elastic committed May 24, 2024
    Configuration menu
    Copy the full SHA
    2e9de17 View commit details
    Browse the repository at this point in the history

  11. Move autoscaling code to own class

    @jan-elastic
    jan-elastic committed May 24, 2024
    Configuration menu
    Copy the full SHA
    7296655 View commit details
    Browse the repository at this point in the history

  12. No autodetection of dynamics changes for latency

    @jan-elastic
    jan-elastic committed May 24, 2024
    Configuration menu
    Copy the full SHA
    c513c67 View commit details
    Browse the repository at this point in the history

  13. Init variance to None

    @jan-elastic
    jan-elastic committed May 24, 2024
    Configuration menu
    Copy the full SHA
    d6f9b5c View commit details
    Browse the repository at this point in the history

  14. Measure latency periodically + documentation

    @jan-elastic
    jan-elastic committed May 24, 2024
    Configuration menu
    Copy the full SHA
    eb2aade View commit details
    Browse the repository at this point in the history

  15. Remove unused estimator

    @jan-elastic
    jan-elastic committed May 24, 2024
    Configuration menu
    Copy the full SHA
    d4cf050 View commit details
    Browse the repository at this point in the history

  16. jump -> jumps

    @jan-elastic
    jan-elastic committed May 24, 2024
    Configuration menu
    Copy the full SHA
    ddb3807 View commit details
    Browse the repository at this point in the history

  17. Increase measurement interval to 10secs

    @jan-elastic
    jan-elastic committed May 24, 2024
    Configuration menu
    Copy the full SHA
    3becc71 View commit details
    Browse the repository at this point in the history

  18. Better process variance upon dynamics changes, and propagate dynamics… 

    … changes to the next iteration.
    @jan-elastic
    jan-elastic committed May 24, 2024
    Configuration menu
    Copy the full SHA
    c2b337f View commit details
    Browse the repository at this point in the history