v1.37.16
What's Changed
- fix - allow non master key to access llm_utils_routes by @ishaan-jaff in #3710
- fix(bedrock_httpx.py): move anthropic bedrock calls to httpx by @krrishdholakia in #3708
- [Feat] Admin UI - use
base_modelfor Slack Alerts by @ishaan-jaff in #3713 - [Admin UI] show max input tokens on UI by @ishaan-jaff in #3714
- fix(proxy_server.py): fix setting model id for db models by @krrishdholakia in #3715
Full Changelog: v1.37.14...v1.37.16
Docker Run LiteLLM Proxy
docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.37.16
Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat
Load Test LiteLLM Proxy Results
| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
|---|---|---|---|---|---|---|---|---|---|
| /chat/completions | Failed ❌ | 9 | 10.2880556709407 | 1.5629325106711098 | 1.5629325106711098 | 468 | 468 | 7.436624999968444 | 83.99098699999286 |
| /health/liveliness | Failed ❌ | 8 | 10.80103857402248 | 15.632664706092875 | 15.632664706092875 | 4681 | 4681 | 6.298579000031168 | 1272.475381999982 |
| /health/readiness | Failed ❌ | 8 | 10.780497224867714 | 15.712815091255495 | 15.712815091255495 | 4705 | 4705 | 6.286180000017794 | 650.4576310000232 |
| Aggregated | Failed ❌ | 8 | 10.766867369799249 | 32.90841230801948 | 32.90841230801948 | 9854 | 9854 | 6.286180000017794 | 1272.475381999982 |
Contributors
krrishdholakia and ishaan-jaff