_LoRAX: Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs_  LoRAX (LoRA eXchange) is a framework that allows users to serve thousands of fine-tuned models on a single GPU, dramatically reducing the cost of serving without compromising on throughput or latency.📖 Table of contents📖 Table of contents 🌳 Features 🏠 Models 🏃♂️ Getting Started - Requirements -…
Source code on GitHub.
Repository Info
README
_LoRAX: Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs_
  
LoRAX (LoRA eXchange) is a framework that allows users to serve thousands of fine-tuned models on a single GPU, dramatically reducing the cost of serving without compromising on throughput or latency.
📖 Table of contents
- 📖 Table of contents
- 🌳 Features
- 🏠 Models
- 🏃♂️ Getting Started - Requirements - Launch LoRAX Server - Prompt via REST API - Prompt via Python Client - Chat via OpenAI API - Next steps
- 🙇 Acknowledgements
- 🗺️ Roadmap
🌳 Features
- 🚅 Dynamic Adapter Loading: include any fine-tuned LoRA adapter from HuggingFace, Predibase, or any filesystem in your request, it will be loaded just-in-time without blocking concurrent requests. Merge adapters per request to instantly create powerful ensembles.
- 🏋️♀️ Heterogeneous Continuous Batching: packs requests for different adapters together into the same ba...