MELT-TinyLlama-1.1B

Lightweight medical LLM based on TinyLlama 1.1B, fine-tuned on 14+ medical datasets. Designed for resource-constrained deployment where large models like Mixtral are impractical.

Despite its small size, the model achieves a 13.76% improvement over the base TinyLlama across three medical benchmarks (overall average: 27.95% vs. 24.57% base).

Training

• Base model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
• Fine-tuning: LoRA (rank 64, alpha 16), LR 2e-4, 3 epochs
• Datasets: MedQA, MedMCQA, Healthcare Magic 100k, and 11+ others (same as MELT-Mixtral)