Home / AI Models / KOS-V4
Model Model family text-generation cc-by-nc-sa-4.0

KOS-V4

🤗 Hugging Face

Details

Parameters 3,015,362,560
License cc-by-nc-sa-4.0

Variants: Kentucky-Open-Science/KOS-V4-Instruct, Kentucky-Open-Science/KOS-V4-Instruct-GGUF (GGUF), Kentucky-Open-Science/KOS-V4-Base

license: cc-by-nc-sa-4.0
library_name: transformers
pipeline_tag: text-generation
language:

  • en
    tags:
  • clinical
  • medical
  • instruction-following
  • tool-calling
  • function-calling
  • KOS-V4

  • from-scratch

<div align="center">
<img src="scratch_llm.png" alt="Scratch LLM" width="400"/>
</div>

Code name: Scratch. The KOS-V4 series is nicknamed Scratch LLM: it was trained completely from scratch
by a small team on a fraction of the data and compute of commercial models. It is not a frontier model.

⚠️ Research use only. This model is provided for research purposes only and must not be used for any commercial,
clinical, legal, or production-grade applications. The user assumes all risks associated with its use.

KOS-V4-Instruct — a from-scratch 3B that reaches original-ChatGPT-level instruction-following

KOS-V4-Instruct is an open-weights 3B language model trained completely from scratch by a University of
Kentucky College of Medicine team (Office for Research,
Center for Clinical and Translational Sciences). It is a decoder-only transformer
(Qwen3 architecture, bespoke 3B config) optimized for instruction following and tool / function calling. Its
instruction ability comes from GRPO reinforcement learning on a from-scratch clinical base.

IFEval reported as strict-avg = (prompt-level strict + instruction-level strict) / 2
— the exact metric the Hugging Face Open LLM Leaderboard publishes as "IFEval."

IFEval strict-avg model who built it, and how
64.7 Qwen2.5-3B-Instruct Alibaba, ~18 trillion tokens
61.6 KOS-V4-Instruct (ours) University research team, 180B tokens, 24 GPUs
55.9 GPT-3.5-turbo-1106 (the original ChatGPT) OpenAI, ~10,000-GPU supercomputer

KOS-V4-Instruct clears the original GPT-3.5-turbo generation (55.9) and lands within ~3 points of the
commercially trained Qwen2.5-3B (64.7). It also adds real tool / function calling (official BFCL 72.75/73/60.5),
which the original ChatGPT lacked at launch — though modern small models score higher there.

Core specifications

Attribute Detail
Architecture Decoder-only Transformer (Qwen3ForCausalLM), Grouped-Query Attention
Parameters 3.015 B
Hidden / Layers 3072 / 28
Attention 24 query / 8 KV heads (GQA 3:1), head_dim 128, per-head QK-RMSNorm
Feed-forward SwiGLU, intermediate 8192
Vocabulary 32,000, custom medical byte-level BPE
Context length 24,576 (max_position_embeddings 65,536)
Position encoding RoPE, θ = 25,000 (pin on export)
Precision bfloat16
Chat template ChatML (<|im_start|> / <|im_end|>, eos = <|im_end|>)
Pretraining tokens 180.3 B (English medical/biomedical + web)

Quickstart (Hugging Face Transformers)

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "Kentucky-Open-Science/KOS-V4-Instruct"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16, device_map="auto")

messages = [{"role": "user", "content": "List three contraindications for ibuprofen. Answer in exactly 3 bullet points."}]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(inputs, max_new_tokens=256, do_sample=False)
print(tok.decode(out[0][inputs.shape[1]:], skip_special_tokens=True))

Serve with vLLM / TGI as a standard Qwen3 causal LM. Pin RoPE θ = 25000 on any GGUF/quantized export.

Prompt / chat format (ChatML)

<|im_start|>system
{system (optional)}<|im_end|>
<|im_start|>user
{user}<|im_end|>
<|im_start|>assistant
{response}<|im_end|>

<|im_end|> is the true eos. Tool calling uses <tool_call> / <tool_response> tokens (pass tools via the chat
template tools= argument).

Pre-training (the KOS-V4 base)

This model is fine-tuned from KOS-V4-Base — the
from-scratch pretrained foundation summarized here.

Trained from scratch, not distilled or continued. Pure next-token cross-entropy (no auxiliary losses), AdamW,
peak LR 3.0e-4 cosine, 1 epoch, seq 24,576 (whole-document neat-packing), bf16 + FlashAttention-2, 305,613 steps /
180.3 B token-positions. Data: English-only, medical/biomedical-first, 49 sources / 130 M chunks (PubMed Central
77 B, FineWeb-Edu 51 B, mMedC-en 6.3 B, BlueScrubs 4.6 B, + ~40 smaller clinical/ontology sources).

Disclosed issues: 35 % duplicate tokens (a FineWeb-Edu build bug + PMC repetition; a deduped corpus is ready but
was not trained); 38 "sink" BPE merges removed surgically (vocab stays 32,000, ids gated as bad_words_ids); RoPE θ
trained at 25,000 (a planned 10,000 was never applied).

Post-training (this model)

SFT: full-parameter (not LoRA), ChatML, LR 7e-5, 1 epoch, NEFTune α=5; corpus ~0.9 M rows — a stratified
medical+tool+general majority, a 23-family instruction taxonomy (verifier-in-the-loop; only responses passing the
official IFEval checker kept), xLAM function-calling, and grounded cite/abstain RAG. Forgetting gate (broad ppl ≤1.5×).

RL (GRPO via verl): deterministic verifiable rewards, no reward model, no LLM judge; the primary reward runs
the official IFEval checker. GRPO lifted the official IFEval strict-avg from 49.4 (SFT base) to 61.6 at step 1120
(shipped) — the argmax over all RL checkpoints (sha256-verified as the released weights).

Evaluation & benchmarks

Benchmarks are official-suite only: IFEval via the EleutherAI lm-evaluation-harness 0.4.12 (`task ifeval, apply_chat_template=True

  • greedy, task-default max_gen_toks=1280), BFCL via the officialbfcl_eval`. Peer strict-avg values are the Open LLM Leaderboard’s "IFEval" (strict). Our 61.6
    is measured on our own copy of that harness; it calibrates cleanly (we measure Qwen2.5-3B at 64.0 vs the leaderboard’s
    64.7, a 0.7-pt gap).

IFEval in context (strict-avg, models our size or larger)

One metric for everyone: strict-avg. Open-model values are the Open LLM Leaderboard "IFEval" (which is
strict-avg). Commercial rows are Proprietary; a * means the strict value is an estimate (no official
strict sub-metrics published — estimated from the model’s published AVG4/prompt-strict, which run a few points above
strict) and a + means the parameter count is unofficial. Our 61.6 is measured on our harness (calibrated above).

model weights company released params IFEval strict-avg
GPT-4o-mini Proprietary OpenAI Jul 2024 8B + 79 *
Llama-3.2-3B-Instruct Open Meta Sep 2024 3.2B 73.9
Qwen2.5-3B-Instruct Open Alibaba Sep 2024 3.0B 64.7
Phi-3-medium-4k-instruct Open Microsoft May 2024 14.0B 64.2
Mistral-Large Proprietary Mistral AI Feb 2024 46.7B + 63 *
KOS-V4-Instruct (ours) Open Univ. of Kentucky Jul 2026 3.0B 61.6
Yi-1.5-9B-Chat Open 01.AI May 2024 8.8B 60.5
Phi-3.5-mini-instruct Open Microsoft Aug 2024 3.8B 57.7
GPT-3.5-turbo-0613 Proprietary OpenAI Jun 2023 20B + 57 *
Command-R Open Cohere Mar 2024 35B 57 *
Phi-3-mini-4k-instruct Open Microsoft Apr 2024 3.8B 56.1
GPT-3.5-turbo-1106 Proprietary OpenAI Nov 2023 20B + 55.9
Mistral-7B-Instruct-v0.2 Open Mistral AI Dec 2023 7.2B 55.0
Gemini-1.0-Pro Proprietary Google Dec 2023 30B + 55 *
Mistral-Medium Proprietary Mistral AI Dec 2023 100B + 55 *
Mistral-7B-Instruct-v0.3 Open Mistral AI May 2024 7.2B 54.7
gemma-1.1-7b-it Open Google Mar 2024 8.5B 50.4
zephyr-7b-beta Open Hugging Face Oct 2023 7.2B 49.5
GPT-3.5-turbo-0125 Proprietary OpenAI Jan 2024 20B + 49 *
Llama-3.1-8B-Instruct Open Meta Jul 2024 8.0B 44.3
Qwen1.5-7B-Chat Open Alibaba Jan 2024 7.7B 43.7
Llama-2-13b-chat Open Meta Jul 2023 13.0B 39.8
Qwen1.5-4B-Chat Open Alibaba Jan 2024 4.0B 31.6

* strict estimate — no official IFEval strict sub-metrics are published for this model; the value is estimated
from its published AVG4 or prompt-strict (loose metrics run ~2–4 pts above strict). + unofficial params — never
disclosed by the provider (industry estimate: GPT-3.5 ~20B, GPT-4o-mini ~8B, Mistral-Large ~46.7B, Mistral-Medium
~100B, Gemini-1.0-Pro ~30B).

Reading. At 3B, KOS-V4 (61.6) beats every measured GPT-3.5-turbo snapshot of the original ChatGPT (1106 = 55.9,
0125 ≈ 49, 0613 ≈ 57 est), plus Yi-1.5-9B, both Mistral-7Bs, gemma-1.1, zephyr, Meta’s Llama-3.1-8B (44.3),
Llama-2-13b and the Qwen1.5 chats. Ahead of it: Llama-3.2-3B (73.9), Qwen2.5-3B (64.7), Phi-3-medium-14B (64.2), and
the frontier proprietary models. This is a strong result for a from-scratch 3B on 180B tokens and 24 GPUs, not a
claim to lead the current field.

Commercial baselines (strict-avg)

commercial model company snapshot strict-avg basis
GPT-4 OpenAI gpt-4-0613 80.6 computed from published strict sub-metrics (77.1 / 84.1), InternLM2 report
Command-R+ Cohere c4ai-command-r-plus 76.6 strict sub-metrics (72.8 / 80.5), Open LLM Leaderboard raw
GPT-3.5-turbo OpenAI gpt-3.5-turbo-1106 55.9 strict sub-metrics (50.5 / 61.2), InternLM2 report
Claude-3.5-Sonnet Anthropic 20240620 ~83 * estimate from published AVG4 86.2 (Llama 3.1 report)
GPT-4o OpenAI 2024-05-13 ~81 * estimate from AVG4 84.3
Gemini-1.5-Pro Google May 2024 ~79 * estimate from AVG4 82.3
Claude-3-Haiku Anthropic Mar 2024 ~65 * estimate from AVG4 68.1

Sources. IFEval definition: google-research/instruction_following_eval. Open-model strict-avg:
open-llm-leaderboard/contents (its "IFEval" column). Commercial strict sub-metrics: InternLM2 report
(arXiv:2403.17297) and Open LLM Leaderboard raw results. ~ * rows are estimates from published AVG4 (strict runs a
few points lower), clearly labeled.

BFCL (official bfcl_eval, function-calling mode; simple / multiple / parallel)

BFCL (official FC) KOS-V4-Instruct Qwen2.5-3B Llama-3.2-3B
simple / multiple / parallel 72.75 / 73.00 / 60.50 95.00 / 92.00 / 74.50 91.75 / 92.50 / 88.50

Deployment (inference)

Precision Approx. VRAM Notes
bfloat16 7 GB native weights (6.03 GB) + activations; a single 16 GB GPU is comfortable
GGUF Q8_0 / Q4_K_M ~4 / ~2.5 GB shipped for llama.cpp — see GGUF quantizations below

GGUF quantizations (llama.cpp)

Ready-to-run llama.cpp builds are published at
Kentucky-Open-Science/KOS-V4-Instruct-GGUF
(llama.cpp b510/18ef86e; lm-eval-harness 0.4.12).

file quant size notes
kosv4-f16.gguf F16 (16.0 bpw) 6.03 GB full-precision reference / requantize source
kosv4-Q8_0.gguf Q8_0 (8.5 bpw) 3.21 GB near-lossless
kosv4-Q4_K_M.gguf Q4_K_M (~4.8 bpw) 1.83 GB recommended — 3.3× smaller than f16, no measurable IFEval loss

SHA256 in SHA256SUMS; raw benchmark JSON + conversion/eval scripts in testing/.

Quantization preserves IFEval. Official EleutherAI lm-eval ifeval (--apply_chat_template), all 541 prompts,
greedy; only the weights differ across rows. strict-avg = (prompt-strict + inst-strict)/2.

format prompt-strict inst-strict strict-avg Δ vs bf16
bf16 (HF reference) 55.82 67.03 61.42 —
Q8_0 (GGUF) 55.27 66.43 60.85 −0.57
Q4_K_M (GGUF) 56.19 67.39 61.79 +0.37

All deltas are within ±1 pt (greedy/run-to-run noise) — the three formats are the same model on this benchmark, and the
bf16 row reproduces the card’s 61.6 headline (measured 61.42).

Serving:

llama-server -m kosv4-Q4_K_M.gguf --jinja -ngl 99 -c 6144 -np 1 --host 0.0.0.0 --port 8080

--jinja is required for the model’s <tool_call> output to be parsed into structured tool_calls; eos is
<|im_end|> (id 0) and no BOS is prepended; keep -c ≤ 6144 for in-distribution instruct behavior.

Edge Deployment & Performance (NVIDIA Jetson Orin Nano)

The highly compact memory footprint of the Q4_K_M quantization makes this model an exceptional candidate for localized, low-power edge computing platforms using unified memory architectures.

When deployed natively via llama.cpp using CUDA-offloaded layers, the model achieves the following baseline performance characteristics on an NVIDIA Jetson Orin Nano (8GB):

Phase Throughput Bottleneck Profile
Prefill (Prompt Processing) ~143.4 t/s Compute-Bound ($GEMM$ execution over unified RAM)
Decode (Token Generation) ~24.8 t/s Memory Bandwidth-Bound (Saturating the 68 GB/s bus)

Deployment Optimization Recommendations:

  • Lock Hardware Clocks: Prior to initializing llama-server, maximize the power envelope and lock the frequency steps to prevent dynamic frequency scaling latency: 
    
sudo nvpmodel -m 1
sudo jetson_clocks

Intended use & limitations

  • Intended use: general instruction following, structured output, and function/tool calling in clinical-adjacent workflows.
  • Not a medical-knowledge QA model. It follows instructions and calls tools; it does not reliably recall parametric medical facts. Ground it with retrieval instead.
  • Below current small models. On IFEval (strict-avg 61.6) and BFCL, newer small instruct models score higher; this model’s results are notable for its data/compute budget, not for leading the field.
  • English only. Strong public-benchmark numbers are not validation on real clinical data.
  • No safety or bias evaluation. This model has not been red-teamed, nor has it been evaluated for toxicity, clinical bias, or hallucination rates. It may produce harmful, biased, or medically inaccurate content.

Imported from hf:Kentucky-Open-Science/KOS-V4. Source last updated 2026-07-20. Synced 2026-07-27.