Instruction tuning, usually done as supervised fine-tuning (SFT), is the step that turns a pretrained language model that merely continues text into an assistant that follows instructions. It is also the most common way practitioners adapt an open model to their own task, tone or output format. This page explains what SFT does mathematically, how to build the dataset, why loss masking and LoRA matter, how to run it with Hugging Face TRL, and when you should not fine-tune at all.
A base model trained on web text has learned grammar, facts and reasoning patterns, but if you ask it a question it may continue with more questions, because that is what text on the internet often does. Instruction tuning continues training on examples of the form (instruction, desired response). After a relatively small amount of such data, the model reliably treats input as a request and produces a response in the demonstrated style.
The idea was popularised by FLAN (Wei et al., 2021), which showed that fine-tuning on many tasks phrased as instructions improves zero-shot performance on unseen tasks, and by InstructGPT (Ouyang et al., 2022), which used SFT on human-written demonstrations as the first stage before RLHF. LIMA (Zhou et al., 2023) argued that a carefully chosen set of about 1,000 high-quality examples can produce a capable assistant, supporting the view that SFT mostly teaches format and behaviour while knowledge comes from pretraining.
For practitioners, SFT is the tool for things prompting does poorly or expensively: a strict output format used millions of times, a house style, domain-specific classification or extraction, or distilling a large model's behaviour into a small, cheap one. A fine-tuned 1B to 8B model can often match a much larger prompted model on a narrow task at a fraction of the inference cost and latency. What SFT is not good at is adding fresh or fast-changing knowledge; retrieval handles that better.
SFT uses the same objective as pretraining, next-token cross-entropy, applied to curated conversations. For a prompt x and response y = (y1, ..., yT), the loss is L = −Σt log pθ(yt | x, y<t). The key detail is loss masking: the sum runs over response tokens only. Prompt tokens are fed in as context but their labels are set to an ignore value (−100 in PyTorch), so the model is not trained to generate user messages.
Conversations are serialised with the model's chat template, which inserts special tokens marking system, user and assistant turns. Using the wrong template, or a different one at inference than at training, is one of the most common causes of a fine-tune that "does nothing" or produces garbled turns.
Data quality dominates everything else. Good SFT sets are consistent (the same kind of request always gets the same style of answer), diverse (they cover the range of real inputs, including edge cases and requests the model should decline), and correct (every response is one you would be happy to ship). Typical sizes range from a few hundred examples for a narrow format task to tens of thousands for broad behaviour. Sources include expert-written answers, cleaned production logs, and synthetic responses from a stronger model that are then filtered. The usual format is one JSON object per line with a list of messages:
{"messages": [
{"role": "system", "content": "You are a concise support assistant for Acme Bank."},
{"role": "user", "content": "How do I reset my card PIN?"},
{"role": "assistant", "content": "Open the app, go to Cards > Manage > Reset PIN, and confirm with your passcode. The new PIN works immediately at ATMs."}
]}Hold out a validation split and, separately, an evaluation set scored on the real task, because validation loss alone does not tell you if outputs are useful.
Full fine-tuning updates every weight, which for a 7B model requires memory for weights, gradients and optimizer states, typically several high-memory GPUs. LoRA (Hu et al., 2021) freezes the original weights and learns a low-rank update ΔW = BA for selected matrices, where B is d × r and A is r × d with rank r much smaller than d. Only A and B are trained, often well under 1% of parameters. QLoRA (Dettmers et al., 2023) goes further by keeping the frozen base in 4-bit precision, making it possible to fine-tune multi-billion-parameter models on a single GPU. Adapters are small files that can be swapped at serving time or merged into the base weights.
| Method | Trainable params | Memory | When to use |
|---|---|---|---|
| Full fine-tuning | All | Highest | Large data, maximum quality, ample hardware |
| LoRA | Typically <1% | Moderate | Default choice for most task adaptation |
| QLoRA | Typically <1% | Lowest | Large models on limited GPUs |
| Prompting / few-shot | None | None | Try first; small volume or changing requirements |
| RAG | None | None | Task needs current or private knowledge |
# pip install trl peft transformers datasets accelerate
from datasets import load_dataset
from peft import LoraConfig
from trl import SFTConfig, SFTTrainer
ds = load_dataset("json", data_files="train.jsonl", split="train") # chat format above
ds = ds.train_test_split(test_size=0.05, seed=0)
peft_cfg = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05,
target_modules="all-linear", task_type="CAUSAL_LM")
cfg = SFTConfig(
output_dir="sft-out",
num_train_epochs=2,
per_device_train_batch_size=4,
gradient_accumulation_steps=4,
learning_rate=2e-4, # typical for LoRA; full fine-tuning uses ~1e-5
lr_scheduler_type="cosine",
warmup_ratio=0.03,
assistant_only_loss=True, # mask prompt tokens (needs a chat template that marks them)
eval_strategy="steps", eval_steps=200,
logging_steps=20, bf16=True,
)
trainer = SFTTrainer(
model="Qwen/Qwen2.5-0.5B-Instruct", # small model so it runs on one GPU
args=cfg, train_dataset=ds["train"], eval_dataset=ds["test"],
peft_config=peft_cfg,
)
trainer.train()
trainer.save_model("sft-out/adapter")This trains a LoRA adapter on a small instruct model using TRL's SFTTrainer, which applies the chat template and, with assistant_only_loss, masks non-assistant tokens. Typical starting points: learning rate around 1e-4 to 2e-4 for LoRA (much lower, around 1e-5, for full fine-tuning), 1 to 3 epochs, and an effective batch size of 16 to 64 sequences. Watch validation loss for overfitting and, more importantly, run your task evaluation on checkpoints. Argument names change between TRL releases, so check the version you install.