Fine-tuning adapts a pretrained large language model to a task, a domain or a style of behaviour by continuing training on a smaller, targeted dataset. This section covers the main methods, how they fit together, and when fine-tuning is the right tool at all.
A pretrained LLM has learned language and a great deal of world knowledge by predicting the next token over a huge corpus, but it is not yet a helpful assistant and knows nothing about your private data or formats. Fine-tuning updates the model's weights, all of them or a small added set, using examples of the behaviour you want. It is one of three main levers for changing what a model does, alongside prompt engineering and retrieval-augmented generation (retrieval).
Instruction tuning (supervised fine-tuning, SFT) teaches the model to follow instructions using prompt and response pairs. Preference tuning then aligns outputs with human judgements: RLHF trains a reward model from human comparisons and optimises the policy with reinforcement learning, while Direct Preference Optimization (DPO) optimises directly on preference pairs without a separate reward model. Parameter-efficient fine-tuning such as LoRA freezes the base weights and trains small low-rank adapters, cutting memory needs dramatically; QLoRA adds 4-bit quantisation of the frozen base so that large models fit on a single GPU. Knowledge distillation trains a smaller student model to imitate a larger teacher, trading a little quality for much cheaper inference.
Fine-tuning is often reached for too early. A useful rule: use prompting first, add retrieval when the model needs facts it does not have or that change often, and fine-tune when you need a consistent behaviour, format or style that prompting cannot reliably produce, when you want to shrink prompts to cut cost and latency, or when you want a smaller model to match a larger one on a narrow task. Fine-tuning is a poor way to add frequently changing knowledge, because every update means retraining and the model may still hallucinate details.
| Goal | Best first tool | Why |
|---|---|---|
| Answer from private, changing documents | RAG | Fresh data without retraining; citations possible |
| Consistent output format or tone | SFT with LoRA | Behaviour is learned, prompts get shorter |
| Follow instructions at all (from a base model) | Instruction tuning | Base models only continue text |
| Prefer helpful, harmless answers | Preference tuning (RLHF, DPO) | Optimises for judged quality, not imitation |
| Fit training on one GPU | QLoRA | 4-bit frozen base plus small adapters |
| Cut inference cost on a narrow task | Distillation | Smaller student imitates large teacher |
| Quick experiment, no training budget | Prompting, few-shot examples | Zero training cost; iterate in minutes |
Whatever you choose, build an evaluation set before training: a few hundred held-out prompts with reference answers or rubric-based grading, so you can show the fine-tuned model is actually better than a well-prompted base model.