LoRA and QLoRA make it practical to fine-tune large language models on a single GPU. This page explains the low-rank idea, the memory tricks QLoRA adds, a working PEFT setup and the mistakes that most often waste a training run.
LoRA (Low-Rank Adaptation, Hu et al., 2021) is a way to fine-tune a large model without updating its original weights. Each chosen weight matrix W is frozen, and a small trainable correction is added beside it, written as the product of two thin matrices B and A. Only those thin matrices are trained, so the number of trainable parameters typically drops by a factor of hundreds or more.
QLoRA (Dettmers et al., 2023) goes further by storing the frozen base model in 4-bit precision while training LoRA adapters in 16-bit. The QLoRA paper showed a 65-billion-parameter model fine-tuned on a single 48 GB GPU. Both belong to the family of parameter-efficient fine-tuning (PEFT) methods, which also includes adapters, prefix tuning and prompt tuning.
Full fine-tuning updates every weight and, with the Adam optimizer in mixed precision, needs roughly 16 bytes of GPU memory per parameter for weights, gradients and optimizer state — on the order of 100 GB or more for a 7-billion-parameter model before counting activations. LoRA removes gradients and optimizer state for the frozen weights, and QLoRA shrinks the frozen weights themselves to about half a byte per parameter. That moves fine-tuning from multi-GPU clusters to a single workstation GPU.
There is an operational benefit too. An adapter for a 7B model is often only tens of megabytes. One base model can serve many tasks or customers by swapping adapters, and serving engines such as vLLM can batch requests for different adapters together.
For a frozen weight W of shape dxk, LoRA learns A (rxk) and B (dxr) with a small rank r, typically 4 to 64. The layer computes
h = W x + (α / r) · B A x
A starts random and B starts at zero, so at step zero the model behaves exactly like the base model. The scaling factor α/r keeps the update size roughly stable when you change the rank. The bet behind LoRA is that the change needed to adapt a pretrained model has low intrinsic rank, an idea supported by earlier work on intrinsic dimensionality (Aghajanyan et al., 2021).
After training, BA can be added into W ("merged"), so inference costs exactly the same as the base model. Adapters are usually applied to the attention projections (q_proj, k_proj, v_proj, o_proj); the QLoRA paper found that adding them to all linear layers, including the MLP, was needed to match full fine-tuning quality.
QLoRA adds three ideas. 4-bit NormalFloat (NF4) is a data type whose 16 levels are spaced for normally distributed weights. Double quantization also quantizes the per-block scaling constants, saving roughly another 0.4 bits per parameter. Paged optimizers use unified memory so optimizer state can spill to CPU RAM during memory spikes instead of crashing. Gradients flow through the dequantized 4-bit weights into the 16-bit adapters; the base weights never change.
A minimal QLoRA setup with transformers, peft and bitsandbytes (NVIDIA GPU required for 4-bit). Swap the model id for any causal LM you have access to.
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
model_id = "Qwen/Qwen2.5-0.5B" # small enough to try; same code for 7B+
bnb = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_use_double_quant=True,
bnb_4bit_compute_dtype=torch.bfloat16,
)
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=bnb,
device_map="auto")
model = prepare_model_for_kbit_training(model) # enables grad checkpointing, casts norms
lora = LoraConfig(
r=16, lora_alpha=32, lora_dropout=0.05, bias="none", task_type="CAUSAL_LM",
target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj"],
)
model = get_peft_model(model, lora)
model.print_trainable_parameters() # prints trainable vs total parameter counts
# ... train with transformers.Trainer or trl.SFTTrainer on your dataset ...
model.save_pretrained("my-adapter") # saves only the adapter weightsTo serve, either load the base model and attach the adapter with PeftModel.from_pretrained(base, "my-adapter"), or merge with merge_and_unload() on a 16-bit copy of the base model and export a normal checkpoint. Merging into a 4-bit model is lossy, so merge in 16-bit.
Most LoRA runs only need decisions on a handful of settings. Rank controls capacity: 8 or 16 is enough for teaching a format or tone, while harder domain shifts such as a new programming language or many specialised tasks may benefit from 32 to 64. Raising rank beyond what the data supports mainly adds overfitting risk. Alpha is commonly set to the rank or twice the rank; because the update is scaled by alpha divided by rank, changing one without the other changes the effective learning rate. Dropout of 0.05 to 0.1 on the adapter path helps with small datasets.
Data quality matters far more than these knobs. A few thousand clean, consistent, representative examples usually beat a much larger noisy set. Hold out a validation split, watch validation loss for the overfitting pattern described on the regularization page, and compare generations from several checkpoints side by side, because loss alone does not reveal whether answers became better or just shorter. Keep a fixed set of general-purpose prompts as a regression test for forgetting.
| Method | Trainable params | Memory | Inference overhead | Notes |
|---|---|---|---|---|
| Full fine-tuning | 100% | Highest | None | Best ceiling; one full copy per task |
| LoRA | Typically well under 1% | Base weights in 16-bit, no optimizer state for them | None if merged | Most widely used PEFT method |
| QLoRA | Same as LoRA | Base weights in 4-bit | Dequantization cost unless merged | Slower per step than LoRA; fits much larger models |
| DoRA | Slightly more than LoRA | Similar to LoRA | None if merged | Splits magnitude and direction; often small gains |
| Adapters (Houlsby) | Small | Low | Extra layers at inference | Earlier PEFT method |
| Prompt / prefix tuning | Tiny | Lowest | Longer effective context | Weaker on smaller models |
model.named_modules() and check the trainable count.