How to Fine-Tune an LLM with LoRA and QLoRA on a Single GPU

A 7B model in full 16-bit precision needs roughly 60 GB of VRAM to train, which prices out anyone without a rack of A100s. LoRA and its 4-bit sibling QLoRA collapse that to a single 24 GB card by freezing the base weights and training small adapter matrices instead. The short version: quantize the model to 4-bit, attach a LoraConfig, and hand both to Hugging Face's SFTTrainer, which runs the supervised loop and saves an adapter you can merge or serve on its own.

This guide gives you the exact code to fine-tune an open model on your own dataset, the VRAM math to pick a method, and the errors that stop most first attempts.

Key takeaways

The short answer: load the model with a 4-bit BitsAndBytesConfig, pass a PEFT LoraConfig and your dataset to TRL's SFTTrainer, then call trainer.train().
LoRA trains under 1% of a model's parameters, so a 7B fine-tune fits on a 24 GB GPU; QLoRA's 4-bit base weights push the same job onto a 16 GB card.
The output is a small adapter (tens of MB), not a full model copy. You load it on top of the base weights at inference or merge it in.
Set target_modules="all-linear" to adapt every linear layer, the QLoRA paper's recommended default over hand-picking attention projections.
Use a higher learning rate than full fine-tuning, near 1e-4 to 2e-4, because only the new adapter weights learn.
Fine-tuning teaches format and behavior, not fresh facts. For knowledge that changes, reach for retrieval instead.

What You Need Before You Start

This is a GPU job. A 7B model with QLoRA runs on a single 16 GB card such as a T4 or a consumer 4080; plain LoRA on a 7B wants 24 GB. Rent one on any cloud if you don't own it.

Python 3.10 or newer with a CUDA-enabled PyTorch build.
The libraries: pip install trl peft transformers datasets accelerate bitsandbytes. TRL 1.x and Transformers 5.x are assumed below.
A dataset in a format SFTTrainer understands: a text column, or a prompt/completion pair, or a conversational messages list. Your own JSONL works once it uses one of these shapes.
A Hugging Face account and token if your base model is gated, which Llama and Gemma are. See accessing gated models on Hugging Face for the access request and login step.

How to Fine-Tune an Open Model with QLoRA

Every step below runs in one Python script. Swap the model id and dataset for your own, then run it. This is the whole procedure.

Load the model in 4-bit. The BitsAndBytesConfig quantizes weights to the NF4 data type and runs compute in bfloat16. This is the QLoRA memory trick.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

model_id = "Qwen/Qwen2.5-3B"

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=bnb_config,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained(model_id)

Define the LoRA adapter. Rank r sets adapter size; 16 is a solid default. lora_alpha scales the update, and target_modules="all-linear" adapts every linear layer, which the QLoRA authors recommend over targeting only attention.
```
from peft import LoraConfig

peft_config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
    target_modules="all-linear",
)
```

Load your dataset. Use a Hub dataset or your own JSONL. SFTTrainer applies the model's chat template automatically for conversational data.

from datasets import load_dataset

dataset = load_dataset("trl-lib/Capybara", split="train")
# Or your own file:
# dataset = load_dataset("json", data_files="train.jsonl", split="train")

Configure and run training. Pass the LoRA config straight to the trainer. TRL wraps the quantized model, freezes the base, and trains only the adapter. The higher learning rate is deliberate.

from trl import SFTConfig, SFTTrainer

args = SFTConfig(
    output_dir="qwen-lora-out",
    num_train_epochs=1,
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,
    learning_rate=2e-4,
    logging_steps=10,
    max_length=1024,
    packing=True,
)

trainer = SFTTrainer(
    model=model,
    args=args,
    train_dataset=dataset,
    peft_config=peft_config,
)
trainer.train()

Save the adapter. This writes only the adapter weights, usually tens of megabytes, not a full model copy.
```
trainer.save_model("qwen-lora-out")
```

That is a complete run. The remaining sections cover what the knobs do, how to use the adapter, and what breaks.

What the Core Settings Actually Control

Four settings decide most of your result. Getting them right matters more than any exotic trick.

Rank, Alpha, and Which Layers to Adapt

Rank r is the width of the low-rank update. Higher rank means more capacity and more VRAM; 8 to 16 covers most instruction-tuning, and 32 or 64 helps when you teach a genuinely new skill. A common heuristic sets lora_alpha to twice r. Targeting "all-linear" beats hand-picking q_proj and v_proj in most benchmarks, at a modest memory cost.

Effective Batch Size and Learning Rate

Your true batch size is per_device_train_batch_size times gradient_accumulation_steps. The example runs an effective batch of 16 while keeping only two sequences in memory at once, which is how you fit training into tight VRAM. Adapter training tolerates a learning rate 5x to 10x higher than full fine-tuning because far fewer weights move.

Choosing LoRA, QLoRA, or Full Fine-Tuning

The method sets your hardware floor. This table shows the tradeoff for a 7B model.

Method	Base weights	Trainable params	Approx VRAM (7B)	Use when
Full fine-tune	16-bit, trainable	100%	60 GB or more	You have the hardware and need maximum quality
LoRA	16-bit, frozen	Under 1%	Around 24 GB	One 24 GB GPU, quality close to full
QLoRA	4-bit, frozen	Under 1%	10 GB to 16 GB	Smallest footprint, minor quality tradeoff

How to Run Inference and Merge the Adapter

An adapter is not a standalone model. You either load it on top of the base weights or merge the two into one set of weights.

To load the adapter for inference:

from peft import AutoPeftModelForCausalLM
from transformers import AutoTokenizer

model = AutoPeftModelForCausalLM.from_pretrained("qwen-lora-out")
tokenizer = AutoTokenizer.from_pretrained("qwen-lora-out")

To merge the adapter into the base weights and export a single model you can serve or convert, load in full precision first, then merge. Do not merge into 4-bit weights; the result degrades.

from peft import AutoPeftModelForCausalLM

model = AutoPeftModelForCausalLM.from_pretrained(
    "qwen-lora-out", torch_dtype="auto",
)
merged = model.merge_and_unload()
merged.save_pretrained("qwen-merged")

Once merged, the model is an ordinary set of weights. Serve it for throughput with vLLM, or convert it to GGUF to run locally with Ollama.

Troubleshooting Common Errors

CUDA out of memory. Drop per_device_train_batch_size to 1 and raise gradient_accumulation_steps to keep the effective batch. Lower max_length, confirm gradient_checkpointing is on (TRL defaults it to true), and switch from LoRA to QLoRA if you have not already.
Loss stays flat or is NaN. A flat loss usually means the learning rate is too low for adapters; move toward 2e-4. NaN loss points at a precision clash, so keep bnb_4bit_compute_dtype and your training dtype both on bfloat16 rather than fp16.
ValueError about the pad token. Some tokenizers ship without a pad token. Set tokenizer.pad_token = tokenizer.eos_token before training and pass the tokenizer as processing_class.
The tuned model ignores your format. Your dataset likely does not match the chat template. Use the conversational messages format so SFTTrainer applies the template, and set assistant_only_loss=True to train on responses rather than the whole transcript.
A 401 or gated-repo error on load. The base model needs access approval. Request it on the model page and run huggingface-cli login, per the gated models guide.

What to Do Next

Fine-tuning changes how a model behaves, not what it knows. When you need current or proprietary facts, pair the model with retrieval instead of, or alongside, an adapter; a pgvector semantic search setup is a practical starting point. From here:

Read the TRL SFTTrainer reference for packing, completion-only loss, and evaluation options.
Study the PEFT LoraConfig reference to tune rank, dropout, and DoRA.
Review the PEFT quantization guide for QLoRA, LoftQ, and other backends.
Skim the 4-bit and QLoRA explainer for the benchmarks behind the method.