How to Quantize an LLM to GGUF with llama.cpp

A 7B model in 16-bit precision wants about 14 GB of memory before you load a single token of context. Quantize it to 4-bit GGUF and the same weights drop to roughly 4.4 GB, which is the difference between "needs a datacenter GPU" and "runs on the laptop you already own." The short version: convert the Hugging Face checkpoint to a full-precision GGUF with convert_hf_to_gguf.py, then compress that file with llama-quantize, starting at Q4_K_M for most work.

Key Takeaways

The short answer: run convert_hf_to_gguf.py to make an f16 GGUF, then llama-quantize model-f16.gguf model-Q4_K_M.gguf Q4_K_M.
GGUF is the single-file format that llama.cpp, Ollama, and LM Studio all load. Quantization is the step that shrinks it.
Convert to f16 first, then quantize from that file. Do not quantize directly during conversion if you plan to compare levels.
Q4_K_M is the default balance of size and quality. Move up to Q5_K_M or Q6_K when you have spare memory, and reserve Q8_0 for validation.
An importance matrix (imatrix) recovers noticeable quality at 2-bit and 3-bit levels, and costs only a short calibration run.

What You Need Before You Start

This is a short list, and skipping any of it is where most people lose an hour.

A machine with enough RAM to hold the model in full precision during conversion. Conversion loads the whole thing before it writes, so a 7B model wants roughly 14 GB free, and disk space for both the source and the output at the same time.
Python 3.9 or newer, plus git and a C++ toolchain (cmake and a compiler). On a GPU box, the CUDA toolkit if you want to build with GPU support.
The model already downloaded in Hugging Face format: .safetensors weights plus config.json and the tokenizer files. Gated models need you to accept the license first; see how to access gated models on Hugging Face.

How Do You Quantize a Model to GGUF?

Here is the full procedure end to end. Run these in order and you will have a working quantized file at the end.

Clone llama.cpp and install its Python requirements.

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
pip install -r requirements.txt

Build the tools. This compiles llama-quantize and llama-cli into build/bin.
```
cmake -B build
cmake --build build --config Release -j
```
For an NVIDIA GPU, configure with CUDA instead: cmake -B build -DGGML_CUDA=ON.

Download the source model into a local folder if you have not already.

pip install huggingface_hub
huggingface-cli download meta-llama/Llama-3.1-8B-Instruct \
  --local-dir ./Llama-3.1-8B-Instruct

Convert the Hugging Face weights to a full-precision GGUF. Use f16 so you keep a clean source to quantize from.
```
python convert_hf_to_gguf.py ./Llama-3.1-8B-Instruct \
  --outtype f16 \
  --outfile llama-3.1-8b-f16.gguf
```

Quantize to your target level. The arguments are input, output, then the type.

./build/bin/llama-quantize \
  llama-3.1-8b-f16.gguf \
  llama-3.1-8b-Q4_K_M.gguf \
  Q4_K_M

Test it before you trust it. Load the quantized file and generate.

./build/bin/llama-cli \
  -m llama-3.1-8b-Q4_K_M.gguf \
  -p "Explain quantization in one sentence." -n 128

That is the whole task. The sections below explain how to pick the type, when to add an importance matrix, and what to do when a step fails.

Which Quantization Type Should You Pick?

Pick by how much memory you can spare, then verify the quality holds for your task. The K-quant types (the ones with a _K in the name) mix precision across a model's layers, which is why they beat the older Q4_0-style formats at the same size. Here is how the common choices compare for a 7B-class model.

Type	Bits/weight	Approx. size (7B)	Quality loss	Use it for
`Q8_0`	~8.5	~7.2 GB	Near zero	A reference to measure the others against
`Q6_K`	~6.6	~5.5 GB	Very small	Quality-first when memory is comfortable
`Q5_K_M`	~5.7	~4.8 GB	Small	A safe step up from Q4 when you have the room
`Q4_K_M`	~4.8	~4.4 GB	Modest	The default for most local and production use
`Q3_K_M`	~3.9	~3.3 GB	Noticeable	Squeezing a bigger model onto small hardware
`IQ2_M`	~2.7	~2.5 GB	Large	Last resort, and only with an imatrix

The honest rule: start at Q4_K_M, and only change it because a measurement told you to. If outputs feel off, step up to Q5_K_M. If the file will not fit, step down, but test hard below Q4 because that is where degradation stops being subtle.

When Does an Importance Matrix Earn Its Keep?

An importance matrix tells the quantizer which weights matter most, so it spends its limited bits where they count. The payoff is small at Q4_K_M and above, and real at Q3 and below, where a good imatrix can claw back a chunk of the quality a naive quant throws away.

Generate one from a calibration text file, then pass it to the quantizer.

./build/bin/llama-imatrix \
  -m llama-3.1-8b-f16.gguf \
  -f calibration.txt \
  -o imatrix.dat

./build/bin/llama-quantize \
  --imatrix imatrix.dat \
  llama-3.1-8b-f16.gguf \
  llama-3.1-8b-IQ3_M.gguf \
  IQ3_M

Use a calibration file that looks like your real traffic. A few hundred KB of representative prose or code is plenty; wikitext is the common generic default.

How Do You Confirm the Quantized Model Is Still Good?

Do not eyeball a single reply and call it fine. Measure perplexity against a held-out text and compare it to the f16 source, so you have a number instead of a vibe.

./build/bin/llama-perplexity \
  -m llama-3.1-8b-Q4_K_M.gguf \
  -f wikitext-2-raw/wiki.test.raw

A small perplexity rise from f16 to Q4_K_M is expected and safe. A large jump means the quant is too aggressive for this model, so move up a level. For anything user-facing, back the perplexity check with a few task-specific prompts, because perplexity and real task quality do not always move together.

Troubleshooting the Errors You Will Actually Hit

"Model architecture not supported." The converter only knows architectures it has code for. Update your llama.cpp clone (git pull) and rebuild, since new model families land there regularly. If it is brand new, the converter may not support it yet.
The converter finds multiple or duplicate weight files. A model folder that ships both .safetensors and legacy .bin copies confuses the script. Keep the .safetensors set and remove the duplicates.
Out-of-memory during conversion. Conversion needs the full-precision model in RAM. Convert on a bigger box, or download in safetensors and close everything else. The quantize step itself is far lighter than conversion.
A tokenizer or vocab mismatch warning. Make sure tokenizer.json, tokenizer_config.json, and config.json are all present in the source folder. A partial download is the usual cause.
Missing Python modules. The requirements pin specific versions; install them into a clean virtual environment rather than your system Python to avoid conflicts.

What to Do Next

You have a portable, single-file model. Point one of these at it:

Serve it locally with Ollama by writing a Modelfile that references your GGUF. Walkthrough: how to run large language models locally with Ollama.
Run a full private setup if the goal is a self-hosted assistant, following how to host an LLM locally for private inference.
Fine-tune first, then quantize. If you adapted the model with LoRA, merge the adapter into the base before converting, per how to fine-tune an LLM with LoRA and QLoRA.
Choose the right serving path. GGUF suits single-node and edge; for high-throughput multi-user serving, weigh serving an open LLM with vLLM instead.

Upload the finished GGUF to a Hugging Face repo and it becomes downloadable by name in Ollama and LM Studio, which is how most public quants get distributed.