Does quantisation reduce model intelligence?

A 4-bit quantised model typically retains 95–99% of FP16 benchmark scores. Most users cannot tell the difference in chat, coding, or reasoning tasks.

What is GGUF and why is it popular?

GGUF is a single-file format used by llama.cpp that bundles quantised weights, metadata, and tokenizer. It runs on CPU, Apple Silicon, and NVIDIA/AMD GPUs with zero dependencies.

Which quant level should I pick for a 7B model on 16 GB VRAM?

Q4_K_M (~4.5 GB) leaves plenty of headroom for context and KV cache. Q5_K_M (~5.5 GB) is also safe and slightly better on complex reasoning.

Can I quantise a model myself from Hugging Face?

Yes. Download the FP16/BF16 weights, convert to GGUF with convert_hf_to_gguf.py , then run llama-quantize to produce Q4_K_M, Q5_K_M, etc.

Quantisation Explained: Why Smaller Models Often Run Better

What Is Quantisation?

Quantisation is the process of reducing the numerical precision of a model's weights and activations. Most models ship in FP16 (16-bit floating point) or BF16. Quantisation converts those weights to INT8 (8-bit integers) or INT4 (4-bit integers). The arithmetic shifts from floating-point multiplication to integer math, which is far cheaper on modern GPUs and CPUs.

The trade-off is tiny: a well-quantised 4-bit model typically retains 95–99% of the original quality on benchmarks, while using roughly one-quarter of the VRAM.

Why It Matters for Local Inference

VRAM is the bottleneck for running large language models locally. A 7B-parameter model in FP16 needs about 14 GB of VRAM just for weights. The same model in 4-bit (GGUF Q4_K_M) needs ~4.5 GB. That difference determines whether a model fits on a 24 GB card, a 16 GB card, or a 12 GB laptop GPU.

Quantisation also speeds up inference. Integer matrix multiplication kernels are heavily optimised on NVIDIA Tensor Cores and Apple Neural Engine. You often see 2–4× tokens-per-second gains over FP16, especially when the model is memory-bound.

Common Quantisation Formats

GGUF / llama.cpp: The de-facto standard for local CPU/GPU inference. Supports k-quant variants (Q2_K, Q3_K_M, Q4_K_M, Q5_K_M, Q6_K, Q8_0) that balance size and quality.
GPTQ / AWQ: Popular for GPU-only deployments via AutoGPTQ or vLLM. Calibrated on a small dataset to preserve accuracy.
HQQ / QuIP#: Newer methods pushing 3-bit and 2-bit with better quality retention.

Choosing the Right Quant Level

For most users, Q4_K_M is the sweet spot: ~4.5 GB for a 7B model, near-FP16 quality, and fast on both CPU and GPU. Q5_K_M adds ~1 GB for a measurable quality bump on reasoning tasks. Q6_K and Q8_0 approach FP16 fidelity but lose much of the memory advantage.

Avoid Q2_K and Q3_K_S unless you are severely memory-constrained — degradation on complex reasoning becomes noticeable.

Quantisation vs. Distillation vs. Pruning

Quantisation keeps the architecture intact and only lowers weight precision. Distillation trains a smaller student model to mimic a larger teacher. Pruning removes entire weights or attention heads. Quantisation is the simplest, fastest to apply, and requires no retraining — making it the first optimisation to try.

Practical Tips

Always benchmark your specific workload. Perplexity on a validation set correlates poorly with subjective quality on chat or code.
Use llama.cpp's llama-quantize or llama.cpp Python bindings to convert HF models to GGUF.
For GPU offload, set -ngl (number of GPU layers) to maximise VRAM usage without OOM.
Calibrated quantisation (GPTQ, AWQ) needs a representative calibration dataset — 128–512 sequences from your target domain works well.

When Not to Quantise

If you are training or fine-tuning, keep weights in FP16/BF16. Quantisation is an inference-time optimisation. Also, some niche tasks (exact arithmetic, multilingual low-resource languages) degrade more than average — test before committing.

The Bottom Line

Quantisation is the single highest-leverage optimisation for local LLM deployment. It turns a model that needs a data-center GPU into one that runs on a MacBook Pro or a single 24 GB consumer card — with quality loss most users never notice. Start with GGUF Q4_K_M, benchmark your prompts, and only move up if quality demands it.