LLM Quantization Without the Alphabet Soup

You find a model on Hugging Face. Instead of one download, you see BF16, Q8_0, Q6_K, Q4_K_M, IQ4_XS, UD-Q4_K_XL and a dozen more.

They are mostly different answers to this question: how much model quality are you willing to trade for a smaller, faster file?

Start with these four rules:

Pick Q4_K_M or a good Q4 dynamic quant first.
Pick Q6_K when memory is comfortable and you want to preserve more quality.
Use Q3 only when Q4 does not fit.
Use Q2 when it is the only way to run a much larger model.

The rest of this guide explains why.

What quantization does

A model is mostly a huge list of numbers called weights. Those numbers control how strongly billions of tiny learned connections affect one another.

The original model may store every weight using 16 bits. Quantization stores an approximation using fewer bits, often four. It resembles saving a photo as a smaller JPEG: fine detail is lost, but a careful compression method protects the parts you need the most.

For a rough size estimate:

model size in GB ≈ parameters in billions × bits per weight ÷ 8

A 30B model at exactly four bits would therefore need about 15 GB just for weights. Published files are a little larger because they include scales, metadata and some higher-precision tensors. The context cache also consumes memory while the model runs.

FP32, FP16 and BF16

These floating-point formats can represent both tiny and huge values.

Format	Bits	Beginner’s version
FP32	32	Very precise, very large; now uncommon for distributing LLMs
FP16	16	Half the FP32 size; good precision but a narrower numerical range
BF16	16	Similar size to FP16, with a much wider range; common for training and original weights

For inference, publishers generally treat FP16 and BF16 checkpoints as the high-quality reference. Both take roughly two bytes per parameter. A 70B BF16 model is around 140 GB before overhead, which is why most home users quantize.

BF16 keeps a wider numerical range than FP16 but gives up some fine precision. That range helps large neural networks remain stable during training.

What BPW means

BPW means bits per weight. Lower BPW tends to produce a smaller file with more quality loss.

The name on the file is only a category. A Q4 file is not always exactly 4.00 BPW because the format also stores scaling information and may keep sensitive tensors at higher precision. It may land around 4.5–5 BPW in practice.

BPW compares size, not quality by itself. A carefully made 4.5 BPW quant can beat an older 5 BPW method. Model architecture, calibration and runtime support affect accuracy and speed.

A rough quality ladder

Treat these rankings as a starting point:

Level	Typical use	Quality
BF16 / FP16	Reference, training, plenty of memory	Original quality
Q8	Near-reference local inference	Loss is extremely hard to notice in most prompts
Q6	Quality-first local inference	Near perfect for most use
Q5	Extra quality without Q6 size	Excellent
Q4	Best general balance	Most of the model’s performance for much less memory
Q3	Tight memory	Acceptable, but mistakes and style damage become easier to notice
Q2	Fit a model that otherwise cannot run	Clear loss; reserve it for severe memory limits

A larger model at Q3 can still beat a smaller model at Q6. Quantization is only one part of model quality. For example, a good 70B Q3 may know and reason more than a 14B Q8, despite being more compressed.

Reading common GGUF names

GGUF is the file format commonly used by llama.cpp, LM Studio, KoboldCpp and similar apps.

`Q4_0` and `Q8_0`

These older block formats are easy for runtimes to support. Q8_0 remains a solid near-lossless choice. Q4_0 is fast and widely supported, although newer K-quants often preserve more quality at a similar size.

`Q4_K_S`, `Q4_K_M` and `Q6_K`

The K means a newer family of GGUF quantization methods. The final letter is a rough size strategy:

S: small
M: medium
L or XL: larger or more quality-focused, when a publisher uses those names

Q4_K_M works well as a first download for many models. It stores some sensitive parts at higher precision than others. Q6_K uses more memory but stays close enough to BF16 that normal chat prompts reveal little difference.

`IQ4_XS` and other I-quants

IQ formats use more sophisticated importance-aware techniques. IQ4_XS aims for strong quality at a small size. I-quants can be excellent when memory is tight, but performance depends more on your CPU, GPU and llama.cpp version. Check that your app supports the exact format before downloading 50 GB.

`Q4_K_XL`

Publishers such as Unsloth use names including Q4_K_XL for quality-focused four-bit recipes. The XL version spends extra space or preserves more important tensors, so it can be better than a smaller Q4 while remaining far below BF16 size.

These suffixes are not one universal standard across every tool. Compare the model publisher’s table, BPW and benchmark notes rather than assuming every XL is identical.

Dynamic and hybrid quants

Old quantization recipes often treated every part of a model similarly. That is easy, but neural networks are uneven. Some layers can survive aggressive compression; a few are very sensitive.

A hybrid quant mixes precisions. Ordinary layers may use four bits while important attention or output tensors use six or eight. Q4_K_M already uses this idea in a limited way.

A dynamic quant decides the mixture more carefully per layer or tensor. Unsloth’s Dynamic GGUF families use calibration tests to find sensitive areas, then spend more bits there. Names such as UD-Q4_K_XL mean an Unsloth Dynamic recipe based around a quality-focused Q4 target.

A school budget provides a decent analogy. Equal cuts are easy to administer, yet some departments suffer more than others. A dynamic quant spends extra bits where compression causes the most damage.

This can preserve more accuracy than a uniform quant of similar size. Results vary by model, unusual formats may need a recent runtime, and a larger label does not guarantee better speed on your hardware.

NVFP4 and MXFP4 are different

New NVIDIA models and checkpoints may use NVFP4. Models such as Kimi K3 use MXFP4. Both store weights at around four bits. Each uses different scaling rules and hardware support from GGUF Q4.

NVFP4 is NVIDIA’s four-bit floating-point format, designed for recent NVIDIA tensor cores.
MXFP4 is a microscaling format that shares small scale values across blocks of weights.
GGUF Q4 formats are designed for flexible local inference across CPUs and many GPUs.

Native FP4 can be extremely fast on hardware that supports it. On unsupported hardware, the runtime may convert values or fail to load the model. A four-bit file is not automatically fast merely because your GPU can store it.

This is especially relevant to Kimi K3: its official MXFP4 checkpoint still occupies around 1.4 TB because the model has 2.8 trillion parameters.

Quality, speed and space pull in different directions

Quantization trades quality for size. Speed adds another variable.

A smaller model transfers fewer bytes from memory, which often makes generation faster. It may also let you move more layers from slow system RAM into fast VRAM. That can be a huge win.

But complicated low-bit formats need extra work to unpack. Some GPUs have fast kernels for Q4 but poor support for IQ2. A smaller quant can therefore run slower than a larger, well-supported one.

The usual priorities are:

Fit the entire model in fast memory if possible.
Choose a format your runtime and hardware accelerate well.
Use the highest quality that still leaves room for your context cache.

Do not download a 24 GB quant for a 24 GB GPU and expect a perfect fit. Leave space for the context, runtime and desktop. A 20 GB model is a safer ceiling on a 24 GB card, depending on context length.

What should you download?

New to quantization:

Choose Q4_K_M. If the model publisher provides a well-tested dynamic Q4 and your runtime supports it, that is also a good choice.

Plenty of memory:

Choose Q6_K. Q8 is valid, but the extra size over Q6 often buys very little visible improvement.

Q4 almost fits:

Try IQ4_XS, Q4_K_S or a smaller dynamic Q4. Reduce context length before dropping to Q3 when you do not need a huge context.

Only Q3 fits:

Use it. Q3 remains capable, though errors and style damage appear more often. Prefer an importance-aware or dynamic recipe from a trusted publisher.

The model only fits at Q2:

Compare the larger Q2 model against a smaller model at Q4 or Q6 on your own tasks. Extra parameters sometimes win, especially on world knowledge; heavy compression can also erase that advantage. Download both and test prompts you use.

Test models, not labels

Perplexity tables and benchmark averages are helpful, but your use matters more. Test a small set of prompts covering:

facts you know well
a reasoning problem
instruction following
your preferred writing style
code in the language you use
the context length you actually need

Use the same temperature and prompt for each file. If Q4 answers the same way as Q6, keep the space. If it repeatedly misses details that Q6 catches, the larger quant earned its memory.