Speculative Decoding and MTP Explained

Typically LLM’s write one token at a time. It runs the model, picks a token, then runs the model again for the next one.

The repeated work is why a powerful GPU can still drip out a long answer. Speculative decoding tries to pull several tokens from one expensive check.

The model doesn’t blindly trust a guess. A smaller helper proposes tokens, and the main model checks them before they reach you.

Tokens first

A token is a small piece of text. It may be a whole short word, part of a longer word, or a punctuation mark.

A sentence like “The cat sat down” could be split into several tokens.

During generation, the model calculates a list of possible next tokens and their odds. It picks one, adds it to the text, and starts again. This is called autoregressive generation.

For a single chat, this process often waits on memory movement. The GPU has plenty of maths hardware, but it has to keep reading a large set of model weights for each new token.

How speculative decoding works

Speculative decoding puts a small draft model next to the large target model.

The draft model is quick, so it guesses a short run of future tokens. The target then checks those guesses together.

Say the draft proposes:

The cat sat on the mat

The target may accept:

The cat sat on

If it disagrees at the, that token and the draft tokens after it are thrown away. The target picks its own next token and drafting starts again.

The full loop:

1. The draft model proposes a few tokens.
2. The target checks all proposed positions in one pass.
3. Matching tokens are kept.
4. At the first rejection, later guesses are discarded.
5. Generation continues from the accepted text.

A proper speculative sampling method keeps the target model’s output distribution. It gains speed by checking work in parallel, rather than quietly accepting a weaker model’s answer.

Acceptance rate

The acceptance rate is the share of draft tokens that survive the target check.

A high acceptance rate means the target gets several useful tokens from one pass. A low rate means the computer spent time making guesses that got binned.

Predictable tasks tend to work well:

• code completion
• answers grounded in supplied documents
• repeated formats like JSON
• common phrases and boilerplate

Creative writing and translation are harder to predict. A draft that speaks English well may still be poor at guessing the target’s German translation.

The draft should also be much cheaper than the target. Loading a 14B draft beside a 30B target may eat enough VRAM and compute to erase the win. Small purpose-built drafters are easier to justify.

A rough timing example

Suppose the target needs 200 milliseconds for each token. Three normal steps take about 600 milliseconds.

A draft proposes three tokens quickly. The target checks all three in a 250-millisecond pass and accepts them. The same three tokens now cost around 250 milliseconds plus the small draft cost.

That’s an idealised example. Some guesses will fail, and checking a larger batch isn’t free. Reported gains often sit around 20 to 50 per cent for interactive use. Strong model and task pairings can do much better. Bad pairings can be a little slower than running the target alone.

MTP: prediction heads included with the model

Multi-Token Prediction, shortened to MTP, trains a model to predict more than one future token. Extra prediction heads learn to guess token two, token three, and later positions.

At inference time, those heads act as a built-in drafter. The main path still verifies the guesses.

Models use several names for related designs:

• DeepSeek V3 includes MTP heads
• newer Qwen models use NextN prediction
• Gemma releases can provide MTP draft data
• EAGLE attaches a small prediction system trained for one target model

They share one goal: produce useful candidate tokens without loading a normal second LLM.

Model-native heads can be small, but support has to exist across both the model file and the runtime. Finding MTP in a model name doesn’t guarantee your current Ollama, llama.cpp, or vLLM build will use it.

EAGLE in plain English

EAGLE uses information from inside the target model to draft future tokens. Its helper is trained for that exact target and can propose several possible branches.

Because it sees the target’s internal features, it can guess better than an unrelated small model. The cost is compatibility. An EAGLE drafter trained for one Llama release can’t be bolted onto any random model.

vLLM, SGLang and TensorRT-LLM support versions of EAGLE.

Why more guesses can be worse

It’s tempting to ask the draft for 20 tokens every time. Most of them may never be used.

Long drafts help when text is predictable and the GPU is waiting between requests. They hurt when the target rejects early, or when many users already keep the GPU busy.

At high request volume, ordinary batching can fill the GPU’s compute capacity. Speculation then piles more verification work onto an already full machine. Red Hat’s EAGLE-3 tests found gains at low request rates but a latency increase for a 70B model at high request rates.

So the best draft length depends on the job. There’s no winning number for every prompt.

Trying it in llama.cpp

Classic draft-target decoding uses a target file and a smaller compatible draft file:

llama-server \
  -m target-model.gguf \
  -md draft-model.gguf \
  -ngl 999

Some recent model files include an MTP drafter and use options like:

llama-server \
  -m model-with-mtp.gguf \
  --spec-type draft-mtp \
  -ngl 999 -fa on

Options are moving quickly. Run llama-server --help and check the model publisher’s instructions before you download a second large file.

Measure the target alone first. Then enable speculation and repeat the same prompt with the same context. Watch generated tokens per second, acceptance rate if available, VRAM use, and the time until the full answer finishes.

When should you bother?

Try speculative decoding when one person is chatting with a model, output speed is the pain point, and a known compatible drafter exists. It’s also promising for coding and document-based answers.

Leave it off when memory is already tight, the model writes quickly enough, or the server handles many requests at once. A setting that makes a benchmark faster can still make your own prompts slower.

If you’re a beginner, treat MTP as an optional speed switch. Get the target model running correctly, measure it, then test the switch. If the result isn’t faster on your prompts, turn it back off.

Read further

• NVIDIA’s introduction to speculative decoding
• Red Hat’s EAGLE-3 and vLLM tests
• The original speculative decoding paper
• llama.cpp repository
• Reasoning models and thinking tokens explained
• Ollama, llama.cpp or vLLM: which one should you use?