Ollama, llama.cpp or vLLM: Which One Should You Use?

You’ve picked a model. Now you need a program that can run it.

That program is usually called an inference engine or runtime. Ollama, llama.cpp and vLLM all do this job, but they’re built for different people.

If you’re setting up local AI for the first time, use Ollama. If you want more control or broad hardware support, use llama.cpp. If you’re serving many people from a powerful Linux GPU server, look at vLLM.

What happens when you send a prompt?

An LLM is mainly a big file full of learned numbers called weights. The engine loads those weights into memory and turns your text into small pieces called tokens.

Then it runs a loop:

1. Read the prompt and the tokens already generated.
2. Calculate the odds of each possible next token.
3. Pick one.
4. Add it to the answer and repeat.

The engine also manages your GPU, system memory, context cache and network API. A good engine can make models easier to install, faster under load, or able to run on hardware another engine can’t touch.

Ollama: the easiest place to start

Ollama wraps the fiddly parts in a small set of commands. Running a model can be as short as:

ollama run qwen3:8b

Ollama downloads the model, stores it, picks sensible settings and starts a chat. It also puts an API at localhost:11434, so apps like Open WebUI and coding tools can talk to your model.

Pick Ollama when:

• you want a working local model with little setup
• one person, or a few people, will use the server
• you’re building a small local app or coding assistant
• you don’t want to tune every memory setting by hand

Ollama uses other runtimes underneath. On many machines that’s llama.cpp. On recent Apple Silicon versions it can use MLX. You get a friendlier surface while the lower-level engine handles the model.

That convenience costs you some controls and therefore speed. If you want to choose exactly which tensors go to the GPU, make your own quantization, or test new runtime flags, llama.cpp is a better fit.

llama.cpp: control and wide hardware support

llama.cpp started as a way to run Llama models on a Mac without a big Python setup. It now works across NVIDIA, AMD and Intel GPUs, Apple Silicon, CPUs and a pile of odd hardware.

This one is my favourite, it exposes the right amount of controls for you optimise your tok/s and it’s fairly easy to setup configs once you’ve done it a few times.

Its common model format is GGUF. A GGUF is usually one file containing the model weights, tokenizer and model info. Download it and point llama.cpp at it:

llama-cli -m model.gguf -p "Explain why the sky is blue"

For an API server, use llama-server:

llama-server -m model.gguf -c 8192 -ngl 999

Pick llama.cpp when:

• you want to run a GGUF directly
• your model has to split across GPU memory and system RAM
• you need detailed control over quantization, context or GPU layers
• you’re on older, small or less common hardware
• you want to bundle an LLM inside a desktop app

The command line can look rough at first. The upside is that very little is hidden from you. LM Studio, Jan, GPT4All and several other friendly apps use llama.cpp too, so you can get its hardware support without living in a terminal.

vLLM: built for a queue of users

A home chat session usually has one request at a time. A public API may have dozens or hundreds. vLLM is built for the second case.

It keeps the GPU busy by combining work from many requests. New requests can join while older ones are still producing tokens. Its PagedAttention system also stores the context cache in blocks, which cuts wasted GPU memory.

None of that matters much when one person sends one prompt. Under heavy use it matters a lot. Research from Red Hat found llama.cpp and vLLM were comparable for one user on the tested system. At 64 concurrent users, vLLM produced about 44 times more tokens per second.

Pick vLLM when:

• many users or programs share one server
• you need steady response times under load
• you run Linux with supported NVIDIA or AMD hardware
• your models are stored as Hugging Face Safetensors, AWQ, GPTQ or FP8 files
• you need tensor parallelism across several GPUs

vLLM takes more setup and expects server-class conditions. It’s a poor first pick for a Windows laptop, a Raspberry Pi, or a single private chat window.

GGUF and Safetensors in plain English

The model format can narrow your choice before you start.

Format	Common tools	What to know
GGUF	llama.cpp, Ollama, LM Studio	One portable file, many quantization choices, strong support for home hardware
Safetensors	vLLM, Transformers, MLX	Common Hugging Face format, often split into several files
AWQ / GPTQ / FP8	vLLM and GPU-focused engines	Quantized formats made mainly for fast GPU serving
MLX	MLX and some Mac apps	Tuned for Apple Silicon’s shared memory

A model name alone doesn’t guarantee every file will work in every engine. Check the download format before you pick a runtime.

My beginner recommendation

Start with Ollama. Learn how model size, quantization and context affect your machine. If Ollama blocks a setting you need, move to llama.cpp or a llama.cpp-based desktop app.

Choose vLLM when you can name the server problem it solves: several users, high request volume, multiple GPUs, or response-time targets. Installing it for one local chat is extra work with little reward.

A short version:

Your situation	Pick
First local model	Ollama
Prefer a desktop interface	LM Studio
Need GGUF control or CPU/GPU splitting	llama.cpp
Apple Silicon and maximum speed	MLX or an app using MLX
Busy shared API server	vLLM

Read further

• Ollama documentation
• llama.cpp repository
• vLLM documentation
• Common AI model formats on Hugging Face
• LLM quantization without the alphabet soup
• GPU offloading for local LLMs
• Speculative decoding and MTP explained