GPU Offloading for Local LLMs

You download a 40 GB model for a card with 24 GB of VRAM. It can’t fit, but you’ve also got 64 GB of system RAM. Can you still run it?

Usually, yes. GPU offloading puts part of the model on the GPU and the rest in system RAM. It’s handy, but the speed can/will drop hard. It’s more tolerable with MOE’s.

Keep the whole model in VRAM when you can. Offload only when the model you want is too large to fit.

VRAM and system RAM do different jobs

Your GPU has its own fast memory called VRAM. An RTX 3090 can read from its VRAM at about 936 GB/s. A desktop may read DDR5 system RAM at roughly 50 to 90 GB/s.

During token generation, the computer repeatedly reads model weights. If some weights sit in slower system RAM, those parts hold up every token.

Picture a cook with ingredients in two rooms. The ones on the counter are quick to reach. The ones out in the garage still work, but every trip delays dinner.

What gets split?

Most LLMs are a stack of transformer layers. Each layer changes the data a little before handing it off to the next one.

llama.cpp can place some layers on the GPU and leave the rest on the CPU. The -ngl flag means “number of GPU layers”:

# CPU only
llama-cli -m model.gguf -ngl 0

# Put 20 layers on the GPU
llama-cli -m model.gguf -ngl 20

# Try to put every layer on the GPU
llama-cli -m model.gguf -ngl 999

-ngl 999 is a common shortcut. It means “offload as much as possible”, not that the model has 999 layers. If there isn’t enough VRAM, you’ll get an out-of-memory error and need a lower number.

LM Studio exposes a similar setting as a GPU offload slider. Ollama normally picks the split for you. vLLM has a cpu_offload_gb option, though it gives you less fine control than llama.cpp.

How much slower will it be?

There’s no fixed percentage. Speed depends on:

• your CPU (Mainly how many RAM channels can it support at full bandwidth. Server/workstation channels handle more than than consumer desktop chips.)
• the memory bandwidth of your RAM (ddr5 6000mhz > ddr4 3200mhz > ddr4 2666mhz. Octa channel > quad channel > dual channel)
• the model’s architecture

If only a few layers spill out of VRAM, the model may still feel comfortable to use. If half the model sits in ordinary desktop RAM, generation can feel much slower.

Before offloading, try a slightly smaller quant. A 30 GB Q6 model may be slower than a 20 GB Q4 model once the Q4 fits fully in VRAM. The smaller file also leaves room for the context cache.

Leave room for the context cache

Loading the weights is only part of memory use. The runtime keeps a KV cache, which stores information from your prompt and previous messages. A longer context needs a bigger cache.

Don’t fill a 24 GB card with a 24 GB model file and expect it to work. The runtime, compute buffers, desktop and KV cache all need memory. An 18-20 GB model is preferable on a 24 GB card, though the exact limit depends on your settings.

If a model nearly fits, drop the context length before you crush it down to a poor quant:

llama-server -m model.gguf -c 8192 -ngl 999

Here, -c 8192 sets an 8,192-token context. Dropping from 32K or 64K to 8K can free a surprising amount of memory.

Mixture-of-experts models are a special case

A dense model uses almost all its weights for every token. A mixture-of-experts, or MoE, model has many expert sections but activates only a few of them per token.

A model might hold hundreds of billions of total parameters while running a much smaller number on each step. That opens a different kind of split:

• keep attention and other always-used parts on the GPU
• keep the large bank of routed experts in system RAM
• let the CPU compute only the experts chosen for the current token

llama.cpp can place individual tensors rather than whole layers, with -ot. Recent builds also have options like --cpu-moe for keeping MoE experts on the CPU.

A typical invocation looks like this:

llama-server \
  -m model.gguf \
  -c 8192 \
  -ngl 999 \
  -fa on \
  --cpu-moe

Flag support shifts as llama.cpp develops, so check llama-server --help for your installed build. Big MoE files also need plenty of system RAM. A 100 GB GGUF doesn’t fit in a machine with 64 GB RAM just because only part of the model is active.

Prompt processing and token generation

Local LLM speed usually comes in two parts.

Prompt processing, sometimes called prefill, is how fast the model reads your prompt. GPUs are very good at this because many prompt tokens can be handled together.

Token generation, or decode, is how fast the answer appears. Tokens arrive one after another, and memory speed often controls the pace.

A split model may read a prompt quickly and then write the answer slowly. That’s normal. Some llama.cpp settings can temporarily lean harder on the GPU during prompt processing, but copying a very large set of weights across PCIe also takes time.

This is why one “tokens per second” number can mislead. Check both prompt processing and generation when you compare settings.

A safe tuning order

Use this order when a model won’t load:

1. Shorten the context to what you actually use day to day.
2. Try a Q4 quant such as Q4_K_M.
3. Close other programs using VRAM.
4. Offload as many layers as fit.
5. Measure prompt and generation speed with one prompt you know.
6. If it’s still too slow, use a smaller model rather than forcing a huge one.

Change one setting at a time. Keep a note of the model file, context, GPU layers and speed. It’s easy to forget which combination worked after ten restarts.

When offloading makes sense

Offloading is worth trying when:

• a preferred model misses your VRAM limit by a small amount
• you want to test a large model before buying more hardware
• an MoE model can keep its often-used parts on the GPU
• answer quality matters more than waiting time

Skip it when a smaller model already handles your work well. A fast 14B or 30B model is often more pleasant than a much larger model that types at walking pace.

Read further

• llama.cpp repository and current command options
• Doctor Shotgun’s llama.cpp MoE offloading guide
• LLM quantization without the alphabet soup
• RTX 3090 for local AI inference
• Ollama, llama.cpp or vLLM: which one should you use?