Text so far
The best pet is a
Inference & Hardware
Local AI
Learn how models run, what hardware they need, and how to build a practical local inference system.
Start here
Getting started with local AI.
Start by understanding the inference loop, build a useful stack, then evaluate it with measured results.
01 / Understand
Understand the inference loop
Learn how tokens, model weights, attention, and memory fit together before choosing a stack.
02 / BuildBuild a local stack
Choose a runtime and connect a model to the hardware and workflow you actually want to use.
03 / EvaluateMeasure before you optimize
Use measured benchmark results to compare inference systems, then use estimates only for early planning.
Interactive fundamentals
See the model’s work, step by step.
Inspect tokenization and cached attention, then compare measured benchmark results with separate early-stage estimates.
From prompt to next token
Tokens and IDs
TheID 464␠bestID 1266␠petID 4273␠isID 318␠aID 257
GPT-2 BPE tokens. ␠ marks a leading space included in the token.
Token source
Prompt IDs use the GPT-2 vocabulary (r50k_base). Continuations use a small example distribution.
Meaning and context
pet · 4273
[0.2, −0.7, …]Embeddings represent meaning. Attention mixes in useful context from the other tokens before the next-token scores are formed.
Possible next tokens
dog
cat
fish
Sampled result: dog
How sampling works
Temperature reshapes the distribution; top-p keeps the smallest cumulative set at or above p, then renormalizes it. Zero temperature chooses the highest-probability candidate. A fixed draw keeps this toy example reproducible.
One token is added
The best pet is a dog
Back revisits the previous generation's Pick step. Changing that choice starts a new continuation from there.
Watch the KV cache pull ahead
Local models keep data private
Start both together. Watch caching skip repeated work and finish first.
BuildReuseIllustrative timing
Without cache
0.0 s · 0/12 tokens
Ready
With cache
0.0 s · 0/12 tokens
Ready
Details
Cache memory
Total K/V work saved
K/V means keys and values: the attention data computed for each token. Words stand in for token positions here; a model’s actual tokenizer may split them differently.
0 cached positions
Representative positions are shown for large contexts.
Both lanes use the same teaching weights: 60 ms to build a position’s K/V, 15 ms to read a position, and 150 ms of other work per output token. Only the cached lane skips rebuilding old K/V. Above 16 context positions, both per-position costs are scaled by 16/context to keep the race watchable. These illustrative times are not hardware measurements or predicted speedups.
Attention still reads the context in both lanes. Step advances the shared clock to the next token arrival; it does not force the two outputs to stay in sync. The last sampled token remains uncached because no further decode step consumes it.
Cache bytes = positions × 32 layers × 2 (K + V) × 8 KV heads × 128 head dimension × bytes per element. Batch size 1; no sharding, quantization scales, or metadata overhead.
Measured benchmarks
View benchmarks →See what these systems actually do.
Compare reproducible inference measurements on the hardware, models, runtimes, and context depths that matter in practice.
Calculator estimate
Open the calculator →Estimate model fit and generation speed.
Compare GPUs and unified-memory systems, check VRAM fit, and estimate tokens per second for a model before you measure it.
Learn the fundamentals
Understand the inference loop.
Build a working mental model before comparing hardware or software.
Choose hardware
Match memory, bandwidth, and budget.
Start broad, then compare the devices and build types that fit your models.
Understand models
Read architecture claims in practical terms.
Connect model design to memory use, speed, quality, and local feasibility.
- 01Qwen3.8-27B measured local inferenceExisting measured work
- 02Qwen3.8 Flash n-gram memoryExisting experiment
- 03Abliteration and refusal in Qwen3.5-9BInterpretability experiment
- 04Dense versus MoE modelsModel analysis
- 05Reasoning models and thinking tokensFundamentals
- 06Speculative decoding and MTPFundamentals
- 07Local feasibility estimatesCalculator estimate
Build a local system
Turn compatible parts into a useful stack.
Start with the runtime choice, then wire up offloading, measurement, and experiments.
Roadmap
Planned guides.
These topics are planned for a future pass; the learning paths above link only to finished work.
- Future guide · Fundamentals
Parameters and model size
- Future guide · Fundamentals
Context windows
- Future guide · Fundamentals
Attention and mixture-of-experts
- Future guide · Software
Linux installation guides
- Future guide · Software
Model storage and downloads
- Future guide · Software
Security and privacy
All articles
Recent Local AI writing.
Chronological updates after the guided paths and interactive fundamentals.
Fundamentals3 min read
Quantization in Pictures: Fewer Bits, Fewer Colours
An interactive demo of the same photo at 2, 3, 4, 6 and 8 bits per channel, with and without error diffusion. All measurements run live in your browser.
Fundamentals7 min read
Why Local LLMs Slow Down as Conversations Grow
Why does an LLM run fast on turn one and crawl as context expands? We break down why memory bandwidth bottlenecks local inference, calculate theoretical speed limits, analyze measured dual-RTX 3090 context sweeps from 0K to 256K, and examine practical settings to keep generation fast.
Model analysis8 min read
DeepSeek V4.1 Flash: 552B Architecture, Engram Memory, and the 890 B/Token KV Cache
DeepSeek V4.1 Flash introduces 748B total parameters, a 196B engram table, 8B prefill / 16B decode asymmetric routing, native vision, and an 890 B/token KV cache. Here is how the architecture works, how it delivers 100–200 tok/s at 150 million tokens per dollar, and why it's my new daily driver for code.
Model analysis7 min read
GLM 5.3 Flash: My Experience Using It
A practical review of GLM 5.3 Flash covering its 320B-A18B architecture, API economics against DeepSeek V4 Flash and Luna, coding behavior, quantized memory footprint, and my plans to run it locally on an M5 Ultra Mac Studio.
Hardware5 min read
Apple M5 Ultra for Local AI
An overview of the Apple M5 Ultra Mac Studio for local large language models, covering its UltraFusion design, 1.2 TB/s memory bandwidth, expected prefill gains, decode expectations against NVIDIA Blackwell workstation cards, and how it compares to DGX Spark.
Model analysis10 min read
Qwen 3.8 27B: The Practical Sweet Spot for Local AI
An in-depth guide to Qwen3.8-27B for local inference. We examine its Artificial Analysis Intelligence Index standing, native multi-token prediction (MTP), weights versus KV cache quantization, parallel context slots, and firsthand daily driver observations on a dual-GPU workstation.
Model analysis12 min read
Qwen 3.8 Flash Explained: The 51B N-Gram Memory Architecture
An in-depth guide to Qwen3.8-Flash-Next and its 51B n-gram embedding table, updated with dual-RTX-3090 ExLlamaV3 results: hot expert placement, working MTP, and the memory tradeoff behind long-context inference.
Fundamentals5 min read
Long-Context and Attention in LLMs Explained (RoPE, YaRN)
A beginner's guide to Rotary Position Embeddings (RoPE) and YaRN. Learn how models track word order with clock hands, why advertised context differs from reliable reasoning, and how to test long sequences on local hardware.
Hardware13 min read
I Built a 48 GB VRAM Local AI Workstation with Two RTX 3090s
A dual RTX 3090 local AI workstation with 48 GB of distributed VRAM, Threadripper Pro, WRX80, 128 GB ECC RAM and enough expansion room for larger multi-GPU experiments.
Software8 min read
How to Access Your Local AI From Anywhere
A step-by-step guide to using Tailscale and SSH for private remote access to a home local AI server, with Ollama and Open WebUI examples.
Fundamentals8 min read
GPU Offloading for Local LLMs
A beginner's guide to splitting a local LLM between GPU VRAM and system RAM, with practical llama.cpp settings and advice for dense and mixture-of-experts models.
Software7 min read
Ollama, llama.cpp or vLLM: Which One Should You Use?
A beginner's guide to Ollama, llama.cpp and vLLM: what an inference engine does, how model formats fit in, and which tool suits a laptop, home server or busy API.
Fundamentals10 min read
Reasoning Models and Thinking Tokens Explained
A beginner's guide to chain-of-thought, reasoning models and test-time compute — from DeepSeek-R1 and GPT-o1 to the dot experiment and compressed reasoning — with practical advice for running distilled reasoning models on home hardware.
Fundamentals7 min read
Speculative Decoding and MTP Explained
A beginner's guide to speculative decoding, draft models and multi-token prediction, including when they speed up local LLMs and when the extra work slows them down.
Model analysis7 min read
Kimi K3 Is a 2.8-Trillion-Parameter Open-Weight Monster
Kimi K3 explained without the research-paper fog: 2.8T parameters, 104B active, Kimi Delta Attention, MXFP4 weights, benchmark results, API pricing, and what running it locally really takes.
Fundamentals8 min read
LLM Quantization Without the Alphabet Soup
A beginner's guide to local LLM quantization: what bits per weight means, how popular GGUF formats compare, when to choose Q2 through Q8, and why Q4 is a strong starting point.
Hardware10 min read
The Best Affordable Multi-GPU Motherboard for Local AI
A comparison of WRX80, TRX40, and AMD EPYC motherboards for multi-GPU local AI builds, with UK marketplace prices, PCIe lanes, memory bandwidth, Threadripper Pro CCD layouts, remote management, and the risks of importing used server hardware.
Model analysis10 min read
DeepSeek V4 Flash Is Almost Too Cheap to Meter
How DeepSeek V4 Flash cuts inference cost with sparse experts, compressed attention, FP4/FP8 weights, disk-backed prefix caching and speculative decoding, plus API cost estimates and real local benchmarks on RTX PRO 6000, DGX Spark and Apple Silicon.
Hardware10 min read
Apple Silicon Macs for Local LLMs
A concise guide to the M3 Ultra Mac Studio, M4 Max Mac Studio, and M5 Max MacBook Pro for local LLMs, including real benchmarks, current and used pricing, MLX and Metal software, Thunderbolt clustering, and current preorder context for the M5 Ultra.
Hardware22 min read
NVIDIA DGX Spark and GB10 for Local LLMs
A practical guide to NVIDIA DGX Spark and other GB10 systems for local LLMs, covering real token speeds, memory bandwidth, MoE models, clustering, thermals, software support, pricing, Strix Halo, and Mac Studio alternatives.
Hardware10 min read
AMD Strix Halo for Local LLMs
A concise guide to AMD Strix Halo for local LLMs, covering real token speeds, memory bandwidth, MoE models, ROCm and Vulkan support, clustering, and the 192 GB Gorgon Halo refresh.
Buying guide22 min read
Best GPUs and Devices for Local AI Inference
A comparison of GPUs, workstation cards, unified-memory devices, and CPU inference setups for local AI, focused on price, speed, memory capacity, software support, and practical tradeoffs.
Benchmark20 min read
RTX 3090 for Local AI Inference
A practical look at the RTX 3090 for local AI workloads, covering specs, VRAM, bandwidth, TFLOPS, used pricing, power limiting, LLM performance, image generation, and why it remains one of the best-value GPUs for running open models at home.