local-ai / July 9, 2026 / 22 min read
Best GPUs and Devices for Local AI Inference
A practical guide to choosing hardware for local LLMs, image generation, and AI workloads.
Intro
Choosing hardware for local AI is mostly a tradeoff between price, speed, and capacity. You can only have just two of the three.
A fast, high-capacity card is expensive. A cheap, high-capacity unified memory device is usually slow. A cheap, fast card usually does not have enough memory for larger models.
For local LLM inference, the two numbers to pay attention to first are memory bandwidth and memory capacity. Memory bandwidth has a major effect on decode speed, which is the output tokens per second you feel while chatting with a model. Memory capacity determines what model size, quantization level, context length, and concurrency level you can actually fit.
There are other important factors: software support, power efficiency, driver quality, used-market pricing, cooling, form factor, and whether your workload depends on CUDA. But most decisions start with the same question: how much model do you need to fit, and how fast do you need it to run?
This guide is meant to help you choose the right GPU or device for your model and workload. There is no one-size-fits-all answer, but the decision gets much easier once you separate capacity, speed, price, and software support.
Quick Recommendations
| Pick | Hardware | Why |
|---|---|---|
| Local baller | RTX PRO 6000 Blackwell 96GB | Huge VRAM, very high bandwidth, Blackwell features, strong CUDA support, workstation-friendly power options, extremely expensive |
| Best used value | RTX 3090 24GB | Affordable used, enough VRAM for many ~30B-class quantized dense models, mature CUDA support, fast enough for most local inference |
| High-end workstation | RTX 5090 32GB | Very high bandwidth, strong performance, useful VRAM for ~30B-class models with higher context, but expensive and power-hungry |
| Middle-ground pro card | RTX PRO 5000 Blackwell 48GB | Better capacity than consumer cards, efficient, less extreme than the 96GB workstation option |
| Budget wildcard | Radeon AI PRO R9700 32GB | Good memory capacity for the price, but weaker software ecosystem and lower bandwidth than high-end NVIDIA |
| Solid entry point | RTX 3060 12GB | Cheap CUDA card for testing local AI before committing to a larger build |
| High-capacity used workstation | RTX A6000 48GB | Good VRAM, decent bandwidth, CUDA support, and potentially fair used-market pricing |
| Cheap high-capacity tinkering | Older datacenter GPUs | Can offer lots of VRAM per dollar, but cooling, drivers, platform support, and missing modern features can be painful |
| Massive-memory niche | CPU server inference | Can load huge models cheaply with enough RAM, but decode speed is very slow compared with GPUs |
How to Think About the Tradeoff
The core tradeoff is simple: price, speed, and capacity. Pick two.
If you want speed and capacity, you are usually looking at expensive workstation or datacentre class hardware. If you want capacity and relatively low price, you are usually looking at older used cards, server cards, unified-memory systems, or CPU inference. If you want speed and low price, you are usually giving up VRAM capacity.
For local AI, capacity is often the first constraint people hit. You can have a fast GPU, but if the model, quantization, context, and KV cache do not fit in memory, the setup becomes compromised. Offloading to system RAM can work, but it reduces speed significantly.
Speed is mostly felt in two places: prefill and decode. Prefill is the processing of the input prompt and context. Decode is the generation of new output tokens. For chat use, decode speed is the number most people notice. For document-heavy workflows, coding agents, long context, batch serving, or multi-user workloads, prefill and memory pressure become more important.
Hardware Comparison
Use this table as a starting point, not a final verdict. Pricing changes quickly, especially on the used market. Specs should be checked against TechPowerUp, manufacturer pages, and current listings before buying.
| Hardware | Memory | Bandwidth | Power | Price Class | Speed Class | Software | Best For | Main Catch | Detail |
|---|---|---|---|---|---|---|---|---|---|
| RTX PRO 6000 Blackwell | 96GB GDDR7 ECC | 1,792GB/s | 600W / 300W Max-Q | Extreme | Very fast | CUDA | No-compromise local workstation | Very expensive | Details |
| RTX 5090 | 32GB GDDR7 | 1,792GB/s | 575W | High | Very fast | CUDA | Fast local workstation builds | Pricing, availability, and power draw | Details |
| RTX PRO 5000 Blackwell | 48GB / 72GB GDDR7 ECC | 1,344GB/s | 300W | High | Fast | CUDA | Efficient high-capacity workstation builds | Price versus RTX 5090, RTX A6000, and RTX PRO 6000 | Details |
| RTX PRO 4500 Blackwell | 32GB GDDR7 ECC | 896GB/s | 200W | Medium/high | Moderate/fast | CUDA | Efficient 32GB workstation builds | RTX 5090 is much faster if you can tolerate power and price | Details |
| RTX PRO 4000 Blackwell | 24GB GDDR7 ECC | 672GB/s | 140W | Medium/high | Moderate | CUDA | Low-power professional 24GB builds | RTX 3090 is often much better value used | Details |
| RTX 3090 | 24GB GDDR6X | 936GB/s | 350W | Medium used | Fast enough | CUDA | Best used-value local LLM card | Older, hot, inefficient, used-card risk | Details |
| RTX A6000 | 48GB GDDR6 ECC | 768GB/s | 300W | Medium/high used | Decent | CUDA | Used 48GB CUDA builds | Older Ampere workstation card | Details |
| A100 80GB PCIe | 80GB HBM2e | 1,935GB/s | 300W | High used | Very strong | CUDA | Serious high-capacity inference | Server card, platform and cooling complexity | Details |
| V100 PCIe | 16GB / 32GB HBM2 | 900GB/s | 250W | Low/medium used | Mixed | CUDA, older | Cheap server experiments | No BF16, no FP8, old tensor cores, weaker modern inference support | Details |
| Radeon AI PRO R9700 | 32GB GDDR6 ECC | 640GB/s | 300W | Medium | Moderate | ROCm | AMD capacity value | No CUDA, weaker app support, bandwidth is mediocre for 32GB | Details |
| Radeon RX 7900 XTX | 24GB GDDR6 | 960GB/s | 355W | Medium | Moderate/fast | ROCm | AMD consumer experimentation | Software support is still more awkward than NVIDIA | Details |
| Intel Arc Pro B70 | 32GB GDDR6 | 608GB/s | 230W branded / 160–290W range | Medium | Experimental | oneAPI / OpenVINO / OpenCL | Low-cost 32GB AI experimentation | Young software stack, weaker ecosystem than CUDA | Details |
| RTX 3060 12GB | 12GB GDDR6 | 360GB/s | 170W | Low | Entry | CUDA | Testing local AI cheaply | Smaller models only | Details |
| Mac Studio M3 Ultra | 96GB–512GB unified | 819GB/s | Up to 480W system | High | Mixed | Metal / MLX / llama.cpp | Huge unified-memory local use | No CUDA, high price, availability varies | Details |
| DGX Spark | 128GB LPDDR5x unified | 273GB/s | System appliance | High | Mixed | CUDA / DGX OS | CUDA unified-memory development | Very low bandwidth for the memory size | Details |
| CPU server inference | System RAM dependent | DDR4 / DDR5 channels | Platform-dependent | Variable | Slow | llama.cpp etc. | Loading huge models cheaply | Decode is very slow compared with GPUs | Details |
How Much VRAM Do You Need?
These are rough practical ranges, not hard rules. The exact answer depends on model architecture, quantization, context length, KV cache size, batch size, backend, and whether you are willing to offload layers to system RAM.
| Memory | Practical Meaning |
|---|---|
| 8GB | Gets you on the ladder, but you will want more. Can still work for some image-generation workflows and small local models. |
| 12GB | Entry level for local LLMs. Good for smaller models and learning the tooling. |
| 16GB | Comfortable for smaller models. ~30B models require low quantization, low context, or offloading. Good for image generation and usable for some video-generation workflows. |
| 24GB | The current sweet spot for many local AI users. Good for ~30B quantized dense models and higher context. |
| 32GB | Great for ~30B models, higher context, and some concurrency. |
| 48GB | Starts opening the door to 70B quantized models, smaller MoE models, and more comfortable multitasking. |
| 80GB+ | High-capacity territory. Useful for larger models, higher precision, batching, or serious experimentation. |
| 128GB+ unified / system memory | Can load very large models, but speed depends heavily on memory bandwidth and software stack. |
Why Bandwidth Matters
Local LLM inference is often memory-bandwidth limited. During generation, the system repeatedly moves model weights through memory to produce the next token. That makes memory bandwidth one of the most important practical speed indicators.
This is why a GPU can look powerful on paper but still be a mediocre local LLM card if it has too little memory bandwidth or not enough VRAM. Gaming performance does not map cleanly to inference performance. For local AI, memory capacity and memory bandwidth are usually more important than gaming benchmarks.
VRAM capacity determines whether the model fits. Memory bandwidth helps determine how fast it feels once it does.
NVIDIA Consumer Cards
RTX 3060 12GB
The RTX 3060 12GB is the low-risk entry point. It is not fast by modern standards, and 12GB becomes restrictive quickly, but it is cheap and has CUDA support.
This is the kind of card to buy if you want to test whether local AI is something you actually use before spending serious money. It is good for smaller models, basic Ollama use, light experimentation, and learning the local inference stack.
The main limitation is capacity. Once you start trying larger models, higher context, image workflows, or more demanding tools, 12GB starts feeling cramped.
RTX 3090
The RTX 3090 is the fan favourite because the used-market value is hard to beat. It has 24GB of GDDR6X, 936GB/s of memory bandwidth, mature CUDA support, and enough speed for a lot of dense local models.
The reason it keeps showing up in local AI builds is simple: 24GB is still a useful capacity tier, and 936GB/s is still respectable. It is faster on bandwidth than the RTX PRO 4000 Blackwell and close to the RTX PRO 4500 Blackwell, while usually being much cheaper used.
The downsides are power, heat, and age. It is a 350W Ampere card, many used cards have been mined on or run hard, and the cooler condition matters. If you buy one, check memory temperatures, fan noise, warranty status, power connectors, and whether the card fits your case. It is also recommened to undervolt and powerlimit, you wont lose much throughput but you can save on the energy bill and the heat output.
RTX 5090
The RTX 5090 is the high-speed consumer workstation option. It has 32GB of GDDR7 and 1,792GB/s of memory bandwidth, which puts it in the same bandwidth class as the RTX PRO 6000 Blackwell. For local LLM inference, that bandwidth matters because decode is often memory-bandwidth sensitive.
The tradeoff is power and price. The 575W board power means you should think carefully about PSU quality, case airflow, connector safety, undervolting, and power limiting. For inference, power limiting and undervolting can often make sense because you may be able to cut a lot of power without losing proportional decode performance.
The RTX 5090 makes sense when speed matters and 32GB is enough. If you need to fit larger models more comfortably, a 48GB, 80GB, or 96GB card may be more useful even if the raw speed is lower.
NVIDIA Workstation and Datacenter Cards
RTX PRO 6000 Blackwell 96GB
The RTX PRO 6000 Blackwell is the local baller option. It combines 96GB of GDDR7 ECC memory with 1,792GB/s of bandwidth, which is a rare mix of capacity and speed in a workstation-friendly card.
There are two workstation versions worth separating: the full-power 600W card and the 300W Max-Q version. For local inference, the Max-Q version is especially interesting because it keeps the same 96GB memory capacity and bandwidth while fitting into a much more reasonable power envelope. The full-power card is for people who care about maximum throughput; the Max-Q card is arguably the cleaner local workstation choice if efficiency, acoustics, and multi-GPU density matter.
This is not a value card anymore, the MSRP just went up 50%. It is what you buy when you want large-model headroom, CUDA, modern Blackwell features, and fewer compromises than old server hardware.
Full RTX PRO 6000 Blackwell guide
RTX PRO 5000 Blackwell 48GB / 72GB
The RTX PRO 5000 Blackwell is the serious middle-ground pro card. It gives you 48GB of GDDR7 ECC, 1,344GB/s of memory bandwidth, and a 300W power limit. There is also a 72GB variant, which would make the card even more interesting for local AI if pricing is was not absurd.
The 48GB model already changes what you can comfortably run. It gives more room for larger dense models, smaller MoE models, higher context, larger KV cache, and multi-app workflows than 24GB or 32GB cards. The 1,344GB/s bandwidth is meaningfully faster than RTX 3090, RTX A6000, RTX PRO 4500, and RTX PRO 4000, while using much less power than the RTX 5090.
The question is price. If it is too close to the RTX PRO 6000, the 96GB card becomes tempting. If it is too close to a scalped RTX 5090, the pro card may be the better capacity play. If used RTX A6000 pricing is good, that remains the older but potentially cheaper CUDA 48GB comparison.
Full RTX PRO 5000 Blackwell guide
RTX PRO 4500 Blackwell 32GB
The RTX PRO 4500 Blackwell is more interesting: 32GB of GDDR7 ECC, 896GB/s of bandwidth, and 200W power.
That puts it close to RTX 3090 bandwidth while using far less power and giving you 8GB more memory. For a quiet workstation, compact professional build, or always-on inference box.
The problem is value. If an RTX 5090 is available near MSRP, the 5090 gives far more bandwidth and raw performance for much lower cost. The RTX PRO 4500 makes sense when you specifically care about efficiency, workstation support, ECC, 32GB capacity, and lower power.
Full RTX PRO 4500 Blackwell guide
RTX PRO 4000 Blackwell 24GB
The RTX PRO 4000 Blackwell is a modern efficient 24GB workstation card. It has 24GB of GDDR7 ECC, 672GB/s of bandwidth, and a 140W power limit.
That makes it interesting for compact, quiet, or power-limited builds. It is much more efficient than an RTX 3090. But for local LLM inference, it has a hard value problem: the RTX 3090 also has 24GB of VRAM and about 40% more memory bandwidth.
The RTX PRO 4000 is not the card I would buy for max tokens per dollar. I would consider it if I needed a professional, efficient, lower-power 24GB CUDA card and did not want the heat, age, and used-market risk of an RTX 3090.
Full RTX PRO 4000 Blackwell guide
RTX A6000 48GB
The RTX A6000 is older, but it remains interesting because 48GB of CUDA-supported VRAM is still very useful. If used pricing is fair, it can be a practical high-capacity option without moving into the most expensive current-generation workstation cards.
It is not the fastest card by modern standards, but for local inference, capacity often matters more than peak compute. If the goal is to fit larger models comfortably, the RTX A6000 can still make sense.
The main things to watch are used condition, price, and whether a newer 48GB card is close enough in price to be worth the jump.
A100 80GB
The A100 80GB is still serious hardware. It has excellent memory capacity and HBM bandwidth, and it can be priced fairly relative to the most expensive workstation cards.
The problem is that it is an old datacenter card, not a normal desktop GPU. Cooling, power, platform compatibility, and used-market condition matter a lot. It is not the card I would recommend to someone who wants a simple local AI desktop.
It makes more sense for people who are comfortable building around server hardware and want high capacity with strong bandwidth.
V100
The V100 can be attractive because it is cheap relative to its original class, but it is old enough that missing modern features matter.
The lack of newer datatype support and modern inference optimizations can hurt practical performance. Prefill can be especially underwhelming compared with newer cards. It can still be useful for cheap experimentation, but it is not where I would start unless the price is excellent and the user understands the tradeoffs.
AMD Cards
AMD is interesting when capacity per dollar is good, but the software story is still the main caveat.
ROCm has improved, and LLM inference is increasingly usable. But CUDA remains the safer default for the broadest tool support. This matters especially for workloads like ComfyUI, where NVIDIA compatibility is usually less painful.
For LLM inference, AMD can make sense if the price and VRAM are compelling and you are comfortable troubleshooting. For people who want the least friction, NVIDIA is still the easier recommendation.
Radeon AI PRO R9700
The Radeon AI PRO R9700 is the AMD budget wildcard. It has 32GB of GDDR6, 640GB/s of bandwidth, ECC support on Linux, and a 300W board power.
The capacity is the appeal. 32GB gives you more room than 24GB cards for higher context, less aggressive quantization, and some concurrency. The weakness is bandwidth: 640GB/s is lower than the RTX PRO 4000 Blackwell, RTX 3090, RTX PRO 4500 Blackwell, RX 7900 XTX, and the higher-end NVIDIA Blackwell cards. It could also be good if you want to build an affordable cluster.
The software story is the other caveat. ROCm is improving, and LLM inference on AMD is much more realistic than it used to be. But CUDA is still the safer ecosystem, especially for ComfyUI, random extensions, research code, and tools that assume NVIDIA. I would consider the R9700 if capacity matters and you are comfortable troubleshooting.
Full Radeon AI PRO R9700 guide
Radeon RX 7900 XTX
The Radeon RX 7900 XTX is still an interesting consumer AMD comparison because it has 24GB of GDDR6 and up to 960GB/s of memory bandwidth. On paper, that is slightly more bandwidth than the RTX 3090.
The catch is software. If your workload runs well under ROCm, the RX 7900 XTX can be a strong value card. If you depend on CUDA-heavy tools, ComfyUI extensions, or software that assumes NVIDIA, it becomes harder to recommend.
For LLM inference, this card is worth testing. For lowest-friction local AI, NVIDIA is still the safer default.
Intel Cards
Intel cards are worth tracking, but they are not the default recommendation for most local AI users yet.
The issue is not just hardware. It is software maturity, framework support, driver behavior, and whether the tools people actually use work reliably. If pricing and memory capacity are strong enough, Intel could become interesting for experimenters, but it is not the safe choice for a first local AI build.
Intel Arc Pro B70
The Intel Arc Pro B70 should not be listed as hypothetical anymore. It is a real 32GB pro GPU with 608GB/s of memory bandwidth, 32 Xe2 cores, 256 XMX engines, and a 230W Intel-branded configuration, with partner designs ranging from 160W to 290W.
That makes it genuinely interesting on paper: 32GB of VRAM, decent power, and lower pricing than many NVIDIA pro cards. For local AI, the memory capacity is the main reason to care.
The catch is the software stack. Intel has OpenVINO, oneAPI, OpenCL, Vulkan, and Linux support, but it is not CUDA. That means fewer tools, more edge cases, and less community knowledge. I would treat the B70 as an experimenter’s card, not the default recommendation for someone who just wants everything to work.
Unified-Memory Devices
Discrete GPUs are not the only way to run local AI. Unified-memory systems are interesting because they can make much larger memory pools available to the model.
The tradeoff is bandwidth and software support. A system with a huge unified-memory pool may load models that a normal GPU cannot, but it may not generate tokens as quickly as a high-bandwidth discrete GPU.
Mac Studio M3 Ultra
The Mac Studio M3 Ultra is not a CUDA box, but the unified-memory ceiling is the appeal. Configurations start at 96GB and going up to 512GB unified memory, with 819GB/s of memory bandwidth.
That memory pool is the whole reason local AI people care. You can load models that would not fit on normal consumer GPUs, and you can do it in a quiet, compact, efficient desktop. The tradeoff is speed and software. 819GB/s is good for unified memory, but it is less than high-end discrete GPU bandwidth, and the Metal / MLX / llama.cpp ecosystem is not the same as CUDA. With exo and thunderbolt you can cluster 4 of these together to run trillion parameter models at high quants and usable throughput, prefill is not great however and the used pricing has sky rocketed after apple discontinued the 512GB model.
This is a capacity-first device. It makes sense if you value large local models, quiet hardware, macOS, and unified memory. It is less compelling if you need CUDA, maximum prefill, maximum decode, ComfyUI compatibility, or best performance per dollar.
For a x86 unified-memory alternative, see AMD Strix Halo for Local LLMs — up to 128 GB of LPDDR5X at 256 GB/s.
DGX Spark
DGX Spark is a CUDA unified-memory appliance. It has 128GB of LPDDR5x coherent unified memory, 273GB/s of memory bandwidth, a Grace Blackwell architecture, DGX OS, and ConnectX-7 networking.
The good part is obvious: 128GB of unified memory with NVIDIA’s software stack is useful for development, testing, and larger models that do not fit into ordinary VRAM. The clustering story is also better than most consumer devices because of ConnectX-7 at 200Gb/s.
The catch is bandwidth. 273GB/s is low compared with basically every serious discrete GPU in this guide. That means DGX Spark is more of a capacity and development box than a decode-speed monster. It’s a good option for larger MOEs with dual sparks running Deepseek V4 Flash, Mimo V2.5 and quad sparks running GLM 5.2.
CPU Server Inference
CPU inference is the cheap massive-memory option. With a server board, enough DDR4 or DDR5 RAM, and enough memory channels, you can load models that would be impossible on consumer GPUs.
The tradeoff is speed. Even with many memory channels, CPU inference is usually slow for interactive use compared with GPUs. It can still make sense for huge models, experiments, batch jobs, or cases where capacity matters much more than responsiveness.
Memory channels matter a lot. DDR4 bandwidth is usually not impressive for inference. DDR5 is better, especially with many channels, but it is still slow compared with high-bandwidth GPU memory.
CPU topology matters too. To use the available memory bandwidth, the CPU needs enough chiplets, cores, memory channels, and NUMA configuration to actually feed the workload. A big RAM number alone does not guarantee good inference speed.
CPU inference is not the best default recommendation, but it is one of the cheapest ways to experiment with enormous models if you can tolerate very slow output.
Full CPU server inference guide
Software Support Matters
For local AI, hardware specs are only half the story. Software support can decide whether a build is pleasant or miserable.
CUDA is still the safest default. NVIDIA has the broadest support across inference engines, image generation tools, ComfyUI, CUDA extensions, research code, and optimization libraries. If you want the fewest surprises, NVIDIA is usually the easiest path.
ROCm is improving, especially for LLM inference, but it still does not match CUDA for compatibility. AMD can be a good choice when the VRAM and price are compelling, but you should expect more troubleshooting.
Metal is viable for many Mac workflows, especially through tools like llama.cpp and MLX, but it is not a replacement for CUDA if your workload depends on the NVIDIA ecosystem.
Intel is worth watching, but I would not recommend it as the default unless you specifically want to experiment with Intel hardware and software.
Buying Rules of Thumb
Do not buy local AI hardware based only on gaming benchmarks. Gaming performance, raster performance, and even peak compute do not tell the full story for inference.
Start with the model you want to run. Then estimate the memory requirement, context length, quantization level, and backend. After that, compare memory capacity, bandwidth, software support, power, and price.
A good local AI card fits your actual workload with the least friction at a price you can justify.
Notes and Future Work
This guide is meant to be a living reference. The detailed posts will include fuller specs, buying notes, setup guidance, compatibility issues, and practical testing for each card or device.
Benchmarks are intentionally kept out of this overview because they deserve more controlled detail. The individual hardware pages will include testing notes, backend-specific results, power behavior, setup problems, and workload-specific observations.
Changelog
- 2026-07-09: Initial upload.