RTX 3090 for Local AI Inference

RTX 3090: The Local AI Sweet Spot

The RTX 3090 is still the fan-favourite GPU for local AI and the reasons are simple: 24 GB of VRAM, 936 GB/s of memory bandwidth, CUDA support and sane used-market pricing.

On paper, it is a 2020 gaming card. In practice, it’s an accidentally affordable local AI workstation card. NVIDIA launched it as the “BFGPU” with 10,496 CUDA cores, 24 GB of GDDR6X memory, and a $1,499 starting price on September 24, 2020.

For gaming, the RTX 3090 was always a bit absurd. For local AI, that absurdity aged beautifully.

It has enough VRAM to fit serious quantized models. It has enough bandwidth to produce good decode speed on dense models. It has mature CUDA support. It works well with llama.cpp, Ollama, ExLlamaV2, ComfyUI, PyTorch, and most AI tooling without drama. And unlike newer flagship cards, used RTX 3090s can still be found at prices that make sense for hobbyists and indie devs.

NVIDIA probably did not intend to create the perfect second-hand local AI card in 2020, but that is what happened.

Key Specs

Spec	RTX 3090
Architecture	NVIDIA Ampere
GPU	GA102
CUDA cores	10,496
Tensor cores	328, 3rd gen
RT cores	82, 2nd gen
Base clock	1.40 GHz
Boost clock	1.70 GHz
FP32 performance	~35.6 TFLOPS
NVIDIA launch marketing	36 shader TFLOPS, 69 RT TFLOPS, 285 Tensor TFLOPS
VRAM	24 GB GDDR6X
Memory bus	384-bit
Memory bandwidth	936.2 GB/s
Memory speed	19.5 Gbps effective
PCIe	PCIe 4.0 x16
NVLink	Yes, 2-way supported
Encoder / decoder	7th-gen NVENC, 5th-gen NVDEC
Rated board power	350 W
Recommended PSU	750 W
Founders Edition size	313 mm, 3-slot

The important number for local LLM inference is the combination of 24 GB VRAM + 936 GB/s bandwidth.

Why Bandwidth Matters for LLMs

LLM inference has two main phases: prompt processing and token generation.

Prompt processing is the phase where the model digests the input context. This is more compute-heavy and benefits from raw shader/Tensor throughput.

Token generation is the phase you feel while chatting. This is the model producing one token after another. For many local single-user workloads, this phase is heavily memory-bandwidth bound because the GPU repeatedly streams model weights and KV-cache data through memory. Research on LLM inference phase-splitting describes prompt computation as more compute-intensive and token generation as more memory-intensive.

That is why the RTX 3090 still feels good. Its 936 GB/s bandwidth is high enough that dense Q4 models can decode at useful interactive speeds, especially in llama.cpp or other optimized inference stacks.

Pricing Reference

As of 09-07-2026, used pricing varies a lot by country, condition, cooler, seller reputation, and whether the card has been mined on or abused. A UK price tracker currently shows the RTX 3090 around £822.70 used, with a UK launch reference of £1,439. A US-focused used-price tracker shows an average used RTX 3090 around $700, while some marketplaces and listings are much higher.

Using current exchange-rate references around $1 = £0.746 and £1 ≈ $1.34, the rough pricing picture looks like this.

Price point	GBP estimate	USD estimate	Notes
Launch MSRP	~£1,439 UK reference	$1,499	US MSRP official; UK launch pricing varies by VAT/retailer
Sensible used target	£650–£850	$800–$1,100	Best value zone if condition is good
UK used tracker	£822.70	~$1,100	Current UK tracker reference
US used tracker	~£522	$700	Current US used average reference

The sweet spot is still the used market. A clean RTX 3090 at around £700–£850 or $800–$1,000 is hard to beat for local AI. Below that, inspect carefully.

Why It Still Beats Many Newer Cards for Local AI

The RTX 5080 is much newer, faster in many gaming and AI paths, and based on Blackwell. But it ships with 16 GB of VRAM, which is a hard limit for local LLMs. NVIDIA’s official RTX 5080 spec page lists 16 GB GDDR7 on a 256-bit bus.

The RTX 5090 finally beats the 3090’s VRAM capacity in a consumer GeForce card with 32 GB GDDR7, but it launched at a much higher $1,999 starting price and current market prices remain ugly. NVIDIA lists the RTX 5090 as a 32 GB GDDR7 card starting at $1,999. UK price tracking in July 2026 shows the RTX 5090 around £3,599 new and £3,066 used, against a UK launch reference around £1,919.

So yes, the 5090 is technically much better. It has more VRAM, more bandwidth, newer Tensor cores, FP4 support, and better performance. But for value, the 3090 is still absurdly competitive. For a full comparison across all current GPU options, see the Best GPUs and Devices for Local AI guide.

The 3090 held the consumer VRAM crown at 24 GB from September 2020 until the RTX 5090’s 32 GB launch in early 2025. That is roughly four and a half years where NVIDIA’s newer gaming cards matched or undercut it on capacity rather than clearly beating it.

What 24 GB Actually Lets You Run

The RTX 3090 is a “run the good stuff locally” GPU.

With 24 GB of VRAM, it is ideal for:

7B to 14B models at high quantization or full-ish precision
20B to 35B-class models at Q4 or similar quantization
Dense 27B to 31B models with sensible context lengths
MoE models where active parameters are low
Local coding assistants
Agentic workflows where throughput matters
SDXL, FLUX, and ComfyUI workflows
Small-to-medium video generation experiments

The limitation is context. A Q4 model might fit, but a huge context window can push the KV cache over the edge. Long context is not free. For local use, 8K to 32K context is usually the practical range depending on model, quant, backend, and whether you quantize KV cache. I would recommend q8 quants for KV cache.

For a single 3090, I would generally use llama.cpp / GGUF first. It is flexible, efficient, and forgiving. For multi-GPU serving, vLLM becomes more attractive because it supports tensor parallel and pipeline parallel serving; its docs show multi-GPU serving with --tensor-parallel-size. llama.cpp also supports CUDA acceleration for NVIDIA GPUs.

Recommended Local LLM Targets

Qwen3.6-27B

For coding, Qwen3.6-27B is one of the most exciting dense models in this class. The model card highlights agentic coding improvements, frontend workflows, and repository-level reasoning.

This is the model I would use when quality matters more than raw throughput.

Benchmark results (llama.cpp, Q4_K_M, Q8 KV cache, single RTX 3090):

Context	Prompt Processing	Token Generation
512	1228.67 t/s	39.14 t/s
2,048	1172.31 t/s	39.11 t/s
4,096	1159.45 t/s	39.08 t/s
8,192	1090.04 t/s	39.05 t/s
16,384	1049.01 t/s	38.98 t/s
32,768	966.87 t/s	38.85 t/s
65,536	821.75 t/s	38.69 t/s

PP scales well with the 3090’s compute, hitting over 1200 t/s at short contexts and remaining above 800 t/s even at 64K. TG stays remarkably stable around 39 t/s across the board — the 936 GB/s bandwidth is enough that the Q4 model’s memory reads are the only bottleneck.

Model card: Qwen3.6-27B.

Qwen3.6-35B-A3B

Qwen3.6-35B-A3B is the more throughput-friendly option. It is a 35B-class MoE model with only a smaller subset of parameters active per token, which makes it much faster than a dense 35B would be. Its model card lists compatibility with Transformers, vLLM, SGLang, and KTransformers.

For agentic workflows, this is often the better practical pick because speed changes how you use the model. More attempts, faster tool calls, quicker retries, and lower waiting time matter.

Expected ballpark on RTX 3090: 80–120 tok/s in favourable local setups.

Model card: Qwen3.6-35B-A3B.

Gemma 4 31B

Gemma 4 31B is the dense Gemma option. Google’s Gemma 4 model cards describe the family as including both dense and MoE architectures, with sizes including E2B, E4B, 12B, 26B A4B, and 31B.

For coding, it may sit slightly behind the best Qwen coding models, but for creative prose, general knowledge, and multimodal-style workflows, it is excellent.

Benchmark results (llama.cpp, Q4_K_M, Q8 KV cache, single RTX 3090):

Context	Prompt Processing	Token Generation
512	1087.93 t/s	34.37 t/s
2,048	1076.66 t/s	34.31 t/s
4,096	1050.22 t/s	34.28 t/s
8,192	1017.71 t/s	34.24 t/s
16,384	943.72 t/s	34.19 t/s
32,768	836.60 t/s	34.14 t/s

PP is competitive with Qwen at short contexts and degrades gracefully as context grows. TG holds steady at ~34 t/s — slightly lower than Qwen’s 39 t/s, as you’d expect from the larger parameter count and wider model.

Model card: google/gemma-4-31B-it. The variant tested here is the uncensored QAT finetune by HauhauCS, available at HauhauCS/Gemma4-31B-QAT-Uncensored-HauhauCS-Balanced-MTP.

Gemma 4 26B-A4B

Gemma 4 26B-A4B is the MoE-style Gemma option. The model card says Gemma 4 includes a 26B A4B variant, and the A4B design means fewer active parameters during inference than the total model size implies.

This makes it a strong agentic model: high throughput, capable reasoning, and enough quality for many practical tasks. On the 3090, expect roughly 64–115 tok/s depending on quant and backend.

Model card: Gemma 4 26B-A4B.

Speculative Decoding and Multi-Token Prediction

Speculative decoding and Multi-Token Prediction can make the 3090 feel much faster when the model and backend support it.

The basic idea: a smaller or cheaper draft path predicts several candidate tokens, and the main model verifies them in parallel. If enough candidates are accepted, you get more output tokens per expensive forward pass. A classic speculative sampling paper reported 2–2.5x decoding speedups without compromising output distribution in its benchmark setup.

Qwen3.5-35B-A3B’s model card includes Multi-Token Prediction examples for SGLang and vLLM, including speculative configuration. Google has also released Gemma 4 MTP drafters, describing up to 3x speedups without output-quality degradation in supported pipelines.

Gemma 4 MTP Draft Model

The HauhauCS Gemma 4 release includes an MTP draft head (mtp-gemma-4-31B-it.gguf, 267 MB) designed for speculative decoding with the main 31B model. In llama.cpp, it can be loaded alongside the target model:

llama-server -m Gemma4-31B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf \
  -md mtp-gemma-4-31B-it.gguf --spec-type draft-mtp \
  -ngl 99 -fa on

The draft model is tiny (267 MB vs 17.4 GB for the main model) and adds minimal VRAM overhead. On a single 3090, the MTP drafter shares the GPU with the already-loaded target model. Google claims up to 3x speedup in supported pipelines, and the HauhauCS release notes report roughly 60% faster generation with identical output in llama.cpp.

In practice, the benefit on a single 3090 depends on batch size, acceptance rate, and whether KV cache fits both models at your target context length. At shorter contexts (up to 16K), the speedup is noticeable; at longer contexts, VRAM pressure from the joint KV cache can eat into the gain.

MTP draft model: Unsloth Gemma 4 MTP (via HauhauCS).

For a 3090, this is worth experimenting with, but do not assume every model/backend combination will magically speed up. Speculative decoding is sensitive to acceptance rate, backend implementation, batching, context length.

When it works, it is one of the cheapest ways to get more throughput.

Power Limiting: The 250–290 W Trick

The RTX 3090 is rated for 350 W, but local inference often does not need the full power limit.

A practical power limit around 250–290 W is one of the best quality-of-life tweaks for this card. You lose some prompt-processing speed because prefill can use more compute, but token generation usually drops much less. Decode is often memory-bound, so pushing the GPU to full power does not always buy much extra output speed.

This saves power, reduces heat, reduces fan noise, and may improve long-term reliability. Community LLM testing has found useful results in the 250–300 W range, and Puget Systems found around 95% of maximum compute performance at roughly 280 W in multi-3090 compute testing.

On Linux, this is easy:

sudo nvidia-smi -pl 280

For a quieter daily local AI machine, I would start at 280 W, test your actual models, then try 260 W and 250 W. If decode speed barely changes, keep the lower limit.

The 3090 can run hot, especially on VRAM junction temperatures.

Image Generation

The RTX 3090 is also strong for image generation.

It is very capable for:

Stable Diffusion 1.5
Stable Diffusion XL
FLUX workflows
LoRA experimentation
ControlNet
Hires fix
Upscaling
Moderate ComfyUI graphs

The 24 GB VRAM matters here too. SDXL and FLUX can run on smaller cards with quantization or low-VRAM tricks, but the 3090 gives you breathing room. FLUX is resource-intensive, but 24 GB cards like the RTX 3090 are commonly treated as a practical full-quality tier for local FLUX workflows. SDXL training/inference guides also commonly place 24 GB cards like the RTX 3090 and 4090 in the serious local-workstation tier.

For video generation, it is usable but not brilliant. Wan small models are possible, especially with careful settings. Wan’s own report describes a 1.3B model with consumer-grade efficiency requiring about 8.19 GB VRAM, while the larger 14B model is the high-capability option.

For Wan large workflows, the 3090’s 24 GB is not enough to be comfortable at high resolution and long duration. You will be trading off resolution, frame count, batch size, and offload settings. Shorter, lower-resolution generations are realistic. Long, high-resolution clips are where bigger cards or multi-GPU setups become much more attractive.

Buying Advice

The RTX 3090 is a used-market card now, so condition matters more than brand.

Before buying, check:

Does the card pass a sustained VRAM stress test?
Are VRAM junction temperatures under control?
Has the thermal pad job been done properly?
Is there coil whine?
Are fans healthy?
Is the seller reputable?
Is it a blower, open-air, hybrid, or water-blocked card?
Will it physically fit your case?
Can your PSU handle transient load?
Do you need NVLink for a dual-3090 build?

For local AI, blower cards can be useful in dense multi-GPU systems, but they are loud and often hot. Open-air cards are better for a normal desktop. Founders Edition cards are compact for what they are, but memory thermals deserve attention. Some AIB cards have better cooling, but they can be enormous.

A dual-3090 build is still interesting because it gives you 48 GB total VRAM across two GPUs. Some workloads shard well; some do not. vLLM, tensor parallelism, and NVLink can help, but multi-GPU adds complexity. For a first local AI machine, one 3090 is the cleanest starting point.

Verdict

The RTX 3090 remains one of the best-value GPUs ever made for local AI.

It is not the fastest card anymore. It is not the most efficient. It does not have Blackwell FP4 support. It does not have 32 GB or 48 GB of VRAM. It is power-hungry, physically large, and old enough that used condition matters.

The core package is still excellent:

24 GB VRAM. 936 GB/s bandwidth. Mature CUDA. Strong software support. Good dense-model decode. Enough capacity for Q4 27B–31B-class models. Good image-generation performance. Used pricing that makes sense.

That combination is why the RTX 3090 refuses to die.

For local AI, NVIDIA accidentally built a monster in 2020. Five years later, it is still one of the smartest buys on the used market.