2 × NVIDIA RTX 3090 24GB
Threadripper PRO 3945WX workstation
275 W limit per GPU
Driver NVIDIA 595.84
Measured results
Benchmarks, not estimates.
Results measured on the lab workstation. For hardware, models, or settings not tested here, use the calculator—and treat its output as an estimate.
Test system
Two discrete GPUs, not unified memory.
Two discrete 24GB VRAM pools; model data is distributed by llama.cpp, not held in one unified 48GB pool.
llama.cpp
llama-bench / llama-batched-bench
Build fc6545d (build 50)
Flash Attention enabled
Per configuration
7 configurations are fully GPU-resident. 2 larger MoE configurations offload expert layers to CPU and system RAM.
Measured configurations
Model, quant, runtime, and placement.
Different quants and tuning are shown explicitly; these are practical system results, not a controlled model-quality comparison.
| Model | Quant | Batch / ubatch | Placement | Detail |
|---|---|---|---|---|
| Llama 3.3 70B | Q4_K_S | 1,024 / 512 | Fully GPU-resident | All layers on GPU; tensor split 1:1 |
| Llama 3.3 Nemotron Super 49B | Q5_K_M | 4,096 / 256 | Fully GPU-resident | All layers on GPU; layer split 0.7:1.3 |
| Qwen3.6 27B MTP | Q4_K_M | 2,048 / 2,048 | Fully GPU-resident | All layers on GPU; tensor split 1:1 |
| Qwen3.5 122B-A10B | UD-Q4_K_S | 4,096 / 256 | CPU offload | Context: automatic fit with 2,048 MiB free per GPU; concurrency: 20 MoE expert layers on CPU, remaining layers split 65:35 |
| GLM-4.5 Air 106B-A12B | i1-Q4_K_S | 2,048 / 512 | CPU offload | 18 MoE expert layers on CPU; remaining layers split 65:35 |
| Gemma 4 31B QAT MTP | Q4_K_M | 2,048 / 512 | Fully GPU-resident | All layers on GPU; tensor split 1:1 |
| Mistral Small 3.2 24B | i1-Q5_K_M | 2,048 / 512 | Fully GPU-resident | All layers on GPU; tensor split 1:1 |
| Qwen3.8 27B CRACK | Q8_0 | 2,048 / 2,048 | Fully GPU-resident | All layers on GPU; tensor split 1:1 |
| Qwen3.8 27B Uncensored OrcaRouter | Q6_K | 2,048 / 2,048 | Fully GPU-resident | All layers on GPU; tensor split 1:1 |
Context scaling
PP and TG at prefilled KV-cache depth.
Tokens per second. A dash means that depth was not measured for that configuration.
Prompt processing (PP)
| Model | 0 | 2K | 4K | 8K | 16K | 32K | 64K | 128K | 256K |
|---|---|---|---|---|---|---|---|---|---|
| Llama 3.3 70B | 786.6 | 746.5 | 721.2 | 673.7 | 598.4 | 486.0 | — | — | — |
| Llama 3.3 Nemotron Super 49B | 962.7 | 913.8 | 879.8 | 815.0 | 706.9 | 550.4 | — | — | — |
| Qwen3.6 27B MTP | 1580.0 | 1529.1 | 1508.6 | 1465.6 | 1387.4 | 1251.8 | 1046.6 | 774.1 | 498.1 |
| Qwen3.5 122B-A10B | 312.4 | 306.3 | 298.8 | 294.7 | 289.9 | 279.6 | — | — | — |
| GLM-4.5 Air 106B-A12B | 284.7 | 279.2 | 274.3 | 261.0 | 239.9 | 206.0 | — | — | — |
| Gemma 4 31B QAT MTP | 1517.6 | 1425.3 | 1392.3 | 1325.3 | 1213.2 | 1025.7 | 788.6 | 541.2 | 332.5 |
| Mistral Small 3.2 24B | 2284.1 | 2216.6 | 2153.0 | 2038.3 | 1855.1 | 1553.6 | — | — | — |
| Qwen3.8 27B CRACK | 1645.0 | 1581.5 | 1563.4 | 1507.4 | 1419.9 | 1277.6 | 1056.0 | 780.8 | 497.2 |
| Qwen3.8 27B Uncensored OrcaRouter | 1476.4 | 1430.1 | 1404.6 | 1371.8 | 1297.2 | 1175.1 | 992.2 | 748.2 | 484.9 |
Token generation (TG)
| Model | 0 | 2K | 4K | 8K | 16K | 32K | 64K | 128K | 256K |
|---|---|---|---|---|---|---|---|---|---|
| Llama 3.3 70B | 31.91 | 30.27 | 28.95 | 26.73 | 23.30 | 18.50 | — | — | — |
| Llama 3.3 Nemotron Super 49B | 22.99 | 22.16 | 21.54 | 20.32 | 18.29 | 15.15 | — | — | — |
| Qwen3.6 27B MTP | 59.05 | 58.71 | 58.11 | 56.99 | 55.04 | 51.25 | 45.08 | 35.86 | 25.41 |
| Qwen3.5 122B-A10B | 31.11 | 30.90 | 30.75 | 30.35 | 29.32 | 27.36 | — | — | — |
| GLM-4.5 Air 106B-A12B | 27.60 | 26.17 | 25.00 | 22.46 | 19.59 | 15.55 | — | — | — |
| Gemma 4 31B QAT MTP | 50.80 | 49.21 | 48.60 | 47.38 | 45.25 | 41.49 | 35.56 | 27.66 | 19.08 |
| Mistral Small 3.2 24B | 72.70 | 70.87 | 68.82 | 65.44 | 59.94 | 51.02 | — | — | — |
| Qwen3.8 27B CRACK | 45.03 | 44.71 | 44.35 | 43.65 | 42.48 | 40.30 | 36.32 | 30.35 | 22.65 |
| Qwen3.8 27B Uncensored OrcaRouter | 48.28 | 47.95 | 47.41 | 46.50 | 45.14 | 42.42 | 38.09 | 31.42 | 22.95 |
Concurrency
One, two, and four simultaneous streams.
Aggregate TG grows with concurrency while TG per stream falls. Rates are measured tokens per second.
| Model | Streams | Aggregate PP | Aggregate TG | TG / stream | Aggregate all |
|---|---|---|---|---|---|
| Llama 3.3 70B | 1 | 750.4 | 31.01 | 31.01 | 85.94 |
| Llama 3.3 70B | 2 | 752.5 | 49.84 | 24.92 | 132.03 |
| Llama 3.3 70B | 4 | 751.8 | 62.93 | 15.73 | 161.71 |
| Llama 3.3 Nemotron Super 49B | 1 | 956.8 | 22.68 | 22.68 | 64.96 |
| Llama 3.3 Nemotron Super 49B | 2 | 962.8 | 39.90 | 19.95 | 110.55 |
| Llama 3.3 Nemotron Super 49B | 4 | 841.5 | 54.79 | 13.70 | 145.44 |
| Qwen3.6 27B MTP | 1 | 1483.5 | 59.22 | 59.22 | 164.53 |
| Qwen3.6 27B MTP | 2 | 1607.4 | 98.37 | 49.19 | 262.93 |
| Qwen3.6 27B MTP | 4 | 1658.2 | 128.72 | 32.18 | 334.28 |
| Qwen3.5 122B-A10B | 1 | 255.6 | 35.29 | 35.29 | 82.96 |
| Qwen3.5 122B-A10B | 2 | 256.7 | 35.25 | 17.63 | 82.97 |
| Qwen3.5 122B-A10B | 4 | 259.9 | 49.97 | 12.49 | 108.22 |
| GLM-4.5 Air 106B-A12B | 1 | 289.6 | 27.14 | 27.14 | 68.57 |
| GLM-4.5 Air 106B-A12B | 2 | 289.9 | 18.56 | 9.28 | 49.34 |
| GLM-4.5 Air 106B-A12B | 4 | 295.8 | 27.51 | 6.88 | 69.59 |
| Gemma 4 31B QAT MTP | 1 | 1442.1 | 50.10 | 50.10 | 140.53 |
| Gemma 4 31B QAT MTP | 2 | 1446.6 | 85.73 | 42.87 | 229.94 |
| Gemma 4 31B QAT MTP | 4 | 1473.1 | 114.99 | 28.75 | 298.39 |
| Mistral Small 3.2 24B | 1 | 2170.4 | 72.29 | 72.29 | 203.31 |
| Mistral Small 3.2 24B | 2 | 2198.7 | 119.82 | 59.91 | 324.14 |
| Mistral Small 3.2 24B | 4 | 2232.6 | 165.58 | 41.39 | 432.57 |
| Qwen3.8 27B CRACK | 1 | 1504.3 | 44.66 | 44.66 | 126.47 |
| Qwen3.8 27B CRACK | 2 | 1652.0 | 79.95 | 39.97 | 218.68 |
| Qwen3.8 27B CRACK | 4 | 1715.2 | 130.86 | 32.71 | 340.60 |
| Qwen3.8 27B Uncensored OrcaRouter | 1 | 1376.9 | 48.20 | 48.20 | 135.14 |
| Qwen3.8 27B Uncensored OrcaRouter | 2 | 1495.3 | 84.98 | 42.49 | 228.92 |
| Qwen3.8 27B Uncensored OrcaRouter | 4 | 1539.4 | 121.48 | 30.37 | 314.76 |
Methodology and provenance
A short, reproducible protocol.
- PP: 512-token prompt processing.
- TG: 256-token generation.
- KV cache: Q8_0 K/V, with Flash Attention.
- Runs: built-in warmup, then 3 measured runs; arithmetic mean reported.
- Context depth: prefilled KV-cache depth before the measured work.
- Timing boundary: llama-bench excludes tokenization and sampling.
Quants, batch settings, layer placement, and model architectures vary. This corpus compares tuned configurations on one system; it is not a controlled comparison of model quality.
Full narrative, tuning notes, interpretation, and reproducibility details are in the source RTX 3090 article.
Need a hypothetical configuration? Open the calculator; its outputs are estimates, not measurements.