2 × RTX 3090
This workstation has two RTX 3090 cards. Individual runs use one or both; each entry identifies its observed GPU count and placement. Outcome-only startup failures remain visible with device placement marked unverified.
Real Benchmarks.
Results measured on the lab workstation. For hardware, models, or settings not tested here, use the calculator—and treat its output as an estimate.
The benchmark system.
llama.cpp / ExLlamaV3
Installed builds are recorded on every configuration. Historical sweeps use llama-bench and llama-batched-bench; newer runs use live llama-server or the native ExLlamaV3 worker.
24 GB per card · Independent memory
Successful runs use one or both cards; each keeps its own memory pool, with no shared 48GB pool. Per-run device count, placement, and allocation are recorded in the entry details.
Measured models.
Displayed rates are measured means; repeat-level spread and sample counts are shown when available. Select a model to open its model page: device and placement, runtime and tool, quantisation, and MTP draft depth, each resolving to a recorded configuration with its exact context-depth table, PP/TG chart, concurrency results, and provenance. Filters and search cover every model's default MTP setup, which is what a card counts. Different tools and protocols are not controlled comparisons. A dash means not measured; failed outcomes remain in the technical record, not as zero-throughput points.
18 models · 45 of 63 entries shown
No entry matches these filters
Scaling
Compare entries.
These comparison charts follow the filters above. Pick models, then the setups of each model you want to overlay; each model page contains its exact setup values and context graphs. Lines break where a setup has no measurement at a plotted depth.
Compare setups
Qwen3.6 35B-A3B
1 of 9 setups shown
Gemma 4 E2B
1 of 4 setups shown
Gemma 4 26B-A4B
1 of 4 setups shown
Start with one setup per model. Select additional setups above to compare quantisation, GPU count or MTP.
Throughput against context depth
- Qwen3.6 35B-A3B · MTP 3UD-IQ4_XS · 1× RTX 3090 · llama.cpp
- Gemma 4 E2B · MTP 3UD-Q8_K_XL · 1× RTX 3090 · llama.cpp
- Gemma 4 26B-A4B · MTP 3UD-Q4_K_XL · 1× RTX 3090 · llama.cpp
Token generation · tok/s. Measured context depths are equally spaced. Gaps mean no measurement at that depth; hover for exact token counts and rates.
Throughput against concurrent streams
- Qwen3.6 35B-A3B · MTP 3UD-IQ4_XS · 1× RTX 3090 · llama.cpp
- Gemma 4 E2B · MTP 3UD-Q8_K_XL · 1× RTX 3090 · llama.cpp
- Gemma 4 26B-A4B · MTP 3UD-Q4_K_XL · 1× RTX 3090 · llama.cpp
8K tokens of existing prefix per stream. Aggregate is total throughput; per-stream is the rate for one conversation.
Methodology and provenance
A short, reproducible protocol.
- Measured work: the historical baseline uses 512 PP tokens and 256 TG tokens after prefilling the cache; test lengths are recorded per entry.
- KV cache: the historical llama.cpp baseline uses Q8_0 K/V, with Flash Attention. EXL uses native K8/V8 with QSA side planes and recurrent state, not llama.cpp Q8_0. Exact settings are recorded per entry.
- Runs: warm-up policy, measured repetitions, aggregation, and timing method are recorded per entry. Historical baseline tables report arithmetic means of 3 runs.
- Context depth: the context coordinate follows each run's recorded protocol (existing KV-cache depth or live prompt depth), not necessarily total context capacity. The 256K column folds historical published protocol depths of 261,376, 261,888, and 262,144 tokens. Other measured maximum prefixes, including fitted single-GPU EXL capacities, retain their exact coordinates; allocation sizes are recorded separately.
- MTP depth: native MTP is measured through live llama-server or the native EXL worker, and every draft depth is its own recorded configuration. A model page lists the depths measured for the same device, tool, and quantisation, and states a setup fact in the option label whenever that draft depth also changed the recorded allocation or expert placement. Depths with different profiles are not a controlled comparison of MTP. The default mode is the fastest single-stream TG at the lowest context depth shared by successful modes; outcome-only modes are never selected as the default.
- Timing boundary: historical llama-bench excludes tokenization and sampling. Live-server timings follow the protocol recorded for that run; do not treat different protocols as identical experiments.
- Bulk versus fresh PP: EXL bulk prefix rates are the mean of depth / prefix-build time across three measured repetitions. They are not the fresh PP512 rate. Prefix rebuilding, model loading, and cache reset are outside fresh PP/TG timings; every depth has one excluded warm-up. Corpus content changes with depth, so this is not an isolated context-length microbenchmark.
- Concurrency: single-stream context sweeps and concurrent-stream sweeps are separate experiments. Each entry retains its own allocation and context coordinates; aggregate rates sum every stream, while TG / stream is the rate each conversation sees. Aggregate all counts prompt and decode tokens per second of wall time and is not comparable across protocols.
Quants, installed builds, batch settings, layer placement, and model architectures vary. These are practical system measurements, not a controlled comparison of model quality.
Only measured modes and depths are published. The 0 column uses no existing prefix, except where a configuration states its warm state explicitly. Explicit runtime metadata overrides are labelled separately. OOM, unsupported context, and runtime failures are retained in the raw artifacts, not published as zero throughput. Coherent generated text is a smoke check, not evidence of model-quality equivalence.
Historical narrative, tuning notes, and interpretation are in the source RTX 3090 article.
Need a hypothetical configuration? Open the calculator; its outputs are estimates, not measurements.