Qwen3.8 27B
Qwen/Qwen3.8-27B
Qwen · 22 recorded configurations
Measured setup
Hardware
Runtime and tool
Quantisation
MTP draft depth
Every option opens a recorded setup. Where the corpus recorded nothing at this page's other settings, the option names the fields it switches with it. Allocation and runtime detail stay in the technical record below.
1× RTX 3090 · ExLlamaV3
EXL3 5.00bpw · MTP 1
PP · left axisTG · right axis
View exact measurements
| Context depth (tokens) | Fresh prefill PP (tok/s) | TG (tok/s) |
|---|---|---|
| 0 | 885.22 | 43.50 |
| 16,384 | 736.91 | 39.26 |
| 113,920 | 349.80 | 32.25 |
Concurrent-stream measurements
| Streams | Context depth (tokens) | Aggregate PP (tok/s) | Aggregate TG (tok/s) | TG / stream (tok/s) | Aggregate all (tok/s) |
|---|---|---|---|---|---|
| 2 | 8,192 | 855.58 | 85.19 | 42.59 | 213.12 |
| 3 | 8,192 | 888.31 | 117.92 | 39.31 | 279.55 |
| 4 | 8,192 | 911.06 | 141.32 | 35.33 | 323.58 |
| 5 | 8,192 | 917.46 | 171.79 | 34.36 | 374.96 |
| 6 | 8,192 | 907.35 | 187.38 | 31.23 | 397.83 |
| 7 | 8,192 | 911.39 | 205.79 | 29.40 | 425.30 |
Technical record and provenance
Source 1 · 2026-10-15
- Source run ID
- trpro-exl-qwen38-27b-5bpw-gpu1-5bpw-mtp-1-context-fit-1-20261015
- Page entry ID
- trpro-exl-qwen38-27b-5bpw-gpu1-mtp-1
- Entry sweep
- context
- Raw entry label
- Qwen3.8 27B EXL3 5.00bpw · MTP-1
- Configuration label
- 1× RTX3090 — no splitting · EXL3 5.00bpw · MTP-1 · context-fit 114688-token allocation
- Measured date
- 2026-10-15
- Hardware
- 1× RTX 3090
- Runtime
- exllamav3 · ExLlamaV3 1.5.0+cu128.torch2.9.0; native extension SHA-256 c1d02b410ef729618077ba287f01f05b229f31d118a7aae04a1b7e56142e3b2e
- Weight placement
- gpu-resident
- Observed devices
- CUDA0
- Requested devices
- CUDA0
- Split mode
- Single GPU — no splitting
- Placement detail
- One physical RTX 3090; target model and native MTP compute tensors were observed on CUDA0, with no CPU expert offload or model split.
- KV-cache precision
- EXL native K8/V8 with QSA side planes and recurrent state
- Requested context per slot
- 114,688 tokens
- Actual context per slot
- 114,688 tokens
- Native model context
- 262,144 tokens
- Prompt tokens
- 512
- Generated tokens
- 256
- Speculative decoding
- MTP · draft max 1
- Warm-ups
- 1
- Measured repetitions
- 3
- Timing method
- ExLlamaV3 persistent worker time.perf_counter_ns with torch.cuda.synchronize at phase boundaries; fresh PP excludes prefix preparation; aggregate TG is across streams
- Throughput spreads
- KV 0, 1 stream: PP mean 885.22 t/s (min 879.78, max 888.90, SD 4.81, n=3); TG mean 43.50 t/s (min 43.40, max 43.61, SD 0.11, n=3); All mean 118.81 t/s (min 118.50, max 119.14, SD 0.32, n=3) · KV 16,384, 1 stream: PP mean 736.91 t/s (min 735.04, max 738.37, SD 1.70, n=3); TG mean 39.26 t/s (min 39.21, max 39.32, SD 0.05, n=3); All mean 106.45 t/s (min 106.33, max 106.60, SD 0.14, n=3) · KV 113,920, 1 stream: PP mean 349.80 t/s (min 349.35, max 350.20, SD 0.43, n=3); TG mean 32.25 t/s (min 32.23, max 32.26, SD 0.01, n=3); All mean 81.69 t/s (min 81.64, max 81.72, SD 0.04, n=3)
- Measurement outcomes
- 0 tokens · 1 stream: measured; 16,384 tokens · 1 stream: measured; 113,920 tokens · 1 stream: measured
- Notes
- 1× RTX3090 — no splitting · EXL3 5.00bpw · MTP-1 · context-fit 114688-token allocation. Persistent EXL native worker: 512 fresh prompt tokens and 256 generated tokens; one warmup then three measured repetitions. PP is fresh-prompt processing after the populated prefix; TG is aggregate decode across concurrent streams. Throughput means and sample standard deviations are recomputed from the retained raw repetitions. EXL native K8/V8 cache storage includes QSA side planes and recurrent state; this is not GGUF Q8_0. Physical GPU 1 (GPU-0a2bed40-39e4-e67f-5234-868704f9704b) was selected as the single device. Runtime evidence maps that physical GPU to one logical CUDA0. Source configuration 500bpw-context-mtp-1-c1; model revision a35e75a73baee51da709329d19294245cbeeb5d8; source manifest SHA-256 dd1f95805c018a081125a90a500a2328f41c49cb3921bb35b781b6d5c9da9a1c. Separately allocated 114688 tokens per stream for this MTP setting. The allocation search advances in 8192-token steps; its recorded fitted capacity is not a token-exact VRAM maximum. The selected boundary and the 0/16K samples are retained from their recorded source attempts.
- Source article
- /benchmarks/
- Raw log path
research/benchmarks/logs/trpro/2026-10-15-exl-context-fit/5bpw-gpu1/runs/mtp-1-114688-boundary/5.00bpw/context/mtp-1-c1
Source 2 · 2026-10-15
- Source run ID
- trpro-exl-qwen38-27b-5bpw-gpu1-5bpw-mtp-1-concurrency-2-20261015
- Page entry ID
- trpro-exl-qwen38-27b-5bpw-gpu1-mtp-1
- Entry sweep
- concurrency
- Raw entry label
- Qwen3.8 27B EXL3 5.00bpw · MTP-1
- Configuration label
- 1× RTX3090 — no splitting · EXL3 5.00bpw · MTP-1 · concurrency 2, 8192-token prefix, 8960-token allocation per stream
- Measured date
- 2026-10-15
- Hardware
- 1× RTX 3090
- Runtime
- exllamav3 · ExLlamaV3 1.5.0+cu128.torch2.9.0; native extension SHA-256 c1d02b410ef729618077ba287f01f05b229f31d118a7aae04a1b7e56142e3b2e
- Weight placement
- gpu-resident
- Observed devices
- CUDA0
- Requested devices
- CUDA0
- Split mode
- Single GPU — no splitting
- Placement detail
- One physical RTX 3090; target model and native MTP compute tensors were observed on CUDA0, with no CPU expert offload or model split.
- KV-cache precision
- EXL native K8/V8 with QSA side planes and recurrent state
- Requested context per slot
- 8,960 tokens
- Actual context per slot
- 8,960 tokens
- Native model context
- 262,144 tokens
- Prompt tokens
- 512
- Generated tokens
- 256
- Speculative decoding
- MTP · draft max 1
- Warm-ups
- 1
- Measured repetitions
- 3
- Timing method
- ExLlamaV3 persistent worker time.perf_counter_ns with torch.cuda.synchronize at phase boundaries; fresh PP excludes prefix preparation; aggregate TG is across streams
- Throughput spreads
- KV 8,192, 2 streams: PP mean 855.58 t/s (min 854.12, max 857.42, SD 1.68, n=3); TG mean 85.19 t/s (min 85.07, max 85.34, SD 0.14, n=3); All mean 213.12 t/s (min 212.95, max 213.42, SD 0.26, n=3)
- Measurement outcomes
- 8,192 tokens · 2 streams: measured
- Notes
- 1× RTX3090 — no splitting · EXL3 5.00bpw · MTP-1 · concurrency 2, 8192-token prefix, 8960-token allocation per stream. Persistent EXL native worker: 512 fresh prompt tokens and 256 generated tokens; one warmup then three measured repetitions. PP is fresh-prompt processing after the populated prefix; TG is aggregate decode across concurrent streams. Throughput means and sample standard deviations are recomputed from the retained raw repetitions. EXL native K8/V8 cache storage includes QSA side planes and recurrent state; this is not GGUF Q8_0. Physical GPU 1 (GPU-0a2bed40-39e4-e67f-5234-868704f9704b) was selected as the single device. Runtime evidence maps that physical GPU to one logical CUDA0. Source configuration 500bpw-concurrency-mtp-1-c2; model revision a35e75a73baee51da709329d19294245cbeeb5d8; source manifest SHA-256 f889b720ea21d731842579e93d41eb5e4ed995a63475a7e69874a2330b529192. The concurrency source used an 8192-token prefix and a fixed 8960-token per-stream cache allocation; this allocation is independent of context-fit capacities.
- Source article
- /benchmarks/
- Raw log path
research/benchmarks/logs/trpro/2026-10-15-independent-gpus/remaining-campaign/trpro-exl-qwen38-27b-5bpw-gpu1-independent-gpus-validation-20261015/5.00bpw/concurrency/mtp-1-c2
Source 3 · 2026-10-15
- Source run ID
- trpro-exl-qwen38-27b-5bpw-gpu1-5bpw-mtp-1-concurrency-3-20261015
- Page entry ID
- trpro-exl-qwen38-27b-5bpw-gpu1-mtp-1
- Entry sweep
- concurrency
- Raw entry label
- Qwen3.8 27B EXL3 5.00bpw · MTP-1
- Configuration label
- 1× RTX3090 — no splitting · EXL3 5.00bpw · MTP-1 · concurrency 3, 8192-token prefix, 8960-token allocation per stream
- Measured date
- 2026-10-15
- Hardware
- 1× RTX 3090
- Runtime
- exllamav3 · ExLlamaV3 1.5.0+cu128.torch2.9.0; native extension SHA-256 c1d02b410ef729618077ba287f01f05b229f31d118a7aae04a1b7e56142e3b2e
- Weight placement
- gpu-resident
- Observed devices
- CUDA0
- Requested devices
- CUDA0
- Split mode
- Single GPU — no splitting
- Placement detail
- One physical RTX 3090; target model and native MTP compute tensors were observed on CUDA0, with no CPU expert offload or model split.
- KV-cache precision
- EXL native K8/V8 with QSA side planes and recurrent state
- Requested context per slot
- 8,960 tokens
- Actual context per slot
- 8,960 tokens
- Native model context
- 262,144 tokens
- Prompt tokens
- 512
- Generated tokens
- 256
- Speculative decoding
- MTP · draft max 1
- Warm-ups
- 1
- Measured repetitions
- 3
- Timing method
- ExLlamaV3 persistent worker time.perf_counter_ns with torch.cuda.synchronize at phase boundaries; fresh PP excludes prefix preparation; aggregate TG is across streams
- Throughput spreads
- KV 8,192, 3 streams: PP mean 888.31 t/s (min 886.27, max 889.79, SD 1.82, n=3); TG mean 117.92 t/s (min 117.63, max 118.10, SD 0.25, n=3); All mean 279.55 t/s (min 278.87, max 279.97, SD 0.59, n=3)
- Measurement outcomes
- 8,192 tokens · 3 streams: measured
- Notes
- 1× RTX3090 — no splitting · EXL3 5.00bpw · MTP-1 · concurrency 3, 8192-token prefix, 8960-token allocation per stream. Persistent EXL native worker: 512 fresh prompt tokens and 256 generated tokens; one warmup then three measured repetitions. PP is fresh-prompt processing after the populated prefix; TG is aggregate decode across concurrent streams. Throughput means and sample standard deviations are recomputed from the retained raw repetitions. EXL native K8/V8 cache storage includes QSA side planes and recurrent state; this is not GGUF Q8_0. Physical GPU 1 (GPU-0a2bed40-39e4-e67f-5234-868704f9704b) was selected as the single device. Runtime evidence maps that physical GPU to one logical CUDA0. Source configuration 500bpw-concurrency-mtp-1-c3; model revision a35e75a73baee51da709329d19294245cbeeb5d8; source manifest SHA-256 f889b720ea21d731842579e93d41eb5e4ed995a63475a7e69874a2330b529192. The concurrency source used an 8192-token prefix and a fixed 8960-token per-stream cache allocation; this allocation is independent of context-fit capacities.
- Source article
- /benchmarks/
- Raw log path
research/benchmarks/logs/trpro/2026-10-15-independent-gpus/remaining-campaign/trpro-exl-qwen38-27b-5bpw-gpu1-independent-gpus-validation-20261015/5.00bpw/concurrency/mtp-1-c3
Source 4 · 2026-10-15
- Source run ID
- trpro-exl-qwen38-27b-5bpw-gpu1-5bpw-mtp-1-concurrency-4-20261015
- Page entry ID
- trpro-exl-qwen38-27b-5bpw-gpu1-mtp-1
- Entry sweep
- concurrency
- Raw entry label
- Qwen3.8 27B EXL3 5.00bpw · MTP-1
- Configuration label
- 1× RTX3090 — no splitting · EXL3 5.00bpw · MTP-1 · concurrency 4, 8192-token prefix, 8960-token allocation per stream
- Measured date
- 2026-10-15
- Hardware
- 1× RTX 3090
- Runtime
- exllamav3 · ExLlamaV3 1.5.0+cu128.torch2.9.0; native extension SHA-256 c1d02b410ef729618077ba287f01f05b229f31d118a7aae04a1b7e56142e3b2e
- Weight placement
- gpu-resident
- Observed devices
- CUDA0
- Requested devices
- CUDA0
- Split mode
- Single GPU — no splitting
- Placement detail
- One physical RTX 3090; target model and native MTP compute tensors were observed on CUDA0, with no CPU expert offload or model split.
- KV-cache precision
- EXL native K8/V8 with QSA side planes and recurrent state
- Requested context per slot
- 8,960 tokens
- Actual context per slot
- 8,960 tokens
- Native model context
- 262,144 tokens
- Prompt tokens
- 512
- Generated tokens
- 256
- Speculative decoding
- MTP · draft max 1
- Warm-ups
- 1
- Measured repetitions
- 3
- Timing method
- ExLlamaV3 persistent worker time.perf_counter_ns with torch.cuda.synchronize at phase boundaries; fresh PP excludes prefix preparation; aggregate TG is across streams
- Throughput spreads
- KV 8,192, 4 streams: PP mean 911.06 t/s (min 908.94, max 913.68, SD 2.41, n=3); TG mean 141.32 t/s (min 141.24, max 141.42, SD 0.09, n=3); All mean 323.58 t/s (min 323.27, max 323.96, SD 0.35, n=3)
- Measurement outcomes
- 8,192 tokens · 4 streams: measured
- Notes
- 1× RTX3090 — no splitting · EXL3 5.00bpw · MTP-1 · concurrency 4, 8192-token prefix, 8960-token allocation per stream. Persistent EXL native worker: 512 fresh prompt tokens and 256 generated tokens; one warmup then three measured repetitions. PP is fresh-prompt processing after the populated prefix; TG is aggregate decode across concurrent streams. Throughput means and sample standard deviations are recomputed from the retained raw repetitions. EXL native K8/V8 cache storage includes QSA side planes and recurrent state; this is not GGUF Q8_0. Physical GPU 1 (GPU-0a2bed40-39e4-e67f-5234-868704f9704b) was selected as the single device. Runtime evidence maps that physical GPU to one logical CUDA0. Source configuration 500bpw-concurrency-mtp-1-c4; model revision a35e75a73baee51da709329d19294245cbeeb5d8; source manifest SHA-256 f889b720ea21d731842579e93d41eb5e4ed995a63475a7e69874a2330b529192. The concurrency source used an 8192-token prefix and a fixed 8960-token per-stream cache allocation; this allocation is independent of context-fit capacities.
- Source article
- /benchmarks/
- Raw log path
research/benchmarks/logs/trpro/2026-10-15-independent-gpus/remaining-campaign/trpro-exl-qwen38-27b-5bpw-gpu1-independent-gpus-validation-20261015/5.00bpw/concurrency/mtp-1-c4
Source 5 · 2026-10-15
- Source run ID
- trpro-exl-qwen38-27b-5bpw-gpu1-5bpw-mtp-1-concurrency-5-20261015
- Page entry ID
- trpro-exl-qwen38-27b-5bpw-gpu1-mtp-1
- Entry sweep
- concurrency
- Raw entry label
- Qwen3.8 27B EXL3 5.00bpw · MTP-1
- Configuration label
- 1× RTX3090 — no splitting · EXL3 5.00bpw · MTP-1 · concurrency 5, 8192-token prefix, 8960-token allocation per stream
- Measured date
- 2026-10-15
- Hardware
- 1× RTX 3090
- Runtime
- exllamav3 · ExLlamaV3 1.5.0+cu128.torch2.9.0; native extension SHA-256 c1d02b410ef729618077ba287f01f05b229f31d118a7aae04a1b7e56142e3b2e
- Weight placement
- gpu-resident
- Observed devices
- CUDA0
- Requested devices
- CUDA0
- Split mode
- Single GPU — no splitting
- Placement detail
- One physical RTX 3090; target model and native MTP compute tensors were observed on CUDA0, with no CPU expert offload or model split.
- KV-cache precision
- EXL native K8/V8 with QSA side planes and recurrent state
- Requested context per slot
- 8,960 tokens
- Actual context per slot
- 8,960 tokens
- Native model context
- 262,144 tokens
- Prompt tokens
- 512
- Generated tokens
- 256
- Speculative decoding
- MTP · draft max 1
- Warm-ups
- 1
- Measured repetitions
- 3
- Timing method
- ExLlamaV3 persistent worker time.perf_counter_ns with torch.cuda.synchronize at phase boundaries; fresh PP excludes prefix preparation; aggregate TG is across streams
- Throughput spreads
- KV 8,192, 5 streams: PP mean 917.46 t/s (min 916.05, max 918.41, SD 1.25, n=3); TG mean 171.79 t/s (min 171.23, max 172.14, SD 0.49, n=3); All mean 374.96 t/s (min 373.91, max 375.62, SD 0.92, n=3)
- Measurement outcomes
- 8,192 tokens · 5 streams: measured
- Notes
- 1× RTX3090 — no splitting · EXL3 5.00bpw · MTP-1 · concurrency 5, 8192-token prefix, 8960-token allocation per stream. Persistent EXL native worker: 512 fresh prompt tokens and 256 generated tokens; one warmup then three measured repetitions. PP is fresh-prompt processing after the populated prefix; TG is aggregate decode across concurrent streams. Throughput means and sample standard deviations are recomputed from the retained raw repetitions. EXL native K8/V8 cache storage includes QSA side planes and recurrent state; this is not GGUF Q8_0. Physical GPU 1 (GPU-0a2bed40-39e4-e67f-5234-868704f9704b) was selected as the single device. Runtime evidence maps that physical GPU to one logical CUDA0. Source configuration 500bpw-concurrency-mtp-1-c5; model revision a35e75a73baee51da709329d19294245cbeeb5d8; source manifest SHA-256 f889b720ea21d731842579e93d41eb5e4ed995a63475a7e69874a2330b529192. The concurrency source used an 8192-token prefix and a fixed 8960-token per-stream cache allocation; this allocation is independent of context-fit capacities.
- Source article
- /benchmarks/
- Raw log path
research/benchmarks/logs/trpro/2026-10-15-independent-gpus/remaining-campaign/trpro-exl-qwen38-27b-5bpw-gpu1-independent-gpus-validation-20261015/5.00bpw/concurrency/mtp-1-c5
Source 6 · 2026-10-15
- Source run ID
- trpro-exl-qwen38-27b-5bpw-gpu1-5bpw-mtp-1-concurrency-6-20261015
- Page entry ID
- trpro-exl-qwen38-27b-5bpw-gpu1-mtp-1
- Entry sweep
- concurrency
- Raw entry label
- Qwen3.8 27B EXL3 5.00bpw · MTP-1
- Configuration label
- 1× RTX3090 — no splitting · EXL3 5.00bpw · MTP-1 · concurrency 6, 8192-token prefix, 8960-token allocation per stream
- Measured date
- 2026-10-15
- Hardware
- 1× RTX 3090
- Runtime
- exllamav3 · ExLlamaV3 1.5.0+cu128.torch2.9.0; native extension SHA-256 c1d02b410ef729618077ba287f01f05b229f31d118a7aae04a1b7e56142e3b2e
- Weight placement
- gpu-resident
- Observed devices
- CUDA0
- Requested devices
- CUDA0
- Split mode
- Single GPU — no splitting
- Placement detail
- One physical RTX 3090; target model and native MTP compute tensors were observed on CUDA0, with no CPU expert offload or model split.
- KV-cache precision
- EXL native K8/V8 with QSA side planes and recurrent state
- Requested context per slot
- 8,960 tokens
- Actual context per slot
- 8,960 tokens
- Native model context
- 262,144 tokens
- Prompt tokens
- 512
- Generated tokens
- 256
- Speculative decoding
- MTP · draft max 1
- Warm-ups
- 1
- Measured repetitions
- 3
- Timing method
- ExLlamaV3 persistent worker time.perf_counter_ns with torch.cuda.synchronize at phase boundaries; fresh PP excludes prefix preparation; aggregate TG is across streams
- Throughput spreads
- KV 8,192, 6 streams: PP mean 907.35 t/s (min 906.08, max 908.20, SD 1.12, n=3); TG mean 187.38 t/s (min 187.06, max 187.57, SD 0.28, n=3); All mean 397.83 t/s (min 397.19, max 398.17, SD 0.56, n=3)
- Measurement outcomes
- 8,192 tokens · 6 streams: measured
- Notes
- 1× RTX3090 — no splitting · EXL3 5.00bpw · MTP-1 · concurrency 6, 8192-token prefix, 8960-token allocation per stream. Persistent EXL native worker: 512 fresh prompt tokens and 256 generated tokens; one warmup then three measured repetitions. PP is fresh-prompt processing after the populated prefix; TG is aggregate decode across concurrent streams. Throughput means and sample standard deviations are recomputed from the retained raw repetitions. EXL native K8/V8 cache storage includes QSA side planes and recurrent state; this is not GGUF Q8_0. Physical GPU 1 (GPU-0a2bed40-39e4-e67f-5234-868704f9704b) was selected as the single device. Runtime evidence maps that physical GPU to one logical CUDA0. Source configuration 500bpw-concurrency-mtp-1-c6; model revision a35e75a73baee51da709329d19294245cbeeb5d8; source manifest SHA-256 f889b720ea21d731842579e93d41eb5e4ed995a63475a7e69874a2330b529192. The concurrency source used an 8192-token prefix and a fixed 8960-token per-stream cache allocation; this allocation is independent of context-fit capacities.
- Source article
- /benchmarks/
- Raw log path
research/benchmarks/logs/trpro/2026-10-15-independent-gpus/remaining-campaign/trpro-exl-qwen38-27b-5bpw-gpu1-independent-gpus-validation-20261015/5.00bpw/concurrency/mtp-1-c6
Source 7 · 2026-10-15
- Source run ID
- trpro-exl-qwen38-27b-5bpw-gpu1-5bpw-mtp-1-concurrency-7-20261015
- Page entry ID
- trpro-exl-qwen38-27b-5bpw-gpu1-mtp-1
- Entry sweep
- concurrency
- Raw entry label
- Qwen3.8 27B EXL3 5.00bpw · MTP-1
- Configuration label
- 1× RTX3090 — no splitting · EXL3 5.00bpw · MTP-1 · concurrency 7, 8192-token prefix, 8960-token allocation per stream
- Measured date
- 2026-10-15
- Hardware
- 1× RTX 3090
- Runtime
- exllamav3 · ExLlamaV3 1.5.0+cu128.torch2.9.0; native extension SHA-256 c1d02b410ef729618077ba287f01f05b229f31d118a7aae04a1b7e56142e3b2e
- Weight placement
- gpu-resident
- Observed devices
- CUDA0
- Requested devices
- CUDA0
- Split mode
- Single GPU — no splitting
- Placement detail
- One physical RTX 3090; target model and native MTP compute tensors were observed on CUDA0, with no CPU expert offload or model split.
- KV-cache precision
- EXL native K8/V8 with QSA side planes and recurrent state
- Requested context per slot
- 8,960 tokens
- Actual context per slot
- 8,960 tokens
- Native model context
- 262,144 tokens
- Prompt tokens
- 512
- Generated tokens
- 256
- Speculative decoding
- MTP · draft max 1
- Warm-ups
- 1
- Measured repetitions
- 3
- Timing method
- ExLlamaV3 persistent worker time.perf_counter_ns with torch.cuda.synchronize at phase boundaries; fresh PP excludes prefix preparation; aggregate TG is across streams
- Throughput spreads
- KV 8,192, 7 streams: PP mean 911.39 t/s (min 910.06, max 913.84, SD 2.12, n=3); TG mean 205.79 t/s (min 205.24, max 206.26, SD 0.51, n=3); All mean 425.30 t/s (min 424.36, max 426.32, SD 0.99, n=3)
- Measurement outcomes
- 8,192 tokens · 7 streams: measured
- Notes
- 1× RTX3090 — no splitting · EXL3 5.00bpw · MTP-1 · concurrency 7, 8192-token prefix, 8960-token allocation per stream. Persistent EXL native worker: 512 fresh prompt tokens and 256 generated tokens; one warmup then three measured repetitions. PP is fresh-prompt processing after the populated prefix; TG is aggregate decode across concurrent streams. Throughput means and sample standard deviations are recomputed from the retained raw repetitions. EXL native K8/V8 cache storage includes QSA side planes and recurrent state; this is not GGUF Q8_0. Physical GPU 1 (GPU-0a2bed40-39e4-e67f-5234-868704f9704b) was selected as the single device. Runtime evidence maps that physical GPU to one logical CUDA0. Source configuration 500bpw-concurrency-mtp-1-c7; model revision a35e75a73baee51da709329d19294245cbeeb5d8; source manifest SHA-256 f889b720ea21d731842579e93d41eb5e4ed995a63475a7e69874a2330b529192. The concurrency source used an 8192-token prefix and a fixed 8960-token per-stream cache allocation; this allocation is independent of context-fit capacities.
- Source article
- /benchmarks/
- Raw log path
research/benchmarks/logs/trpro/2026-10-15-independent-gpus/remaining-campaign/trpro-exl-qwen38-27b-5bpw-gpu1-independent-gpus-validation-20261015/5.00bpw/concurrency/mtp-1-c7
Source 8 · 2026-10-15
- Source run ID
- trpro-exl-qwen38-27b-5bpw-gpu1-5bpw-mtp-1-concurrency-8-20261015
- Page entry ID
- trpro-exl-qwen38-27b-5bpw-gpu1-mtp-1
- Entry sweep
- concurrency
- Raw entry label
- Qwen3.8 27B EXL3 5.00bpw · MTP-1
- Configuration label
- Single RTX3090 requested — placement unverified · EXL3 5.00bpw · MTP-1 · concurrency 8, 8192-token prefix, requested 8960-token allocation (actual unknown) per stream
- Measured date
- 2026-10-15
- Hardware
- Not observed (outcome-only startup failure)
- Runtime
- exllamav3 · ExLlamaV3 1.5.0+cu128.torch2.9.0; native extension SHA-256 c1d02b410ef729618077ba287f01f05b229f31d118a7aae04a1b7e56142e3b2e
- Weight placement
- Unverified (startup failed)
- Observed devices
- None observed
- Requested devices
- CUDA0
- Split mode
- None
- KV-cache precision
- EXL native K8/V8 with QSA side planes and recurrent state
- Prompt tokens
- 512
- Generated tokens
- 256
- Speculative decoding
- MTP · draft max 1
- Warm-ups
- 1
- Measured repetitions
- 3
- Timing method
- ExLlamaV3 persistent worker time.perf_counter_ns with torch.cuda.synchronize at phase boundaries; fresh PP excludes prefix preparation; aggregate TG is across streams
- Measurement outcomes
- 8,192 tokens · 8 streams: oom — RuntimeError: Insufficient VRAM in split for model and cache
- Notes
- Single RTX3090 requested — placement unverified · EXL3 5.00bpw · MTP-1 · concurrency 8, 8192-token prefix, requested 8960-token allocation (actual unknown) per stream. Persistent EXL native worker: 512 fresh prompt tokens and 256 generated tokens; one warmup then three measured repetitions. PP is fresh-prompt processing after the populated prefix; TG is aggregate decode across concurrent streams. Throughput means and sample standard deviations are recomputed from the retained raw repetitions. EXL native K8/V8 cache storage includes QSA side planes and recurrent state; this is not GGUF Q8_0. Physical GPU 1 (GPU-0a2bed40-39e4-e67f-5234-868704f9704b) was selected as the single device. No successful model placement was observed for this outcome-only source. Source configuration 500bpw-concurrency-mtp-1-c8; model revision a35e75a73baee51da709329d19294245cbeeb5d8; source manifest SHA-256 f889b720ea21d731842579e93d41eb5e4ed995a63475a7e69874a2330b529192. The concurrency source used an 8192-token prefix and a fixed 8960-token per-stream cache allocation (actual allocation unknown); this allocation is independent of context-fit capacities.
- Source article
- /benchmarks/
- Raw log path
research/benchmarks/logs/trpro/2026-10-15-independent-gpus/remaining-campaign/trpro-exl-qwen38-27b-5bpw-gpu1-independent-gpus-validation-20261015/5.00bpw/concurrency/mtp-1-c8