Qwen3.8 27B
Qwen/Qwen3.8-27B
Qwen · 22 recorded configurations
Measured setup
Hardware
Runtime and tool
Quantisation
MTP draft depth
Every option opens a recorded setup. Where the corpus recorded nothing at this page's other settings, the option names the fields it switches with it. Allocation and runtime detail stay in the technical record below.
1× RTX 3090 · ExLlamaV3
EXL3 4.00bpw · MTP 3
PP · left axisTG · right axis
View exact measurements
| Context depth (tokens) | Fresh prefill PP (tok/s) | TG (tok/s) |
|---|---|---|
| 0 | 916.91 | 72.30 |
| 16,384 | 754.17 | 51.22 |
| 171,264 | 279.08 | 35.57 |
Concurrent-stream measurements
| Streams | Context depth (tokens) | Aggregate PP (tok/s) | Aggregate TG (tok/s) | TG / stream (tok/s) | Aggregate all (tok/s) |
|---|---|---|---|---|---|
| 2 | 8,192 | 894.30 | 120.78 | 60.39 | 285.28 |
| 3 | 8,192 | 927.15 | 157.34 | 52.45 | 352.41 |
| 4 | 8,192 | 940.48 | 189.42 | 47.36 | 405.09 |
| 5 | 8,192 | 939.40 | 160.40 | 32.08 | 358.71 |
| 6 | 8,192 | 925.72 | 169.82 | 28.30 | 372.72 |
| 7 | 8,192 | 927.52 | 179.46 | 25.64 | 388.18 |
Technical record and provenance
Source 1 · 2026-10-15
- Source run ID
- trpro-exl-qwen38-27b-4bpw-gpu0-4bpw-mtp-3-context-fit-1-20261015
- Page entry ID
- trpro-exl-qwen38-27b-4bpw-gpu0-mtp-3
- Entry sweep
- context
- Raw entry label
- Qwen3.8 27B EXL3 4.00bpw · MTP-3
- Configuration label
- 1× RTX3090 — no splitting · EXL3 4.00bpw · MTP-3 · context-fit 172032-token allocation
- Measured date
- 2026-10-15
- Hardware
- 1× RTX 3090
- Runtime
- exllamav3 · ExLlamaV3 1.5.0+cu128.torch2.9.0; native extension SHA-256 c1d02b410ef729618077ba287f01f05b229f31d118a7aae04a1b7e56142e3b2e
- Weight placement
- gpu-resident
- Observed devices
- CUDA0
- Requested devices
- CUDA0
- Split mode
- Single GPU — no splitting
- Placement detail
- One physical RTX 3090; target model and native MTP compute tensors were observed on CUDA0, with no CPU expert offload or model split.
- KV-cache precision
- EXL native K8/V8 with QSA side planes and recurrent state
- Requested context per slot
- 172,032 tokens
- Actual context per slot
- 172,032 tokens
- Native model context
- 262,144 tokens
- Prompt tokens
- 512
- Generated tokens
- 256
- Speculative decoding
- MTP · draft max 3
- Warm-ups
- 1
- Measured repetitions
- 3
- Timing method
- ExLlamaV3 persistent worker time.perf_counter_ns with torch.cuda.synchronize at phase boundaries; fresh PP excludes prefix preparation; aggregate TG is across streams
- Throughput spreads
- KV 0, 1 stream: PP mean 916.91 t/s (min 870.52, max 941.21, SD 40.19, n=3); TG mean 72.30 t/s (min 72.04, max 72.45, SD 0.23, n=3); All mean 187.34 t/s (min 185.43, max 188.29, SD 1.65, n=3) · KV 16,384, 1 stream: PP mean 754.17 t/s (min 752.13, max 756.85, SD 2.42, n=3); TG mean 51.22 t/s (min 51.19, max 51.25, SD 0.03, n=3); All mean 135.29 t/s (min 135.16, max 135.42, SD 0.13, n=3) · KV 171,264, 1 stream: PP mean 279.08 t/s (min 278.40, max 279.44, SD 0.59, n=3); TG mean 35.57 t/s (min 35.51, max 35.62, SD 0.05, n=3); All mean 85.03 t/s (min 84.88, max 85.14, SD 0.13, n=3)
- Measurement outcomes
- 0 tokens · 1 stream: measured; 16,384 tokens · 1 stream: measured; 171,264 tokens · 1 stream: measured
- Notes
- 1× RTX3090 — no splitting · EXL3 4.00bpw · MTP-3 · context-fit 172032-token allocation. Persistent EXL native worker: 512 fresh prompt tokens and 256 generated tokens; one warmup then three measured repetitions. PP is fresh-prompt processing after the populated prefix; TG is aggregate decode across concurrent streams. Throughput means and sample standard deviations are recomputed from the retained raw repetitions. EXL native K8/V8 cache storage includes QSA side planes and recurrent state; this is not GGUF Q8_0. Physical GPU 0 (GPU-7aa53e24-dc9d-bf8d-96cc-ef8c05c66dab) was selected as the single device. Runtime evidence maps that physical GPU to one logical CUDA0. Source configuration 400bpw-context-mtp-3-c1; model revision 113cf7ab958054860e43fb7f3063b1af19171095; source manifest SHA-256 99427ccc7780c9c510c2e94cba8aa9d051ea431804c95b3f38a162629505099b. Separately allocated 172032 tokens per stream for this MTP setting. The allocation search advances in 8192-token steps; its recorded fitted capacity is not a token-exact VRAM maximum. The selected boundary and the 0/16K samples are retained from their recorded source attempts.
- Source article
- /benchmarks/
- Raw log path
research/benchmarks/logs/trpro/2026-10-15-exl-context-fit/4bpw-gpu0/runs/mtp-3-172032-boundary/4.00bpw/context/mtp-3-c1
Source 2 · 2026-10-15
- Source run ID
- trpro-exl-qwen38-27b-4bpw-gpu0-4bpw-mtp-3-concurrency-2-20261015
- Page entry ID
- trpro-exl-qwen38-27b-4bpw-gpu0-mtp-3
- Entry sweep
- concurrency
- Raw entry label
- Qwen3.8 27B EXL3 4.00bpw · MTP-3
- Configuration label
- 1× RTX3090 — no splitting · EXL3 4.00bpw · MTP-3 · concurrency 2, 8192-token prefix, 8960-token allocation per stream
- Measured date
- 2026-10-15
- Hardware
- 1× RTX 3090
- Runtime
- exllamav3 · ExLlamaV3 1.5.0+cu128.torch2.9.0; native extension SHA-256 c1d02b410ef729618077ba287f01f05b229f31d118a7aae04a1b7e56142e3b2e
- Weight placement
- gpu-resident
- Observed devices
- CUDA0
- Requested devices
- CUDA0
- Split mode
- Single GPU — no splitting
- Placement detail
- One physical RTX 3090; target model and native MTP compute tensors were observed on CUDA0, with no CPU expert offload or model split.
- KV-cache precision
- EXL native K8/V8 with QSA side planes and recurrent state
- Requested context per slot
- 8,960 tokens
- Actual context per slot
- 8,960 tokens
- Native model context
- 262,144 tokens
- Prompt tokens
- 512
- Generated tokens
- 256
- Speculative decoding
- MTP · draft max 3
- Warm-ups
- 1
- Measured repetitions
- 3
- Timing method
- ExLlamaV3 persistent worker time.perf_counter_ns with torch.cuda.synchronize at phase boundaries; fresh PP excludes prefix preparation; aggregate TG is across streams
- Throughput spreads
- KV 8,192, 2 streams: PP mean 894.30 t/s (min 893.61, max 895.22, SD 0.83, n=3); TG mean 120.78 t/s (min 120.49, max 120.94, SD 0.25, n=3); All mean 285.28 t/s (min 284.80, max 285.53, SD 0.42, n=3)
- Measurement outcomes
- 8,192 tokens · 2 streams: measured
- Notes
- 1× RTX3090 — no splitting · EXL3 4.00bpw · MTP-3 · concurrency 2, 8192-token prefix, 8960-token allocation per stream. Persistent EXL native worker: 512 fresh prompt tokens and 256 generated tokens; one warmup then three measured repetitions. PP is fresh-prompt processing after the populated prefix; TG is aggregate decode across concurrent streams. Throughput means and sample standard deviations are recomputed from the retained raw repetitions. EXL native K8/V8 cache storage includes QSA side planes and recurrent state; this is not GGUF Q8_0. Physical GPU 0 (GPU-7aa53e24-dc9d-bf8d-96cc-ef8c05c66dab) was selected as the single device. Runtime evidence maps that physical GPU to one logical CUDA0. Source configuration 400bpw-concurrency-mtp-3-c2; model revision 113cf7ab958054860e43fb7f3063b1af19171095; source manifest SHA-256 f4213ce466cd62992d252fcd14c623cb23cee191ed865703639e884f2df15d01. The concurrency source used an 8192-token prefix and a fixed 8960-token per-stream cache allocation; this allocation is independent of context-fit capacities.
- Source article
- /benchmarks/
- Raw log path
research/benchmarks/logs/trpro/2026-10-15-independent-gpus/remaining-campaign/trpro-exl-qwen38-27b-4bpw-gpu0-independent-gpus-validation-20261015/4.00bpw/concurrency/mtp-3-c2
Source 3 · 2026-10-15
- Source run ID
- trpro-exl-qwen38-27b-4bpw-gpu0-4bpw-mtp-3-concurrency-3-20261015
- Page entry ID
- trpro-exl-qwen38-27b-4bpw-gpu0-mtp-3
- Entry sweep
- concurrency
- Raw entry label
- Qwen3.8 27B EXL3 4.00bpw · MTP-3
- Configuration label
- 1× RTX3090 — no splitting · EXL3 4.00bpw · MTP-3 · concurrency 3, 8192-token prefix, 8960-token allocation per stream
- Measured date
- 2026-10-15
- Hardware
- 1× RTX 3090
- Runtime
- exllamav3 · ExLlamaV3 1.5.0+cu128.torch2.9.0; native extension SHA-256 c1d02b410ef729618077ba287f01f05b229f31d118a7aae04a1b7e56142e3b2e
- Weight placement
- gpu-resident
- Observed devices
- CUDA0
- Requested devices
- CUDA0
- Split mode
- Single GPU — no splitting
- Placement detail
- One physical RTX 3090; target model and native MTP compute tensors were observed on CUDA0, with no CPU expert offload or model split.
- KV-cache precision
- EXL native K8/V8 with QSA side planes and recurrent state
- Requested context per slot
- 8,960 tokens
- Actual context per slot
- 8,960 tokens
- Native model context
- 262,144 tokens
- Prompt tokens
- 512
- Generated tokens
- 256
- Speculative decoding
- MTP · draft max 3
- Warm-ups
- 1
- Measured repetitions
- 3
- Timing method
- ExLlamaV3 persistent worker time.perf_counter_ns with torch.cuda.synchronize at phase boundaries; fresh PP excludes prefix preparation; aggregate TG is across streams
- Throughput spreads
- KV 8,192, 3 streams: PP mean 927.15 t/s (min 925.87, max 928.04, SD 1.14, n=3); TG mean 157.34 t/s (min 157.18, max 157.59, SD 0.22, n=3); All mean 352.41 t/s (min 352.02, max 352.91, SD 0.46, n=3)
- Measurement outcomes
- 8,192 tokens · 3 streams: measured
- Notes
- 1× RTX3090 — no splitting · EXL3 4.00bpw · MTP-3 · concurrency 3, 8192-token prefix, 8960-token allocation per stream. Persistent EXL native worker: 512 fresh prompt tokens and 256 generated tokens; one warmup then three measured repetitions. PP is fresh-prompt processing after the populated prefix; TG is aggregate decode across concurrent streams. Throughput means and sample standard deviations are recomputed from the retained raw repetitions. EXL native K8/V8 cache storage includes QSA side planes and recurrent state; this is not GGUF Q8_0. Physical GPU 0 (GPU-7aa53e24-dc9d-bf8d-96cc-ef8c05c66dab) was selected as the single device. Runtime evidence maps that physical GPU to one logical CUDA0. Source configuration 400bpw-concurrency-mtp-3-c3; model revision 113cf7ab958054860e43fb7f3063b1af19171095; source manifest SHA-256 f4213ce466cd62992d252fcd14c623cb23cee191ed865703639e884f2df15d01. The concurrency source used an 8192-token prefix and a fixed 8960-token per-stream cache allocation; this allocation is independent of context-fit capacities.
- Source article
- /benchmarks/
- Raw log path
research/benchmarks/logs/trpro/2026-10-15-independent-gpus/remaining-campaign/trpro-exl-qwen38-27b-4bpw-gpu0-independent-gpus-validation-20261015/4.00bpw/concurrency/mtp-3-c3
Source 4 · 2026-10-15
- Source run ID
- trpro-exl-qwen38-27b-4bpw-gpu0-4bpw-mtp-3-concurrency-4-20261015
- Page entry ID
- trpro-exl-qwen38-27b-4bpw-gpu0-mtp-3
- Entry sweep
- concurrency
- Raw entry label
- Qwen3.8 27B EXL3 4.00bpw · MTP-3
- Configuration label
- 1× RTX3090 — no splitting · EXL3 4.00bpw · MTP-3 · concurrency 4, 8192-token prefix, 8960-token allocation per stream
- Measured date
- 2026-10-15
- Hardware
- 1× RTX 3090
- Runtime
- exllamav3 · ExLlamaV3 1.5.0+cu128.torch2.9.0; native extension SHA-256 c1d02b410ef729618077ba287f01f05b229f31d118a7aae04a1b7e56142e3b2e
- Weight placement
- gpu-resident
- Observed devices
- CUDA0
- Requested devices
- CUDA0
- Split mode
- Single GPU — no splitting
- Placement detail
- One physical RTX 3090; target model and native MTP compute tensors were observed on CUDA0, with no CPU expert offload or model split.
- KV-cache precision
- EXL native K8/V8 with QSA side planes and recurrent state
- Requested context per slot
- 8,960 tokens
- Actual context per slot
- 8,960 tokens
- Native model context
- 262,144 tokens
- Prompt tokens
- 512
- Generated tokens
- 256
- Speculative decoding
- MTP · draft max 3
- Warm-ups
- 1
- Measured repetitions
- 3
- Timing method
- ExLlamaV3 persistent worker time.perf_counter_ns with torch.cuda.synchronize at phase boundaries; fresh PP excludes prefix preparation; aggregate TG is across streams
- Throughput spreads
- KV 8,192, 4 streams: PP mean 940.48 t/s (min 938.97, max 941.83, SD 1.44, n=3); TG mean 189.42 t/s (min 189.22, max 189.58, SD 0.18, n=3); All mean 405.09 t/s (min 404.60, max 405.34, SD 0.42, n=3)
- Measurement outcomes
- 8,192 tokens · 4 streams: measured
- Notes
- 1× RTX3090 — no splitting · EXL3 4.00bpw · MTP-3 · concurrency 4, 8192-token prefix, 8960-token allocation per stream. Persistent EXL native worker: 512 fresh prompt tokens and 256 generated tokens; one warmup then three measured repetitions. PP is fresh-prompt processing after the populated prefix; TG is aggregate decode across concurrent streams. Throughput means and sample standard deviations are recomputed from the retained raw repetitions. EXL native K8/V8 cache storage includes QSA side planes and recurrent state; this is not GGUF Q8_0. Physical GPU 0 (GPU-7aa53e24-dc9d-bf8d-96cc-ef8c05c66dab) was selected as the single device. Runtime evidence maps that physical GPU to one logical CUDA0. Source configuration 400bpw-concurrency-mtp-3-c4; model revision 113cf7ab958054860e43fb7f3063b1af19171095; source manifest SHA-256 f4213ce466cd62992d252fcd14c623cb23cee191ed865703639e884f2df15d01. The concurrency source used an 8192-token prefix and a fixed 8960-token per-stream cache allocation; this allocation is independent of context-fit capacities.
- Source article
- /benchmarks/
- Raw log path
research/benchmarks/logs/trpro/2026-10-15-independent-gpus/remaining-campaign/trpro-exl-qwen38-27b-4bpw-gpu0-independent-gpus-validation-20261015/4.00bpw/concurrency/mtp-3-c4
Source 5 · 2026-10-15
- Source run ID
- trpro-exl-qwen38-27b-4bpw-gpu0-4bpw-mtp-3-concurrency-5-20261015
- Page entry ID
- trpro-exl-qwen38-27b-4bpw-gpu0-mtp-3
- Entry sweep
- concurrency
- Raw entry label
- Qwen3.8 27B EXL3 4.00bpw · MTP-3
- Configuration label
- 1× RTX3090 — no splitting · EXL3 4.00bpw · MTP-3 · concurrency 5, 8192-token prefix, 8960-token allocation per stream
- Measured date
- 2026-10-15
- Hardware
- 1× RTX 3090
- Runtime
- exllamav3 · ExLlamaV3 1.5.0+cu128.torch2.9.0; native extension SHA-256 c1d02b410ef729618077ba287f01f05b229f31d118a7aae04a1b7e56142e3b2e
- Weight placement
- gpu-resident
- Observed devices
- CUDA0
- Requested devices
- CUDA0
- Split mode
- Single GPU — no splitting
- Placement detail
- One physical RTX 3090; target model and native MTP compute tensors were observed on CUDA0, with no CPU expert offload or model split.
- KV-cache precision
- EXL native K8/V8 with QSA side planes and recurrent state
- Requested context per slot
- 8,960 tokens
- Actual context per slot
- 8,960 tokens
- Native model context
- 262,144 tokens
- Prompt tokens
- 512
- Generated tokens
- 256
- Speculative decoding
- MTP · draft max 3
- Warm-ups
- 1
- Measured repetitions
- 3
- Timing method
- ExLlamaV3 persistent worker time.perf_counter_ns with torch.cuda.synchronize at phase boundaries; fresh PP excludes prefix preparation; aggregate TG is across streams
- Throughput spreads
- KV 8,192, 5 streams: PP mean 939.40 t/s (min 939.21, max 939.50, SD 0.16, n=3); TG mean 160.40 t/s (min 160.31, max 160.48, SD 0.09, n=3); All mean 358.71 t/s (min 358.56, max 358.82, SD 0.14, n=3)
- Measurement outcomes
- 8,192 tokens · 5 streams: measured
- Notes
- 1× RTX3090 — no splitting · EXL3 4.00bpw · MTP-3 · concurrency 5, 8192-token prefix, 8960-token allocation per stream. Persistent EXL native worker: 512 fresh prompt tokens and 256 generated tokens; one warmup then three measured repetitions. PP is fresh-prompt processing after the populated prefix; TG is aggregate decode across concurrent streams. Throughput means and sample standard deviations are recomputed from the retained raw repetitions. EXL native K8/V8 cache storage includes QSA side planes and recurrent state; this is not GGUF Q8_0. Physical GPU 0 (GPU-7aa53e24-dc9d-bf8d-96cc-ef8c05c66dab) was selected as the single device. Runtime evidence maps that physical GPU to one logical CUDA0. Source configuration 400bpw-concurrency-mtp-3-c5; model revision 113cf7ab958054860e43fb7f3063b1af19171095; source manifest SHA-256 f4213ce466cd62992d252fcd14c623cb23cee191ed865703639e884f2df15d01. The concurrency source used an 8192-token prefix and a fixed 8960-token per-stream cache allocation; this allocation is independent of context-fit capacities.
- Source article
- /benchmarks/
- Raw log path
research/benchmarks/logs/trpro/2026-10-15-independent-gpus/remaining-campaign/trpro-exl-qwen38-27b-4bpw-gpu0-independent-gpus-validation-20261015/4.00bpw/concurrency/mtp-3-c5
Source 6 · 2026-10-15
- Source run ID
- trpro-exl-qwen38-27b-4bpw-gpu0-4bpw-mtp-3-concurrency-6-20261015
- Page entry ID
- trpro-exl-qwen38-27b-4bpw-gpu0-mtp-3
- Entry sweep
- concurrency
- Raw entry label
- Qwen3.8 27B EXL3 4.00bpw · MTP-3
- Configuration label
- 1× RTX3090 — no splitting · EXL3 4.00bpw · MTP-3 · concurrency 6, 8192-token prefix, 8960-token allocation per stream
- Measured date
- 2026-10-15
- Hardware
- 1× RTX 3090
- Runtime
- exllamav3 · ExLlamaV3 1.5.0+cu128.torch2.9.0; native extension SHA-256 c1d02b410ef729618077ba287f01f05b229f31d118a7aae04a1b7e56142e3b2e
- Weight placement
- gpu-resident
- Observed devices
- CUDA0
- Requested devices
- CUDA0
- Split mode
- Single GPU — no splitting
- Placement detail
- One physical RTX 3090; target model and native MTP compute tensors were observed on CUDA0, with no CPU expert offload or model split.
- KV-cache precision
- EXL native K8/V8 with QSA side planes and recurrent state
- Requested context per slot
- 8,960 tokens
- Actual context per slot
- 8,960 tokens
- Native model context
- 262,144 tokens
- Prompt tokens
- 512
- Generated tokens
- 256
- Speculative decoding
- MTP · draft max 3
- Warm-ups
- 1
- Measured repetitions
- 3
- Timing method
- ExLlamaV3 persistent worker time.perf_counter_ns with torch.cuda.synchronize at phase boundaries; fresh PP excludes prefix preparation; aggregate TG is across streams
- Throughput spreads
- KV 8,192, 6 streams: PP mean 925.72 t/s (min 923.65, max 927.27, SD 1.87, n=3); TG mean 169.82 t/s (min 169.80, max 169.86, SD 0.03, n=3); All mean 372.72 t/s (min 372.56, max 372.86, SD 0.15, n=3)
- Measurement outcomes
- 8,192 tokens · 6 streams: measured
- Notes
- 1× RTX3090 — no splitting · EXL3 4.00bpw · MTP-3 · concurrency 6, 8192-token prefix, 8960-token allocation per stream. Persistent EXL native worker: 512 fresh prompt tokens and 256 generated tokens; one warmup then three measured repetitions. PP is fresh-prompt processing after the populated prefix; TG is aggregate decode across concurrent streams. Throughput means and sample standard deviations are recomputed from the retained raw repetitions. EXL native K8/V8 cache storage includes QSA side planes and recurrent state; this is not GGUF Q8_0. Physical GPU 0 (GPU-7aa53e24-dc9d-bf8d-96cc-ef8c05c66dab) was selected as the single device. Runtime evidence maps that physical GPU to one logical CUDA0. Source configuration 400bpw-concurrency-mtp-3-c6; model revision 113cf7ab958054860e43fb7f3063b1af19171095; source manifest SHA-256 f4213ce466cd62992d252fcd14c623cb23cee191ed865703639e884f2df15d01. The concurrency source used an 8192-token prefix and a fixed 8960-token per-stream cache allocation; this allocation is independent of context-fit capacities.
- Source article
- /benchmarks/
- Raw log path
research/benchmarks/logs/trpro/2026-10-15-independent-gpus/remaining-campaign/trpro-exl-qwen38-27b-4bpw-gpu0-independent-gpus-validation-20261015/4.00bpw/concurrency/mtp-3-c6
Source 7 · 2026-10-15
- Source run ID
- trpro-exl-qwen38-27b-4bpw-gpu0-4bpw-mtp-3-concurrency-7-20261015
- Page entry ID
- trpro-exl-qwen38-27b-4bpw-gpu0-mtp-3
- Entry sweep
- concurrency
- Raw entry label
- Qwen3.8 27B EXL3 4.00bpw · MTP-3
- Configuration label
- 1× RTX3090 — no splitting · EXL3 4.00bpw · MTP-3 · concurrency 7, 8192-token prefix, 8960-token allocation per stream
- Measured date
- 2026-10-15
- Hardware
- 1× RTX 3090
- Runtime
- exllamav3 · ExLlamaV3 1.5.0+cu128.torch2.9.0; native extension SHA-256 c1d02b410ef729618077ba287f01f05b229f31d118a7aae04a1b7e56142e3b2e
- Weight placement
- gpu-resident
- Observed devices
- CUDA0
- Requested devices
- CUDA0
- Split mode
- Single GPU — no splitting
- Placement detail
- One physical RTX 3090; target model and native MTP compute tensors were observed on CUDA0, with no CPU expert offload or model split.
- KV-cache precision
- EXL native K8/V8 with QSA side planes and recurrent state
- Requested context per slot
- 8,960 tokens
- Actual context per slot
- 8,960 tokens
- Native model context
- 262,144 tokens
- Prompt tokens
- 512
- Generated tokens
- 256
- Speculative decoding
- MTP · draft max 3
- Warm-ups
- 1
- Measured repetitions
- 3
- Timing method
- ExLlamaV3 persistent worker time.perf_counter_ns with torch.cuda.synchronize at phase boundaries; fresh PP excludes prefix preparation; aggregate TG is across streams
- Throughput spreads
- KV 8,192, 7 streams: PP mean 927.52 t/s (min 927.39, max 927.65, SD 0.13, n=3); TG mean 179.46 t/s (min 179.41, max 179.53, SD 0.06, n=3); All mean 388.18 t/s (min 388.10, max 388.27, SD 0.09, n=3)
- Measurement outcomes
- 8,192 tokens · 7 streams: measured
- Notes
- 1× RTX3090 — no splitting · EXL3 4.00bpw · MTP-3 · concurrency 7, 8192-token prefix, 8960-token allocation per stream. Persistent EXL native worker: 512 fresh prompt tokens and 256 generated tokens; one warmup then three measured repetitions. PP is fresh-prompt processing after the populated prefix; TG is aggregate decode across concurrent streams. Throughput means and sample standard deviations are recomputed from the retained raw repetitions. EXL native K8/V8 cache storage includes QSA side planes and recurrent state; this is not GGUF Q8_0. Physical GPU 0 (GPU-7aa53e24-dc9d-bf8d-96cc-ef8c05c66dab) was selected as the single device. Runtime evidence maps that physical GPU to one logical CUDA0. Source configuration 400bpw-concurrency-mtp-3-c7; model revision 113cf7ab958054860e43fb7f3063b1af19171095; source manifest SHA-256 f4213ce466cd62992d252fcd14c623cb23cee191ed865703639e884f2df15d01. The concurrency source used an 8192-token prefix and a fixed 8960-token per-stream cache allocation; this allocation is independent of context-fit capacities.
- Source article
- /benchmarks/
- Raw log path
research/benchmarks/logs/trpro/2026-10-15-independent-gpus/remaining-campaign/trpro-exl-qwen38-27b-4bpw-gpu0-independent-gpus-validation-20261015/4.00bpw/concurrency/mtp-3-c7
Source 8 · 2026-10-15
- Source run ID
- trpro-exl-qwen38-27b-4bpw-gpu0-4bpw-mtp-3-concurrency-8-20261015
- Page entry ID
- trpro-exl-qwen38-27b-4bpw-gpu0-mtp-3
- Entry sweep
- concurrency
- Raw entry label
- Qwen3.8 27B EXL3 4.00bpw · MTP-3
- Configuration label
- 1× RTX3090 — no splitting · EXL3 4.00bpw · MTP-3 · concurrency 8, 8192-token prefix, 8960-token allocation per stream
- Measured date
- 2026-10-15
- Hardware
- 1× RTX 3090
- Runtime
- exllamav3 · ExLlamaV3 1.5.0+cu128.torch2.9.0; native extension SHA-256 c1d02b410ef729618077ba287f01f05b229f31d118a7aae04a1b7e56142e3b2e
- Weight placement
- gpu-resident
- Observed devices
- CUDA0
- Requested devices
- CUDA0
- Split mode
- Single GPU — no splitting
- Placement detail
- One physical RTX 3090; target model and native MTP compute tensors were observed on CUDA0, with no CPU expert offload or model split.
- KV-cache precision
- EXL native K8/V8 with QSA side planes and recurrent state
- Requested context per slot
- 8,960 tokens
- Actual context per slot
- 8,960 tokens
- Native model context
- 262,144 tokens
- Prompt tokens
- 512
- Generated tokens
- 256
- Speculative decoding
- MTP · draft max 3
- Warm-ups
- 1
- Measured repetitions
- 3
- Timing method
- ExLlamaV3 persistent worker time.perf_counter_ns with torch.cuda.synchronize at phase boundaries; fresh PP excludes prefix preparation; aggregate TG is across streams
- Measurement outcomes
- 8,192 tokens · 8 streams: oom — OutOfMemoryError: CUDA out of memory. Tried to allocate 256.00 MiB. GPU 0 has a total capacity of 23.56 GiB of which 151.06 MiB is free. Including non-PyTorch memory, this process has 23.40 GiB memory in use. 23.56 GiB allowed; Of the allocated memory 23.05 GiB is allocated by PyTorch, and 23.51 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
- Notes
- 1× RTX3090 — no splitting · EXL3 4.00bpw · MTP-3 · concurrency 8, 8192-token prefix, 8960-token allocation per stream. Persistent EXL native worker: 512 fresh prompt tokens and 256 generated tokens; one warmup then three measured repetitions. PP is fresh-prompt processing after the populated prefix; TG is aggregate decode across concurrent streams. Throughput means and sample standard deviations are recomputed from the retained raw repetitions. EXL native K8/V8 cache storage includes QSA side planes and recurrent state; this is not GGUF Q8_0. Physical GPU 0 (GPU-7aa53e24-dc9d-bf8d-96cc-ef8c05c66dab) was selected as the single device. Runtime evidence maps that physical GPU to one logical CUDA0. Source configuration 400bpw-concurrency-mtp-3-c8; model revision 113cf7ab958054860e43fb7f3063b1af19171095; source manifest SHA-256 f4213ce466cd62992d252fcd14c623cb23cee191ed865703639e884f2df15d01. The concurrency source used an 8192-token prefix and a fixed 8960-token per-stream cache allocation; this allocation is independent of context-fit capacities.
- Source article
- /benchmarks/
- Raw log path
research/benchmarks/logs/trpro/2026-10-15-independent-gpus/remaining-campaign/trpro-exl-qwen38-27b-4bpw-gpu0-independent-gpus-validation-20261015/4.00bpw/concurrency/mtp-3-c8