Qwen3.8 27B
Qwen/Qwen3.8-27B
Qwen · 22 recorded configurations
Measured setup
Hardware
Runtime and tool
Quantisation
MTP draft depth
Every option opens a recorded setup. Where the corpus recorded nothing at this page's other settings, the option names the fields it switches with it. Allocation and runtime detail stay in the technical record below.
1× RTX 3090 · ExLlamaV3
EXL3 4.00bpw · MTP 2
PP · left axisTG · right axis
View exact measurements
| Context depth (tokens) | Fresh prefill PP (tok/s) | TG (tok/s) |
|---|---|---|
| 0 | 934.94 | 68.01 |
| 16,384 | 755.31 | 45.40 |
| 171,264 | 278.54 | 31.21 |
Concurrent-stream measurements
| Streams | Context depth (tokens) | Aggregate PP (tok/s) | Aggregate TG (tok/s) | TG / stream (tok/s) | Aggregate all (tok/s) |
|---|---|---|---|---|---|
| 2 | 8,192 | 895.15 | 112.96 | 56.48 | 270.59 |
| 3 | 8,192 | 919.54 | 146.06 | 48.69 | 332.54 |
| 4 | 8,192 | 940.19 | 174.70 | 43.68 | 382.10 |
| 5 | 8,192 | 932.52 | 208.53 | 41.71 | 432.27 |
| 6 | 8,192 | 924.19 | 155.46 | 25.91 | 348.98 |
| 7 | 8,192 | 925.78 | 167.16 | 23.88 | 368.44 |
| 8 | 8,192 | 913.52 | 186.82 | 23.35 | 397.76 |
Technical record and provenance
Source 1 · 2026-10-15
- Source run ID
- trpro-exl-qwen38-27b-4bpw-gpu0-4bpw-mtp-2-context-fit-1-20261015
- Page entry ID
- trpro-exl-qwen38-27b-4bpw-gpu0-mtp-2
- Entry sweep
- context
- Raw entry label
- Qwen3.8 27B EXL3 4.00bpw · MTP-2
- Configuration label
- 1× RTX3090 — no splitting · EXL3 4.00bpw · MTP-2 · context-fit 172032-token allocation
- Measured date
- 2026-10-15
- Hardware
- 1× RTX 3090
- Runtime
- exllamav3 · ExLlamaV3 1.5.0+cu128.torch2.9.0; native extension SHA-256 c1d02b410ef729618077ba287f01f05b229f31d118a7aae04a1b7e56142e3b2e
- Weight placement
- gpu-resident
- Observed devices
- CUDA0
- Requested devices
- CUDA0
- Split mode
- Single GPU — no splitting
- Placement detail
- One physical RTX 3090; target model and native MTP compute tensors were observed on CUDA0, with no CPU expert offload or model split.
- KV-cache precision
- EXL native K8/V8 with QSA side planes and recurrent state
- Requested context per slot
- 172,032 tokens
- Actual context per slot
- 172,032 tokens
- Native model context
- 262,144 tokens
- Prompt tokens
- 512
- Generated tokens
- 256
- Speculative decoding
- MTP · draft max 2
- Warm-ups
- 1
- Measured repetitions
- 3
- Timing method
- ExLlamaV3 persistent worker time.perf_counter_ns with torch.cuda.synchronize at phase boundaries; fresh PP excludes prefix preparation; aggregate TG is across streams
- Throughput spreads
- KV 0, 1 stream: PP mean 934.94 t/s (min 930.76, max 940.15, SD 4.78, n=3); TG mean 68.01 t/s (min 67.94, max 68.10, SD 0.08, n=3); All mean 178.13 t/s (min 178.00, max 178.29, SD 0.15, n=3) · KV 16,384, 1 stream: PP mean 755.31 t/s (min 754.20, max 756.73, SD 1.29, n=3); TG mean 45.40 t/s (min 45.37, max 45.44, SD 0.04, n=3); All mean 121.58 t/s (min 121.52, max 121.67, SD 0.08, n=3) · KV 171,264, 1 stream: PP mean 278.54 t/s (min 278.33, max 278.69, SD 0.19, n=3); TG mean 31.21 t/s (min 31.19, max 31.23, SD 0.02, n=3); All mean 76.49 t/s (min 76.46, max 76.51, SD 0.03, n=3)
- Measurement outcomes
- 0 tokens · 1 stream: measured; 16,384 tokens · 1 stream: measured; 171,264 tokens · 1 stream: measured
- Notes
- 1× RTX3090 — no splitting · EXL3 4.00bpw · MTP-2 · context-fit 172032-token allocation. Persistent EXL native worker: 512 fresh prompt tokens and 256 generated tokens; one warmup then three measured repetitions. PP is fresh-prompt processing after the populated prefix; TG is aggregate decode across concurrent streams. Throughput means and sample standard deviations are recomputed from the retained raw repetitions. EXL native K8/V8 cache storage includes QSA side planes and recurrent state; this is not GGUF Q8_0. Physical GPU 0 (GPU-7aa53e24-dc9d-bf8d-96cc-ef8c05c66dab) was selected as the single device. Runtime evidence maps that physical GPU to one logical CUDA0. Source configuration 400bpw-context-mtp-2-c1; model revision 113cf7ab958054860e43fb7f3063b1af19171095; source manifest SHA-256 c6320d4dc839c08a5362579bcb8399340e3c99723bbd561e8c5fd7627c7ff081. Separately allocated 172032 tokens per stream for this MTP setting. The allocation search advances in 8192-token steps; its recorded fitted capacity is not a token-exact VRAM maximum. The selected boundary and the 0/16K samples are retained from their recorded source attempts.
- Source article
- /benchmarks/
- Raw log path
research/benchmarks/logs/trpro/2026-10-15-exl-context-fit/4bpw-gpu0/runs/mtp-2-172032-boundary/4.00bpw/context/mtp-2-c1
Source 2 · 2026-10-15
- Source run ID
- trpro-exl-qwen38-27b-4bpw-gpu0-4bpw-mtp-2-concurrency-2-20261015
- Page entry ID
- trpro-exl-qwen38-27b-4bpw-gpu0-mtp-2
- Entry sweep
- concurrency
- Raw entry label
- Qwen3.8 27B EXL3 4.00bpw · MTP-2
- Configuration label
- 1× RTX3090 — no splitting · EXL3 4.00bpw · MTP-2 · concurrency 2, 8192-token prefix, 8960-token allocation per stream
- Measured date
- 2026-10-15
- Hardware
- 1× RTX 3090
- Runtime
- exllamav3 · ExLlamaV3 1.5.0+cu128.torch2.9.0; native extension SHA-256 c1d02b410ef729618077ba287f01f05b229f31d118a7aae04a1b7e56142e3b2e
- Weight placement
- gpu-resident
- Observed devices
- CUDA0
- Requested devices
- CUDA0
- Split mode
- Single GPU — no splitting
- Placement detail
- One physical RTX 3090; target model and native MTP compute tensors were observed on CUDA0, with no CPU expert offload or model split.
- KV-cache precision
- EXL native K8/V8 with QSA side planes and recurrent state
- Requested context per slot
- 8,960 tokens
- Actual context per slot
- 8,960 tokens
- Native model context
- 262,144 tokens
- Prompt tokens
- 512
- Generated tokens
- 256
- Speculative decoding
- MTP · draft max 2
- Warm-ups
- 1
- Measured repetitions
- 3
- Timing method
- ExLlamaV3 persistent worker time.perf_counter_ns with torch.cuda.synchronize at phase boundaries; fresh PP excludes prefix preparation; aggregate TG is across streams
- Throughput spreads
- KV 8,192, 2 streams: PP mean 895.15 t/s (min 894.91, max 895.49, SD 0.30, n=3); TG mean 112.96 t/s (min 112.68, max 113.25, SD 0.28, n=3); All mean 270.59 t/s (min 270.03, max 271.13, SD 0.55, n=3)
- Measurement outcomes
- 8,192 tokens · 2 streams: measured
- Notes
- 1× RTX3090 — no splitting · EXL3 4.00bpw · MTP-2 · concurrency 2, 8192-token prefix, 8960-token allocation per stream. Persistent EXL native worker: 512 fresh prompt tokens and 256 generated tokens; one warmup then three measured repetitions. PP is fresh-prompt processing after the populated prefix; TG is aggregate decode across concurrent streams. Throughput means and sample standard deviations are recomputed from the retained raw repetitions. EXL native K8/V8 cache storage includes QSA side planes and recurrent state; this is not GGUF Q8_0. Physical GPU 0 (GPU-7aa53e24-dc9d-bf8d-96cc-ef8c05c66dab) was selected as the single device. Runtime evidence maps that physical GPU to one logical CUDA0. Source configuration 400bpw-concurrency-mtp-2-c2; model revision 113cf7ab958054860e43fb7f3063b1af19171095; source manifest SHA-256 f4213ce466cd62992d252fcd14c623cb23cee191ed865703639e884f2df15d01. The concurrency source used an 8192-token prefix and a fixed 8960-token per-stream cache allocation; this allocation is independent of context-fit capacities.
- Source article
- /benchmarks/
- Raw log path
research/benchmarks/logs/trpro/2026-10-15-independent-gpus/remaining-campaign/trpro-exl-qwen38-27b-4bpw-gpu0-independent-gpus-validation-20261015/4.00bpw/concurrency/mtp-2-c2
Source 3 · 2026-10-15
- Source run ID
- trpro-exl-qwen38-27b-4bpw-gpu0-4bpw-mtp-2-concurrency-3-20261015
- Page entry ID
- trpro-exl-qwen38-27b-4bpw-gpu0-mtp-2
- Entry sweep
- concurrency
- Raw entry label
- Qwen3.8 27B EXL3 4.00bpw · MTP-2
- Configuration label
- 1× RTX3090 — no splitting · EXL3 4.00bpw · MTP-2 · concurrency 3, 8192-token prefix, 8960-token allocation per stream
- Measured date
- 2026-10-15
- Hardware
- 1× RTX 3090
- Runtime
- exllamav3 · ExLlamaV3 1.5.0+cu128.torch2.9.0; native extension SHA-256 c1d02b410ef729618077ba287f01f05b229f31d118a7aae04a1b7e56142e3b2e
- Weight placement
- gpu-resident
- Observed devices
- CUDA0
- Requested devices
- CUDA0
- Split mode
- Single GPU — no splitting
- Placement detail
- One physical RTX 3090; target model and native MTP compute tensors were observed on CUDA0, with no CPU expert offload or model split.
- KV-cache precision
- EXL native K8/V8 with QSA side planes and recurrent state
- Requested context per slot
- 8,960 tokens
- Actual context per slot
- 8,960 tokens
- Native model context
- 262,144 tokens
- Prompt tokens
- 512
- Generated tokens
- 256
- Speculative decoding
- MTP · draft max 2
- Warm-ups
- 1
- Measured repetitions
- 3
- Timing method
- ExLlamaV3 persistent worker time.perf_counter_ns with torch.cuda.synchronize at phase boundaries; fresh PP excludes prefix preparation; aggregate TG is across streams
- Throughput spreads
- KV 8,192, 3 streams: PP mean 919.54 t/s (min 917.69, max 922.46, SD 2.56, n=3); TG mean 146.06 t/s (min 146.04, max 146.08, SD 0.02, n=3); All mean 332.54 t/s (min 332.41, max 332.78, SD 0.21, n=3)
- Measurement outcomes
- 8,192 tokens · 3 streams: measured
- Notes
- 1× RTX3090 — no splitting · EXL3 4.00bpw · MTP-2 · concurrency 3, 8192-token prefix, 8960-token allocation per stream. Persistent EXL native worker: 512 fresh prompt tokens and 256 generated tokens; one warmup then three measured repetitions. PP is fresh-prompt processing after the populated prefix; TG is aggregate decode across concurrent streams. Throughput means and sample standard deviations are recomputed from the retained raw repetitions. EXL native K8/V8 cache storage includes QSA side planes and recurrent state; this is not GGUF Q8_0. Physical GPU 0 (GPU-7aa53e24-dc9d-bf8d-96cc-ef8c05c66dab) was selected as the single device. Runtime evidence maps that physical GPU to one logical CUDA0. Source configuration 400bpw-concurrency-mtp-2-c3; model revision 113cf7ab958054860e43fb7f3063b1af19171095; source manifest SHA-256 f4213ce466cd62992d252fcd14c623cb23cee191ed865703639e884f2df15d01. The concurrency source used an 8192-token prefix and a fixed 8960-token per-stream cache allocation; this allocation is independent of context-fit capacities.
- Source article
- /benchmarks/
- Raw log path
research/benchmarks/logs/trpro/2026-10-15-independent-gpus/remaining-campaign/trpro-exl-qwen38-27b-4bpw-gpu0-independent-gpus-validation-20261015/4.00bpw/concurrency/mtp-2-c3
Source 4 · 2026-10-15
- Source run ID
- trpro-exl-qwen38-27b-4bpw-gpu0-4bpw-mtp-2-concurrency-4-20261015
- Page entry ID
- trpro-exl-qwen38-27b-4bpw-gpu0-mtp-2
- Entry sweep
- concurrency
- Raw entry label
- Qwen3.8 27B EXL3 4.00bpw · MTP-2
- Configuration label
- 1× RTX3090 — no splitting · EXL3 4.00bpw · MTP-2 · concurrency 4, 8192-token prefix, 8960-token allocation per stream
- Measured date
- 2026-10-15
- Hardware
- 1× RTX 3090
- Runtime
- exllamav3 · ExLlamaV3 1.5.0+cu128.torch2.9.0; native extension SHA-256 c1d02b410ef729618077ba287f01f05b229f31d118a7aae04a1b7e56142e3b2e
- Weight placement
- gpu-resident
- Observed devices
- CUDA0
- Requested devices
- CUDA0
- Split mode
- Single GPU — no splitting
- Placement detail
- One physical RTX 3090; target model and native MTP compute tensors were observed on CUDA0, with no CPU expert offload or model split.
- KV-cache precision
- EXL native K8/V8 with QSA side planes and recurrent state
- Requested context per slot
- 8,960 tokens
- Actual context per slot
- 8,960 tokens
- Native model context
- 262,144 tokens
- Prompt tokens
- 512
- Generated tokens
- 256
- Speculative decoding
- MTP · draft max 2
- Warm-ups
- 1
- Measured repetitions
- 3
- Timing method
- ExLlamaV3 persistent worker time.perf_counter_ns with torch.cuda.synchronize at phase boundaries; fresh PP excludes prefix preparation; aggregate TG is across streams
- Throughput spreads
- KV 8,192, 4 streams: PP mean 940.19 t/s (min 939.49, max 940.70, SD 0.63, n=3); TG mean 174.70 t/s (min 173.41, max 177.27, SD 2.23, n=3); All mean 382.10 t/s (min 380.09, max 386.11, SD 3.48, n=3)
- Measurement outcomes
- 8,192 tokens · 4 streams: measured
- Notes
- 1× RTX3090 — no splitting · EXL3 4.00bpw · MTP-2 · concurrency 4, 8192-token prefix, 8960-token allocation per stream. Persistent EXL native worker: 512 fresh prompt tokens and 256 generated tokens; one warmup then three measured repetitions. PP is fresh-prompt processing after the populated prefix; TG is aggregate decode across concurrent streams. Throughput means and sample standard deviations are recomputed from the retained raw repetitions. EXL native K8/V8 cache storage includes QSA side planes and recurrent state; this is not GGUF Q8_0. Physical GPU 0 (GPU-7aa53e24-dc9d-bf8d-96cc-ef8c05c66dab) was selected as the single device. Runtime evidence maps that physical GPU to one logical CUDA0. Source configuration 400bpw-concurrency-mtp-2-c4; model revision 113cf7ab958054860e43fb7f3063b1af19171095; source manifest SHA-256 f4213ce466cd62992d252fcd14c623cb23cee191ed865703639e884f2df15d01. The concurrency source used an 8192-token prefix and a fixed 8960-token per-stream cache allocation; this allocation is independent of context-fit capacities.
- Source article
- /benchmarks/
- Raw log path
research/benchmarks/logs/trpro/2026-10-15-independent-gpus/remaining-campaign/trpro-exl-qwen38-27b-4bpw-gpu0-independent-gpus-validation-20261015/4.00bpw/concurrency/mtp-2-c4
Source 5 · 2026-10-15
- Source run ID
- trpro-exl-qwen38-27b-4bpw-gpu0-4bpw-mtp-2-concurrency-5-20261015
- Page entry ID
- trpro-exl-qwen38-27b-4bpw-gpu0-mtp-2
- Entry sweep
- concurrency
- Raw entry label
- Qwen3.8 27B EXL3 4.00bpw · MTP-2
- Configuration label
- 1× RTX3090 — no splitting · EXL3 4.00bpw · MTP-2 · concurrency 5, 8192-token prefix, 8960-token allocation per stream
- Measured date
- 2026-10-15
- Hardware
- 1× RTX 3090
- Runtime
- exllamav3 · ExLlamaV3 1.5.0+cu128.torch2.9.0; native extension SHA-256 c1d02b410ef729618077ba287f01f05b229f31d118a7aae04a1b7e56142e3b2e
- Weight placement
- gpu-resident
- Observed devices
- CUDA0
- Requested devices
- CUDA0
- Split mode
- Single GPU — no splitting
- Placement detail
- One physical RTX 3090; target model and native MTP compute tensors were observed on CUDA0, with no CPU expert offload or model split.
- KV-cache precision
- EXL native K8/V8 with QSA side planes and recurrent state
- Requested context per slot
- 8,960 tokens
- Actual context per slot
- 8,960 tokens
- Native model context
- 262,144 tokens
- Prompt tokens
- 512
- Generated tokens
- 256
- Speculative decoding
- MTP · draft max 2
- Warm-ups
- 1
- Measured repetitions
- 3
- Timing method
- ExLlamaV3 persistent worker time.perf_counter_ns with torch.cuda.synchronize at phase boundaries; fresh PP excludes prefix preparation; aggregate TG is across streams
- Throughput spreads
- KV 8,192, 5 streams: PP mean 932.52 t/s (min 931.42, max 933.91, SD 1.27, n=3); TG mean 208.53 t/s (min 207.83, max 208.96, SD 0.61, n=3); All mean 432.27 t/s (min 431.23, max 432.86, SD 0.90, n=3)
- Measurement outcomes
- 8,192 tokens · 5 streams: measured
- Notes
- 1× RTX3090 — no splitting · EXL3 4.00bpw · MTP-2 · concurrency 5, 8192-token prefix, 8960-token allocation per stream. Persistent EXL native worker: 512 fresh prompt tokens and 256 generated tokens; one warmup then three measured repetitions. PP is fresh-prompt processing after the populated prefix; TG is aggregate decode across concurrent streams. Throughput means and sample standard deviations are recomputed from the retained raw repetitions. EXL native K8/V8 cache storage includes QSA side planes and recurrent state; this is not GGUF Q8_0. Physical GPU 0 (GPU-7aa53e24-dc9d-bf8d-96cc-ef8c05c66dab) was selected as the single device. Runtime evidence maps that physical GPU to one logical CUDA0. Source configuration 400bpw-concurrency-mtp-2-c5; model revision 113cf7ab958054860e43fb7f3063b1af19171095; source manifest SHA-256 f4213ce466cd62992d252fcd14c623cb23cee191ed865703639e884f2df15d01. The concurrency source used an 8192-token prefix and a fixed 8960-token per-stream cache allocation; this allocation is independent of context-fit capacities.
- Source article
- /benchmarks/
- Raw log path
research/benchmarks/logs/trpro/2026-10-15-independent-gpus/remaining-campaign/trpro-exl-qwen38-27b-4bpw-gpu0-independent-gpus-validation-20261015/4.00bpw/concurrency/mtp-2-c5
Source 6 · 2026-10-15
- Source run ID
- trpro-exl-qwen38-27b-4bpw-gpu0-4bpw-mtp-2-concurrency-6-20261015
- Page entry ID
- trpro-exl-qwen38-27b-4bpw-gpu0-mtp-2
- Entry sweep
- concurrency
- Raw entry label
- Qwen3.8 27B EXL3 4.00bpw · MTP-2
- Configuration label
- 1× RTX3090 — no splitting · EXL3 4.00bpw · MTP-2 · concurrency 6, 8192-token prefix, 8960-token allocation per stream
- Measured date
- 2026-10-15
- Hardware
- 1× RTX 3090
- Runtime
- exllamav3 · ExLlamaV3 1.5.0+cu128.torch2.9.0; native extension SHA-256 c1d02b410ef729618077ba287f01f05b229f31d118a7aae04a1b7e56142e3b2e
- Weight placement
- gpu-resident
- Observed devices
- CUDA0
- Requested devices
- CUDA0
- Split mode
- Single GPU — no splitting
- Placement detail
- One physical RTX 3090; target model and native MTP compute tensors were observed on CUDA0, with no CPU expert offload or model split.
- KV-cache precision
- EXL native K8/V8 with QSA side planes and recurrent state
- Requested context per slot
- 8,960 tokens
- Actual context per slot
- 8,960 tokens
- Native model context
- 262,144 tokens
- Prompt tokens
- 512
- Generated tokens
- 256
- Speculative decoding
- MTP · draft max 2
- Warm-ups
- 1
- Measured repetitions
- 3
- Timing method
- ExLlamaV3 persistent worker time.perf_counter_ns with torch.cuda.synchronize at phase boundaries; fresh PP excludes prefix preparation; aggregate TG is across streams
- Throughput spreads
- KV 8,192, 6 streams: PP mean 924.19 t/s (min 923.37, max 925.21, SD 0.94, n=3); TG mean 155.46 t/s (min 155.34, max 155.59, SD 0.13, n=3); All mean 348.98 t/s (min 348.69, max 349.29, SD 0.30, n=3)
- Measurement outcomes
- 8,192 tokens · 6 streams: measured
- Notes
- 1× RTX3090 — no splitting · EXL3 4.00bpw · MTP-2 · concurrency 6, 8192-token prefix, 8960-token allocation per stream. Persistent EXL native worker: 512 fresh prompt tokens and 256 generated tokens; one warmup then three measured repetitions. PP is fresh-prompt processing after the populated prefix; TG is aggregate decode across concurrent streams. Throughput means and sample standard deviations are recomputed from the retained raw repetitions. EXL native K8/V8 cache storage includes QSA side planes and recurrent state; this is not GGUF Q8_0. Physical GPU 0 (GPU-7aa53e24-dc9d-bf8d-96cc-ef8c05c66dab) was selected as the single device. Runtime evidence maps that physical GPU to one logical CUDA0. Source configuration 400bpw-concurrency-mtp-2-c6; model revision 113cf7ab958054860e43fb7f3063b1af19171095; source manifest SHA-256 f4213ce466cd62992d252fcd14c623cb23cee191ed865703639e884f2df15d01. The concurrency source used an 8192-token prefix and a fixed 8960-token per-stream cache allocation; this allocation is independent of context-fit capacities.
- Source article
- /benchmarks/
- Raw log path
research/benchmarks/logs/trpro/2026-10-15-independent-gpus/remaining-campaign/trpro-exl-qwen38-27b-4bpw-gpu0-independent-gpus-validation-20261015/4.00bpw/concurrency/mtp-2-c6
Source 7 · 2026-10-15
- Source run ID
- trpro-exl-qwen38-27b-4bpw-gpu0-4bpw-mtp-2-concurrency-7-20261015
- Page entry ID
- trpro-exl-qwen38-27b-4bpw-gpu0-mtp-2
- Entry sweep
- concurrency
- Raw entry label
- Qwen3.8 27B EXL3 4.00bpw · MTP-2
- Configuration label
- 1× RTX3090 — no splitting · EXL3 4.00bpw · MTP-2 · concurrency 7, 8192-token prefix, 8960-token allocation per stream
- Measured date
- 2026-10-15
- Hardware
- 1× RTX 3090
- Runtime
- exllamav3 · ExLlamaV3 1.5.0+cu128.torch2.9.0; native extension SHA-256 c1d02b410ef729618077ba287f01f05b229f31d118a7aae04a1b7e56142e3b2e
- Weight placement
- gpu-resident
- Observed devices
- CUDA0
- Requested devices
- CUDA0
- Split mode
- Single GPU — no splitting
- Placement detail
- One physical RTX 3090; target model and native MTP compute tensors were observed on CUDA0, with no CPU expert offload or model split.
- KV-cache precision
- EXL native K8/V8 with QSA side planes and recurrent state
- Requested context per slot
- 8,960 tokens
- Actual context per slot
- 8,960 tokens
- Native model context
- 262,144 tokens
- Prompt tokens
- 512
- Generated tokens
- 256
- Speculative decoding
- MTP · draft max 2
- Warm-ups
- 1
- Measured repetitions
- 3
- Timing method
- ExLlamaV3 persistent worker time.perf_counter_ns with torch.cuda.synchronize at phase boundaries; fresh PP excludes prefix preparation; aggregate TG is across streams
- Throughput spreads
- KV 8,192, 7 streams: PP mean 925.78 t/s (min 924.23, max 927.40, SD 1.59, n=3); TG mean 167.16 t/s (min 167.04, max 167.41, SD 0.22, n=3); All mean 368.44 t/s (min 368.07, max 369.01, SD 0.51, n=3)
- Measurement outcomes
- 8,192 tokens · 7 streams: measured
- Notes
- 1× RTX3090 — no splitting · EXL3 4.00bpw · MTP-2 · concurrency 7, 8192-token prefix, 8960-token allocation per stream. Persistent EXL native worker: 512 fresh prompt tokens and 256 generated tokens; one warmup then three measured repetitions. PP is fresh-prompt processing after the populated prefix; TG is aggregate decode across concurrent streams. Throughput means and sample standard deviations are recomputed from the retained raw repetitions. EXL native K8/V8 cache storage includes QSA side planes and recurrent state; this is not GGUF Q8_0. Physical GPU 0 (GPU-7aa53e24-dc9d-bf8d-96cc-ef8c05c66dab) was selected as the single device. Runtime evidence maps that physical GPU to one logical CUDA0. Source configuration 400bpw-concurrency-mtp-2-c7; model revision 113cf7ab958054860e43fb7f3063b1af19171095; source manifest SHA-256 f4213ce466cd62992d252fcd14c623cb23cee191ed865703639e884f2df15d01. The concurrency source used an 8192-token prefix and a fixed 8960-token per-stream cache allocation; this allocation is independent of context-fit capacities.
- Source article
- /benchmarks/
- Raw log path
research/benchmarks/logs/trpro/2026-10-15-independent-gpus/remaining-campaign/trpro-exl-qwen38-27b-4bpw-gpu0-independent-gpus-validation-20261015/4.00bpw/concurrency/mtp-2-c7
Source 8 · 2026-10-15
- Source run ID
- trpro-exl-qwen38-27b-4bpw-gpu0-4bpw-mtp-2-concurrency-8-20261015
- Page entry ID
- trpro-exl-qwen38-27b-4bpw-gpu0-mtp-2
- Entry sweep
- concurrency
- Raw entry label
- Qwen3.8 27B EXL3 4.00bpw · MTP-2
- Configuration label
- 1× RTX3090 — no splitting · EXL3 4.00bpw · MTP-2 · concurrency 8, 8192-token prefix, 8960-token allocation per stream
- Measured date
- 2026-10-15
- Hardware
- 1× RTX 3090
- Runtime
- exllamav3 · ExLlamaV3 1.5.0+cu128.torch2.9.0; native extension SHA-256 c1d02b410ef729618077ba287f01f05b229f31d118a7aae04a1b7e56142e3b2e
- Weight placement
- gpu-resident
- Observed devices
- CUDA0
- Requested devices
- CUDA0
- Split mode
- Single GPU — no splitting
- Placement detail
- One physical RTX 3090; target model and native MTP compute tensors were observed on CUDA0, with no CPU expert offload or model split.
- KV-cache precision
- EXL native K8/V8 with QSA side planes and recurrent state
- Requested context per slot
- 8,960 tokens
- Actual context per slot
- 8,960 tokens
- Native model context
- 262,144 tokens
- Prompt tokens
- 512
- Generated tokens
- 256
- Speculative decoding
- MTP · draft max 2
- Warm-ups
- 1
- Measured repetitions
- 3
- Timing method
- ExLlamaV3 persistent worker time.perf_counter_ns with torch.cuda.synchronize at phase boundaries; fresh PP excludes prefix preparation; aggregate TG is across streams
- Throughput spreads
- KV 8,192, 8 streams: PP mean 913.52 t/s (min 912.24, max 914.50, SD 1.16, n=3); TG mean 186.82 t/s (min 186.70, max 186.95, SD 0.13, n=3); All mean 397.76 t/s (min 397.42, max 398.01, SD 0.30, n=3)
- Measurement outcomes
- 8,192 tokens · 8 streams: measured
- Notes
- 1× RTX3090 — no splitting · EXL3 4.00bpw · MTP-2 · concurrency 8, 8192-token prefix, 8960-token allocation per stream. Persistent EXL native worker: 512 fresh prompt tokens and 256 generated tokens; one warmup then three measured repetitions. PP is fresh-prompt processing after the populated prefix; TG is aggregate decode across concurrent streams. Throughput means and sample standard deviations are recomputed from the retained raw repetitions. EXL native K8/V8 cache storage includes QSA side planes and recurrent state; this is not GGUF Q8_0. Physical GPU 0 (GPU-7aa53e24-dc9d-bf8d-96cc-ef8c05c66dab) was selected as the single device. Runtime evidence maps that physical GPU to one logical CUDA0. Source configuration 400bpw-concurrency-mtp-2-c8; model revision 113cf7ab958054860e43fb7f3063b1af19171095; source manifest SHA-256 f4213ce466cd62992d252fcd14c623cb23cee191ed865703639e884f2df15d01. The concurrency source used an 8192-token prefix and a fixed 8960-token per-stream cache allocation; this allocation is independent of context-fit capacities.
- Source article
- /benchmarks/
- Raw log path
research/benchmarks/logs/trpro/2026-10-15-independent-gpus/remaining-campaign/trpro-exl-qwen38-27b-4bpw-gpu0-independent-gpus-validation-20261015/4.00bpw/concurrency/mtp-2-c8