Qwen3.8 27B
Qwen/Qwen3.8-27B
Qwen · 22 recorded configurations
Measured setup
Hardware
Runtime and tool
Quantisation
MTP draft depth
Every option opens a recorded setup. Where the corpus recorded nothing at this page's other settings, the option names the fields it switches with it. Allocation and runtime detail stay in the technical record below.
1× RTX 3090 · ExLlamaV3
EXL3 4.00bpw · MTP 1
PP · left axisTG · right axis
View exact measurements
| Context depth (tokens) | Fresh prefill PP (tok/s) | TG (tok/s) |
|---|---|---|
| 0 | 934.87 | 55.56 |
| 16,384 | 755.73 | 42.70 |
| 179,456 | 269.98 | 32.38 |
Concurrent-stream measurements
| Streams | Context depth (tokens) | Aggregate PP (tok/s) | Aggregate TG (tok/s) | TG / stream (tok/s) | Aggregate all (tok/s) |
|---|---|---|---|---|---|
| 2 | 8,192 | 893.56 | 101.86 | 50.93 | 248.84 |
| 3 | 8,192 | 927.85 | 139.00 | 46.33 | 320.87 |
| 4 | 8,192 | 943.14 | 171.78 | 42.94 | 377.74 |
| 5 | 8,192 | 942.93 | 187.78 | 37.56 | 402.88 |
| 6 | 8,192 | 923.61 | 217.91 | 36.32 | 444.14 |
| 7 | 8,192 | 922.17 | 237.66 | 33.95 | 470.47 |
| 8 | 8,192 | 921.32 | 260.50 | 32.56 | 499.20 |
Technical record and provenance
Source 1 · 2026-10-15
- Source run ID
- trpro-exl-qwen38-27b-4bpw-gpu0-4bpw-mtp-1-context-fit-1-20261015
- Page entry ID
- trpro-exl-qwen38-27b-4bpw-gpu0-mtp-1
- Entry sweep
- context
- Raw entry label
- Qwen3.8 27B EXL3 4.00bpw · MTP-1
- Configuration label
- 1× RTX3090 — no splitting · EXL3 4.00bpw · MTP-1 · context-fit 180224-token allocation
- Measured date
- 2026-10-15
- Hardware
- 1× RTX 3090
- Runtime
- exllamav3 · ExLlamaV3 1.5.0+cu128.torch2.9.0; native extension SHA-256 c1d02b410ef729618077ba287f01f05b229f31d118a7aae04a1b7e56142e3b2e
- Weight placement
- gpu-resident
- Observed devices
- CUDA0
- Requested devices
- CUDA0
- Split mode
- Single GPU — no splitting
- Placement detail
- One physical RTX 3090; target model and native MTP compute tensors were observed on CUDA0, with no CPU expert offload or model split.
- KV-cache precision
- EXL native K8/V8 with QSA side planes and recurrent state
- Requested context per slot
- 180,224 tokens
- Actual context per slot
- 180,224 tokens
- Native model context
- 262,144 tokens
- Prompt tokens
- 512
- Generated tokens
- 256
- Speculative decoding
- MTP · draft max 1
- Warm-ups
- 1
- Measured repetitions
- 3
- Timing method
- ExLlamaV3 persistent worker time.perf_counter_ns with torch.cuda.synchronize at phase boundaries; fresh PP excludes prefix preparation; aggregate TG is across streams
- Throughput spreads
- KV 0, 1 stream: PP mean 934.87 t/s (min 927.22, max 939.71, SD 6.70, n=3); TG mean 55.56 t/s (min 55.54, max 55.58, SD 0.02, n=3); All mean 148.96 t/s (min 148.89, max 149.01, SD 0.06, n=3) · KV 16,384, 1 stream: PP mean 755.73 t/s (min 755.12, max 756.63, SD 0.80, n=3); TG mean 42.70 t/s (min 42.67, max 42.74, SD 0.03, n=3); All mean 115.10 t/s (min 115.02, max 115.17, SD 0.08, n=3) · KV 179,456, 1 stream: PP mean 269.98 t/s (min 269.49, max 270.49, SD 0.50, n=3); TG mean 32.38 t/s (min 32.38, max 32.39, SD 0.01, n=3); All mean 78.35 t/s (min 78.32, max 78.40, SD 0.04, n=3)
- Measurement outcomes
- 0 tokens · 1 stream: measured; 16,384 tokens · 1 stream: measured; 179,456 tokens · 1 stream: measured
- Notes
- 1× RTX3090 — no splitting · EXL3 4.00bpw · MTP-1 · context-fit 180224-token allocation. Persistent EXL native worker: 512 fresh prompt tokens and 256 generated tokens; one warmup then three measured repetitions. PP is fresh-prompt processing after the populated prefix; TG is aggregate decode across concurrent streams. Throughput means and sample standard deviations are recomputed from the retained raw repetitions. EXL native K8/V8 cache storage includes QSA side planes and recurrent state; this is not GGUF Q8_0. Physical GPU 0 (GPU-7aa53e24-dc9d-bf8d-96cc-ef8c05c66dab) was selected as the single device. Runtime evidence maps that physical GPU to one logical CUDA0. Source configuration 400bpw-context-mtp-1-c1; model revision 113cf7ab958054860e43fb7f3063b1af19171095; source manifest SHA-256 3e7006238aed1bf8cfd86a4db459506daafc4a2a904253f9cad9d0bd5d3ba139. Separately allocated 180224 tokens per stream for this MTP setting. The allocation search advances in 8192-token steps; its recorded fitted capacity is not a token-exact VRAM maximum. The selected boundary and the 0/16K samples are retained from their recorded source attempts.
- Source article
- /benchmarks/
- Raw log path
research/benchmarks/logs/trpro/2026-10-15-exl-context-fit/4bpw-gpu0/runs/mtp-1-180224-boundary/4.00bpw/context/mtp-1-c1
Source 2 · 2026-10-15
- Source run ID
- trpro-exl-qwen38-27b-4bpw-gpu0-4bpw-mtp-1-concurrency-2-20261015
- Page entry ID
- trpro-exl-qwen38-27b-4bpw-gpu0-mtp-1
- Entry sweep
- concurrency
- Raw entry label
- Qwen3.8 27B EXL3 4.00bpw · MTP-1
- Configuration label
- 1× RTX3090 — no splitting · EXL3 4.00bpw · MTP-1 · concurrency 2, 8192-token prefix, 8960-token allocation per stream
- Measured date
- 2026-10-15
- Hardware
- 1× RTX 3090
- Runtime
- exllamav3 · ExLlamaV3 1.5.0+cu128.torch2.9.0; native extension SHA-256 c1d02b410ef729618077ba287f01f05b229f31d118a7aae04a1b7e56142e3b2e
- Weight placement
- gpu-resident
- Observed devices
- CUDA0
- Requested devices
- CUDA0
- Split mode
- Single GPU — no splitting
- Placement detail
- One physical RTX 3090; target model and native MTP compute tensors were observed on CUDA0, with no CPU expert offload or model split.
- KV-cache precision
- EXL native K8/V8 with QSA side planes and recurrent state
- Requested context per slot
- 8,960 tokens
- Actual context per slot
- 8,960 tokens
- Native model context
- 262,144 tokens
- Prompt tokens
- 512
- Generated tokens
- 256
- Speculative decoding
- MTP · draft max 1
- Warm-ups
- 1
- Measured repetitions
- 3
- Timing method
- ExLlamaV3 persistent worker time.perf_counter_ns with torch.cuda.synchronize at phase boundaries; fresh PP excludes prefix preparation; aggregate TG is across streams
- Throughput spreads
- KV 8,192, 2 streams: PP mean 893.56 t/s (min 892.16, max 895.87, SD 2.02, n=3); TG mean 101.86 t/s (min 101.68, max 102.02, SD 0.17, n=3); All mean 248.84 t/s (min 248.41, max 249.29, SD 0.44, n=3)
- Measurement outcomes
- 8,192 tokens · 2 streams: measured
- Notes
- 1× RTX3090 — no splitting · EXL3 4.00bpw · MTP-1 · concurrency 2, 8192-token prefix, 8960-token allocation per stream. Persistent EXL native worker: 512 fresh prompt tokens and 256 generated tokens; one warmup then three measured repetitions. PP is fresh-prompt processing after the populated prefix; TG is aggregate decode across concurrent streams. Throughput means and sample standard deviations are recomputed from the retained raw repetitions. EXL native K8/V8 cache storage includes QSA side planes and recurrent state; this is not GGUF Q8_0. Physical GPU 0 (GPU-7aa53e24-dc9d-bf8d-96cc-ef8c05c66dab) was selected as the single device. Runtime evidence maps that physical GPU to one logical CUDA0. Source configuration 400bpw-concurrency-mtp-1-c2; model revision 113cf7ab958054860e43fb7f3063b1af19171095; source manifest SHA-256 f4213ce466cd62992d252fcd14c623cb23cee191ed865703639e884f2df15d01. The concurrency source used an 8192-token prefix and a fixed 8960-token per-stream cache allocation; this allocation is independent of context-fit capacities.
- Source article
- /benchmarks/
- Raw log path
research/benchmarks/logs/trpro/2026-10-15-independent-gpus/remaining-campaign/trpro-exl-qwen38-27b-4bpw-gpu0-independent-gpus-validation-20261015/4.00bpw/concurrency/mtp-1-c2
Source 3 · 2026-10-15
- Source run ID
- trpro-exl-qwen38-27b-4bpw-gpu0-4bpw-mtp-1-concurrency-3-20261015
- Page entry ID
- trpro-exl-qwen38-27b-4bpw-gpu0-mtp-1
- Entry sweep
- concurrency
- Raw entry label
- Qwen3.8 27B EXL3 4.00bpw · MTP-1
- Configuration label
- 1× RTX3090 — no splitting · EXL3 4.00bpw · MTP-1 · concurrency 3, 8192-token prefix, 8960-token allocation per stream
- Measured date
- 2026-10-15
- Hardware
- 1× RTX 3090
- Runtime
- exllamav3 · ExLlamaV3 1.5.0+cu128.torch2.9.0; native extension SHA-256 c1d02b410ef729618077ba287f01f05b229f31d118a7aae04a1b7e56142e3b2e
- Weight placement
- gpu-resident
- Observed devices
- CUDA0
- Requested devices
- CUDA0
- Split mode
- Single GPU — no splitting
- Placement detail
- One physical RTX 3090; target model and native MTP compute tensors were observed on CUDA0, with no CPU expert offload or model split.
- KV-cache precision
- EXL native K8/V8 with QSA side planes and recurrent state
- Requested context per slot
- 8,960 tokens
- Actual context per slot
- 8,960 tokens
- Native model context
- 262,144 tokens
- Prompt tokens
- 512
- Generated tokens
- 256
- Speculative decoding
- MTP · draft max 1
- Warm-ups
- 1
- Measured repetitions
- 3
- Timing method
- ExLlamaV3 persistent worker time.perf_counter_ns with torch.cuda.synchronize at phase boundaries; fresh PP excludes prefix preparation; aggregate TG is across streams
- Throughput spreads
- KV 8,192, 3 streams: PP mean 927.85 t/s (min 925.37, max 929.56, SD 2.20, n=3); TG mean 139.00 t/s (min 138.09, max 139.75, SD 0.84, n=3); All mean 320.87 t/s (min 319.38, max 322.26, SD 1.44, n=3)
- Measurement outcomes
- 8,192 tokens · 3 streams: measured
- Notes
- 1× RTX3090 — no splitting · EXL3 4.00bpw · MTP-1 · concurrency 3, 8192-token prefix, 8960-token allocation per stream. Persistent EXL native worker: 512 fresh prompt tokens and 256 generated tokens; one warmup then three measured repetitions. PP is fresh-prompt processing after the populated prefix; TG is aggregate decode across concurrent streams. Throughput means and sample standard deviations are recomputed from the retained raw repetitions. EXL native K8/V8 cache storage includes QSA side planes and recurrent state; this is not GGUF Q8_0. Physical GPU 0 (GPU-7aa53e24-dc9d-bf8d-96cc-ef8c05c66dab) was selected as the single device. Runtime evidence maps that physical GPU to one logical CUDA0. Source configuration 400bpw-concurrency-mtp-1-c3; model revision 113cf7ab958054860e43fb7f3063b1af19171095; source manifest SHA-256 f4213ce466cd62992d252fcd14c623cb23cee191ed865703639e884f2df15d01. The concurrency source used an 8192-token prefix and a fixed 8960-token per-stream cache allocation; this allocation is independent of context-fit capacities.
- Source article
- /benchmarks/
- Raw log path
research/benchmarks/logs/trpro/2026-10-15-independent-gpus/remaining-campaign/trpro-exl-qwen38-27b-4bpw-gpu0-independent-gpus-validation-20261015/4.00bpw/concurrency/mtp-1-c3
Source 4 · 2026-10-15
- Source run ID
- trpro-exl-qwen38-27b-4bpw-gpu0-4bpw-mtp-1-concurrency-4-20261015
- Page entry ID
- trpro-exl-qwen38-27b-4bpw-gpu0-mtp-1
- Entry sweep
- concurrency
- Raw entry label
- Qwen3.8 27B EXL3 4.00bpw · MTP-1
- Configuration label
- 1× RTX3090 — no splitting · EXL3 4.00bpw · MTP-1 · concurrency 4, 8192-token prefix, 8960-token allocation per stream
- Measured date
- 2026-10-15
- Hardware
- 1× RTX 3090
- Runtime
- exllamav3 · ExLlamaV3 1.5.0+cu128.torch2.9.0; native extension SHA-256 c1d02b410ef729618077ba287f01f05b229f31d118a7aae04a1b7e56142e3b2e
- Weight placement
- gpu-resident
- Observed devices
- CUDA0
- Requested devices
- CUDA0
- Split mode
- Single GPU — no splitting
- Placement detail
- One physical RTX 3090; target model and native MTP compute tensors were observed on CUDA0, with no CPU expert offload or model split.
- KV-cache precision
- EXL native K8/V8 with QSA side planes and recurrent state
- Requested context per slot
- 8,960 tokens
- Actual context per slot
- 8,960 tokens
- Native model context
- 262,144 tokens
- Prompt tokens
- 512
- Generated tokens
- 256
- Speculative decoding
- MTP · draft max 1
- Warm-ups
- 1
- Measured repetitions
- 3
- Timing method
- ExLlamaV3 persistent worker time.perf_counter_ns with torch.cuda.synchronize at phase boundaries; fresh PP excludes prefix preparation; aggregate TG is across streams
- Throughput spreads
- KV 8,192, 4 streams: PP mean 943.14 t/s (min 941.07, max 944.29, SD 1.80, n=3); TG mean 171.78 t/s (min 171.75, max 171.84, SD 0.05, n=3); All mean 377.74 t/s (min 377.47, max 377.96, SD 0.25, n=3)
- Measurement outcomes
- 8,192 tokens · 4 streams: measured
- Notes
- 1× RTX3090 — no splitting · EXL3 4.00bpw · MTP-1 · concurrency 4, 8192-token prefix, 8960-token allocation per stream. Persistent EXL native worker: 512 fresh prompt tokens and 256 generated tokens; one warmup then three measured repetitions. PP is fresh-prompt processing after the populated prefix; TG is aggregate decode across concurrent streams. Throughput means and sample standard deviations are recomputed from the retained raw repetitions. EXL native K8/V8 cache storage includes QSA side planes and recurrent state; this is not GGUF Q8_0. Physical GPU 0 (GPU-7aa53e24-dc9d-bf8d-96cc-ef8c05c66dab) was selected as the single device. Runtime evidence maps that physical GPU to one logical CUDA0. Source configuration 400bpw-concurrency-mtp-1-c4; model revision 113cf7ab958054860e43fb7f3063b1af19171095; source manifest SHA-256 f4213ce466cd62992d252fcd14c623cb23cee191ed865703639e884f2df15d01. The concurrency source used an 8192-token prefix and a fixed 8960-token per-stream cache allocation; this allocation is independent of context-fit capacities.
- Source article
- /benchmarks/
- Raw log path
research/benchmarks/logs/trpro/2026-10-15-independent-gpus/remaining-campaign/trpro-exl-qwen38-27b-4bpw-gpu0-independent-gpus-validation-20261015/4.00bpw/concurrency/mtp-1-c4
Source 5 · 2026-10-15
- Source run ID
- trpro-exl-qwen38-27b-4bpw-gpu0-4bpw-mtp-1-concurrency-5-20261015
- Page entry ID
- trpro-exl-qwen38-27b-4bpw-gpu0-mtp-1
- Entry sweep
- concurrency
- Raw entry label
- Qwen3.8 27B EXL3 4.00bpw · MTP-1
- Configuration label
- 1× RTX3090 — no splitting · EXL3 4.00bpw · MTP-1 · concurrency 5, 8192-token prefix, 8960-token allocation per stream
- Measured date
- 2026-10-15
- Hardware
- 1× RTX 3090
- Runtime
- exllamav3 · ExLlamaV3 1.5.0+cu128.torch2.9.0; native extension SHA-256 c1d02b410ef729618077ba287f01f05b229f31d118a7aae04a1b7e56142e3b2e
- Weight placement
- gpu-resident
- Observed devices
- CUDA0
- Requested devices
- CUDA0
- Split mode
- Single GPU — no splitting
- Placement detail
- One physical RTX 3090; target model and native MTP compute tensors were observed on CUDA0, with no CPU expert offload or model split.
- KV-cache precision
- EXL native K8/V8 with QSA side planes and recurrent state
- Requested context per slot
- 8,960 tokens
- Actual context per slot
- 8,960 tokens
- Native model context
- 262,144 tokens
- Prompt tokens
- 512
- Generated tokens
- 256
- Speculative decoding
- MTP · draft max 1
- Warm-ups
- 1
- Measured repetitions
- 3
- Timing method
- ExLlamaV3 persistent worker time.perf_counter_ns with torch.cuda.synchronize at phase boundaries; fresh PP excludes prefix preparation; aggregate TG is across streams
- Throughput spreads
- KV 8,192, 5 streams: PP mean 942.93 t/s (min 941.65, max 943.90, SD 1.16, n=3); TG mean 187.78 t/s (min 187.18, max 188.22, SD 0.54, n=3); All mean 402.88 t/s (min 401.80, max 403.66, SD 0.97, n=3)
- Measurement outcomes
- 8,192 tokens · 5 streams: measured
- Notes
- 1× RTX3090 — no splitting · EXL3 4.00bpw · MTP-1 · concurrency 5, 8192-token prefix, 8960-token allocation per stream. Persistent EXL native worker: 512 fresh prompt tokens and 256 generated tokens; one warmup then three measured repetitions. PP is fresh-prompt processing after the populated prefix; TG is aggregate decode across concurrent streams. Throughput means and sample standard deviations are recomputed from the retained raw repetitions. EXL native K8/V8 cache storage includes QSA side planes and recurrent state; this is not GGUF Q8_0. Physical GPU 0 (GPU-7aa53e24-dc9d-bf8d-96cc-ef8c05c66dab) was selected as the single device. Runtime evidence maps that physical GPU to one logical CUDA0. Source configuration 400bpw-concurrency-mtp-1-c5; model revision 113cf7ab958054860e43fb7f3063b1af19171095; source manifest SHA-256 f4213ce466cd62992d252fcd14c623cb23cee191ed865703639e884f2df15d01. The concurrency source used an 8192-token prefix and a fixed 8960-token per-stream cache allocation; this allocation is independent of context-fit capacities.
- Source article
- /benchmarks/
- Raw log path
research/benchmarks/logs/trpro/2026-10-15-independent-gpus/remaining-campaign/trpro-exl-qwen38-27b-4bpw-gpu0-independent-gpus-validation-20261015/4.00bpw/concurrency/mtp-1-c5
Source 6 · 2026-10-15
- Source run ID
- trpro-exl-qwen38-27b-4bpw-gpu0-4bpw-mtp-1-concurrency-6-20261015
- Page entry ID
- trpro-exl-qwen38-27b-4bpw-gpu0-mtp-1
- Entry sweep
- concurrency
- Raw entry label
- Qwen3.8 27B EXL3 4.00bpw · MTP-1
- Configuration label
- 1× RTX3090 — no splitting · EXL3 4.00bpw · MTP-1 · concurrency 6, 8192-token prefix, 8960-token allocation per stream
- Measured date
- 2026-10-15
- Hardware
- 1× RTX 3090
- Runtime
- exllamav3 · ExLlamaV3 1.5.0+cu128.torch2.9.0; native extension SHA-256 c1d02b410ef729618077ba287f01f05b229f31d118a7aae04a1b7e56142e3b2e
- Weight placement
- gpu-resident
- Observed devices
- CUDA0
- Requested devices
- CUDA0
- Split mode
- Single GPU — no splitting
- Placement detail
- One physical RTX 3090; target model and native MTP compute tensors were observed on CUDA0, with no CPU expert offload or model split.
- KV-cache precision
- EXL native K8/V8 with QSA side planes and recurrent state
- Requested context per slot
- 8,960 tokens
- Actual context per slot
- 8,960 tokens
- Native model context
- 262,144 tokens
- Prompt tokens
- 512
- Generated tokens
- 256
- Speculative decoding
- MTP · draft max 1
- Warm-ups
- 1
- Measured repetitions
- 3
- Timing method
- ExLlamaV3 persistent worker time.perf_counter_ns with torch.cuda.synchronize at phase boundaries; fresh PP excludes prefix preparation; aggregate TG is across streams
- Throughput spreads
- KV 8,192, 6 streams: PP mean 923.61 t/s (min 922.83, max 924.81, SD 1.06, n=3); TG mean 217.91 t/s (min 217.77, max 218.01, SD 0.13, n=3); All mean 444.14 t/s (min 443.83, max 444.47, SD 0.32, n=3)
- Measurement outcomes
- 8,192 tokens · 6 streams: measured
- Notes
- 1× RTX3090 — no splitting · EXL3 4.00bpw · MTP-1 · concurrency 6, 8192-token prefix, 8960-token allocation per stream. Persistent EXL native worker: 512 fresh prompt tokens and 256 generated tokens; one warmup then three measured repetitions. PP is fresh-prompt processing after the populated prefix; TG is aggregate decode across concurrent streams. Throughput means and sample standard deviations are recomputed from the retained raw repetitions. EXL native K8/V8 cache storage includes QSA side planes and recurrent state; this is not GGUF Q8_0. Physical GPU 0 (GPU-7aa53e24-dc9d-bf8d-96cc-ef8c05c66dab) was selected as the single device. Runtime evidence maps that physical GPU to one logical CUDA0. Source configuration 400bpw-concurrency-mtp-1-c6; model revision 113cf7ab958054860e43fb7f3063b1af19171095; source manifest SHA-256 f4213ce466cd62992d252fcd14c623cb23cee191ed865703639e884f2df15d01. The concurrency source used an 8192-token prefix and a fixed 8960-token per-stream cache allocation; this allocation is independent of context-fit capacities.
- Source article
- /benchmarks/
- Raw log path
research/benchmarks/logs/trpro/2026-10-15-independent-gpus/remaining-campaign/trpro-exl-qwen38-27b-4bpw-gpu0-independent-gpus-validation-20261015/4.00bpw/concurrency/mtp-1-c6
Source 7 · 2026-10-15
- Source run ID
- trpro-exl-qwen38-27b-4bpw-gpu0-4bpw-mtp-1-concurrency-7-20261015
- Page entry ID
- trpro-exl-qwen38-27b-4bpw-gpu0-mtp-1
- Entry sweep
- concurrency
- Raw entry label
- Qwen3.8 27B EXL3 4.00bpw · MTP-1
- Configuration label
- 1× RTX3090 — no splitting · EXL3 4.00bpw · MTP-1 · concurrency 7, 8192-token prefix, 8960-token allocation per stream
- Measured date
- 2026-10-15
- Hardware
- 1× RTX 3090
- Runtime
- exllamav3 · ExLlamaV3 1.5.0+cu128.torch2.9.0; native extension SHA-256 c1d02b410ef729618077ba287f01f05b229f31d118a7aae04a1b7e56142e3b2e
- Weight placement
- gpu-resident
- Observed devices
- CUDA0
- Requested devices
- CUDA0
- Split mode
- Single GPU — no splitting
- Placement detail
- One physical RTX 3090; target model and native MTP compute tensors were observed on CUDA0, with no CPU expert offload or model split.
- KV-cache precision
- EXL native K8/V8 with QSA side planes and recurrent state
- Requested context per slot
- 8,960 tokens
- Actual context per slot
- 8,960 tokens
- Native model context
- 262,144 tokens
- Prompt tokens
- 512
- Generated tokens
- 256
- Speculative decoding
- MTP · draft max 1
- Warm-ups
- 1
- Measured repetitions
- 3
- Timing method
- ExLlamaV3 persistent worker time.perf_counter_ns with torch.cuda.synchronize at phase boundaries; fresh PP excludes prefix preparation; aggregate TG is across streams
- Throughput spreads
- KV 8,192, 7 streams: PP mean 922.17 t/s (min 921.50, max 923.35, SD 1.03, n=3); TG mean 237.66 t/s (min 237.60, max 237.76, SD 0.09, n=3); All mean 470.47 t/s (min 470.32, max 470.61, SD 0.14, n=3)
- Measurement outcomes
- 8,192 tokens · 7 streams: measured
- Notes
- 1× RTX3090 — no splitting · EXL3 4.00bpw · MTP-1 · concurrency 7, 8192-token prefix, 8960-token allocation per stream. Persistent EXL native worker: 512 fresh prompt tokens and 256 generated tokens; one warmup then three measured repetitions. PP is fresh-prompt processing after the populated prefix; TG is aggregate decode across concurrent streams. Throughput means and sample standard deviations are recomputed from the retained raw repetitions. EXL native K8/V8 cache storage includes QSA side planes and recurrent state; this is not GGUF Q8_0. Physical GPU 0 (GPU-7aa53e24-dc9d-bf8d-96cc-ef8c05c66dab) was selected as the single device. Runtime evidence maps that physical GPU to one logical CUDA0. Source configuration 400bpw-concurrency-mtp-1-c7; model revision 113cf7ab958054860e43fb7f3063b1af19171095; source manifest SHA-256 f4213ce466cd62992d252fcd14c623cb23cee191ed865703639e884f2df15d01. The concurrency source used an 8192-token prefix and a fixed 8960-token per-stream cache allocation; this allocation is independent of context-fit capacities.
- Source article
- /benchmarks/
- Raw log path
research/benchmarks/logs/trpro/2026-10-15-independent-gpus/remaining-campaign/trpro-exl-qwen38-27b-4bpw-gpu0-independent-gpus-validation-20261015/4.00bpw/concurrency/mtp-1-c7
Source 8 · 2026-10-15
- Source run ID
- trpro-exl-qwen38-27b-4bpw-gpu0-4bpw-mtp-1-concurrency-8-20261015
- Page entry ID
- trpro-exl-qwen38-27b-4bpw-gpu0-mtp-1
- Entry sweep
- concurrency
- Raw entry label
- Qwen3.8 27B EXL3 4.00bpw · MTP-1
- Configuration label
- 1× RTX3090 — no splitting · EXL3 4.00bpw · MTP-1 · concurrency 8, 8192-token prefix, 8960-token allocation per stream
- Measured date
- 2026-10-15
- Hardware
- 1× RTX 3090
- Runtime
- exllamav3 · ExLlamaV3 1.5.0+cu128.torch2.9.0; native extension SHA-256 c1d02b410ef729618077ba287f01f05b229f31d118a7aae04a1b7e56142e3b2e
- Weight placement
- gpu-resident
- Observed devices
- CUDA0
- Requested devices
- CUDA0
- Split mode
- Single GPU — no splitting
- Placement detail
- One physical RTX 3090; target model and native MTP compute tensors were observed on CUDA0, with no CPU expert offload or model split.
- KV-cache precision
- EXL native K8/V8 with QSA side planes and recurrent state
- Requested context per slot
- 8,960 tokens
- Actual context per slot
- 8,960 tokens
- Native model context
- 262,144 tokens
- Prompt tokens
- 512
- Generated tokens
- 256
- Speculative decoding
- MTP · draft max 1
- Warm-ups
- 1
- Measured repetitions
- 3
- Timing method
- ExLlamaV3 persistent worker time.perf_counter_ns with torch.cuda.synchronize at phase boundaries; fresh PP excludes prefix preparation; aggregate TG is across streams
- Throughput spreads
- KV 8,192, 8 streams: PP mean 921.32 t/s (min 919.84, max 922.73, SD 1.45, n=3); TG mean 260.50 t/s (min 260.03, max 260.74, SD 0.40, n=3); All mean 499.20 t/s (min 498.90, max 499.48, SD 0.29, n=3)
- Measurement outcomes
- 8,192 tokens · 8 streams: measured
- Notes
- 1× RTX3090 — no splitting · EXL3 4.00bpw · MTP-1 · concurrency 8, 8192-token prefix, 8960-token allocation per stream. Persistent EXL native worker: 512 fresh prompt tokens and 256 generated tokens; one warmup then three measured repetitions. PP is fresh-prompt processing after the populated prefix; TG is aggregate decode across concurrent streams. Throughput means and sample standard deviations are recomputed from the retained raw repetitions. EXL native K8/V8 cache storage includes QSA side planes and recurrent state; this is not GGUF Q8_0. Physical GPU 0 (GPU-7aa53e24-dc9d-bf8d-96cc-ef8c05c66dab) was selected as the single device. Runtime evidence maps that physical GPU to one logical CUDA0. Source configuration 400bpw-concurrency-mtp-1-c8; model revision 113cf7ab958054860e43fb7f3063b1af19171095; source manifest SHA-256 f4213ce466cd62992d252fcd14c623cb23cee191ed865703639e884f2df15d01. The concurrency source used an 8192-token prefix and a fixed 8960-token per-stream cache allocation; this allocation is independent of context-fit capacities.
- Source article
- /benchmarks/
- Raw log path
research/benchmarks/logs/trpro/2026-10-15-independent-gpus/remaining-campaign/trpro-exl-qwen38-27b-4bpw-gpu0-independent-gpus-validation-20261015/4.00bpw/concurrency/mtp-1-c8