Qwen3.8 Flash

Qwen/Qwen3.8-Flash-Next

6B active / 180B totalMoEText generationImage → textText inputImage inputn gram table

Qwen · 2 recorded configurations

Measured setup

Hardware

  • 2× RTX 3090 · CPU offload

Runtime and tool

  • ExLlamaV3 worker

Quantisation

  • EXL3 4.05bpw_h6_ng6

MTP draft depth

  • MTP 2

Recorded setup

Every option opens a recorded setup. Where the corpus recorded nothing at this page's other settings, the option names the fields it switches with it. Allocation and runtime detail stay in the technical record below.

2× RTX 3090 · ExLlamaV3

EXL3 4.05bpw_h6_ng6 · MTP 2

PP · left axisTG · right axis
View exact measurements
Exact context depth and measured fresh prefill, bulk prefix, and decode rates for Qwen3.8 Flash
Context depth (tokens)Bulk prefix PP (tok/s)Fresh prefill PP (tok/s)TG (tok/s)
0—390.6895.67
2,048773.84287.79101.93
4,0961,105.00332.1697.72
8,1921,480.44264.9799.62
16,3841,490.36320.63101.65
32,7681,485.79252.26100.30
65,5361,476.22253.72100.57
Technical record and provenance

Run details

Source run ID
rtx3090x2-qwen3.8-flash-exl3-mtp2-reload-2026-09-27
Configuration label
256 CPU / 256 GPU · reload per depth
Measured date
2026-09-27
Hardware
2× RTX 3090
Runtime
exllamav3 · ExLlamaV3 1.5.0+cu128.torch2.9.0; installed source/binary hashes in evidence
Weight placement
partially-offloaded
Placement detail
256 CPU / 256 GPU experts per target layer; static decode-trained profile. Full-GPU MTP2 draft on CUDA 1, all 512 draft experts; n-gram table in RAM.
KV-cache precision
EXL K8/V8 with QSA side planes and recurrent state
Prompt tokens
512
Generated tokens
256
Speculative decoding
MTP · draft max 2
Measured repetitions
3
Notes
Earlier reload-per-depth cohort, NOT a fixed-allocation context experiment. Target 256 CPU / 256 GPU experts per layer. Cache capacity varies with depth (8192 minimum loader probe); target layer 25 moves CUDA 0 to CUDA 1 from 16384 tokens. Seven measured rows only through 65536; do not join them to the later 288-CPU-expert curve. C1; existing prefix + fresh PP512 + TG256; one warm-up and three measured repetitions per depth, arithmetic mean of per-repetition rates. Bulk prefix is rebuilt in chunks up to 8192 tokens and timed separately: bulkPrefillTps = mean(prefix tokens * 1000 / prefixMs), not fresh PP512. Synchronized host timing includes final-position logits and the draft prompt cache in PP; TG includes first sampling from PP logits and native MTP draft/verify to exactly 256 outputs (EOS ignored). Load, reset, prefix build and serialization are excluded from fresh PP/TG; post-decode queue drain is excluded from TG. Frozen decode-trained expert profile SHA256 6ac003f37ca2dcb4e56b15fa72d0d5726e8722cd2a7e6ee3d3f73b0d406b2b34; static hot-expert placement, no dynamic swapping. Target has 48 routed layers, 512 experts per layer, top 10 selected per token; CPU threads 12, slots 4. Native MTP2 draft stays on CUDA 1 with all 512 experts and an independent GPU embedding. N-gram table in RAM (trellis_ram); vision was never loaded, not CPU-offloaded. EXL K8/V8 includes QSA side planes and recurrent state, not llama.cpp Q8_0. Model revision 55a732e0c4c3d4614bc42b68493bb930d9b02c0a. Corpus content changes with depth; not an isolated context-length microbenchmark or quality-equivalence proof. Native MTP stream-tap semantics remain an upstream caveat.
Source article
/local-ai/qwen3-8-flash-local-ai/
Raw log path
research/benchmarks/logs/trpro/2026-09-27-exl-context-sweep/publication-evidence.json

Compare measured entries