Qwen 3.8 Flash Explained: The 51B N-Gram Memory Architecture

Qwen3.8-Flash-Next is an interesting model.

The checkpoint contains roughly 176 billion parameters across more than 100 GB of files and only about 6 billion active parameters:

• 125B MoE Backbone: Routes tokens through sparse expert layers where only ~6B parameters compute per step.
• 51B N-Gram Embedding Table: An auxiliary, 20-million-row learned table storing short multi-token patterns.
• 4B Multi-Token Prediction (MTP) Head: A speculative draft head intended to propose future tokens in parallel.

Released as an architectural preview for the future Qwen4 family, the innovation is the 51B n-gram embedding table. Understanding why it exists, how it works, and how runtimes handle its footprint requires starting from first principles.

From First Principles: What Is a Token Embedding?

Language models do not understand text directly; they operate on numbers.

When you feed a prompt into an LLM, a tokenizer chops text into numerical identifiers called token IDs. Words like " San" and " Francisco" each receive a distinct integer ID.

Because neural networks cannot perform arithmetic on raw integer IDs, the model passes each ID into an input embedding table: a dictionary matrix (Vocabulary Size × Hidden Dimension).

For a model with 150,000 vocabulary tokens and hidden dimension 2,560, the table contains 150,000 rows. When token A enters, the model retrieves row A: a continuous vector of 2,560 floating-point coordinates.

Standard embeddings are strictly unigrams (isolated single tokens). The vector for " San" enters without local context, unaware of whether the text refers to San Francisco, San Diego, or a sanitation worker.

Bigrams, Trigrams, and the Cost of Transformer Compute

An n-gram is simply a contiguous sequence of n tokens:

• Unigram (n=1): A single token (" San").
• Bigram (n=2): A two-token sequence (" San" + " Francisco", " machine" + " learning").
• Trigram (n=3): A three-token sequence (" New" + " York" + " City", " import" + " numpy" + " as").

In conventional architectures, discovering that " San" and " Francisco" form a single concept requires computation. Unigram vectors pass through self-attention and feed-forward matrix multiplications (quadratic in hidden dimension, O(d²)) before the network recognizes the unified entity.

Language is packed with predictable local patterns: compound names (“Hong Kong”), idioms (“on the other hand”), and boilerplate (“if name == ‘main’:”). Standard transformers burn compute repeatedly reconstructing the obvious meaning of common pairs.

Qwen3.8-Flash-Next’s design is simple: move common local pattern knowledge into cheap memory lookups rather than repeatedly reconstructing it through transformer matrix multiplication. Looking up a pre-learned vector requires negligible compute compared to running matrix multiplications through dozens of neural layers.

Inside the 20-Million-Row Table: The Math Behind 51B Parameters

According to official documentation, the n-gram component contains 20,000,000 entries, each matching the model’s primary hidden width of 2,560:

That single lookup array accounts for 20,000,000 entries × 2,560 hidden dimension = 51,200,000,000 parameters

Combinatorial Scale, Hashing, and Collisions

Why 20 million entries? Storing every n-gram is impossible: a 150,000-token vocabulary yields 150,000² = 22.5 billion possible bigrams and over 3.38 × 10¹⁵ trigrams. Qwen maps this vast space into 20 million rows.

• Confirmed by Qwen: The official model card confirms a 20,000,000-entry table covering bigrams and trigrams located at layer 2.
• Observed in Implementations: Dissections by runtime developers (such as NVIDIA NeMo Automodel notes) report multi-hash indexing (8 bigram and 8 trigram hashes queried in parallel) and residual stream injection at layer 2.

Because possible n-grams vastly outnumber 20 million entries, hash collisions are mathematically inevitable; unrelated token tuples map to identical rows. While Qwen has not published a formal collision specification, empirical implementations treat shared rows cautiously. How training dynamics tolerate or mitigate collisions, we do not know.

Where the Vector Enters: Layer 2

The retrieved vector does not replace the unigram embedding at input. Official documentation places the lookup at Layer 2 (the second decoder block). In community implementations, Layers 0 and 1 establish baseline token identities and rotary position embeddings (RoPE), after which the retrieved vector is injected into the residual stream.

What the N-Gram Table Is—and What It Is Not

To avoid misunderstandings:

• It IS learned numeric memory: Each row contains 2,560 floating-point weights trained via gradient descent, capturing learned local token-pattern features and statistical associations rather than human-readable facts.
• It is NOT readable text storage: You cannot open the table and read text strings; it contains continuous numerical vectors.
• It is NOT prompt caching: Prompt caching preserves dynamic KV states from past prompts across sessions. The n-gram table is static weight data.
• It is NOT web retrieval or RAG: The table cannot search external sources or incorporate new facts after training.
• It is NOT an attention replacement: An n-gram lookup only inspects the previous one or two tokens, with no awareness of long-range syntax. Global reasoning still relies entirely on attention layers.
• It is NOT a complete reasoning system: The table provides associative recall of local patterns, offloading low-level pattern recognition so the active 6B backbone can focus on reasoning.

Memory Traffic, System RAM Offload, and Asynchronous Prefetch

Why doesn’t a 51B table cripple inference speeds? Because an n-gram lookup is an address-based memory read, not a matrix multiplication.

The PCIe Bottleneck vs. Sparse Lookups

Standard inference is bottlenecked by memory bandwidth: each token streams weights into GPU. Offloading dense layers to system RAM over PCIe (31.5 GB/s on PCIe 4.0 vs. 900–2,000 GB/s in VRAM) drops generation speed massively.

An n-gram lookup reads only specific rows selected by active hashes. In community implementations, generating a token reads 16 rows (8 bigram + 8 trigram hashes) of 2,560 FP16 values (~5 KB each), totaling ~82 KB per token. Moving 82 KB over PCIe 4.0 takes roughly 2.5 microseconds.

Asynchronous Prefetching

Furthermore, runtimes can execute this lookup asynchronously. When token t is sampled, the CPU knows the latest token sequence. While the GPU computes Layers 0 and 1, the CPU initiates a transfer over PCIe.

By Layer 2, the retrieved vectors are already in VRAM, hiding transfer latency behind GPU computation. This allows the 51B table to fit in consumer system memory rather than demanding high-bandwidth VRAM.

Checkpoint Architecture and Local Hardware Realities

Here is how the complete Qwen3.8-Flash-Next checkpoint is structured:

Component	Total Parameters	Active Compute per Token	Recommended Placement
MoE Backbone	~125 Billion	~6 Billion (sparse routing)	Split VRAM + Host RAM
N-Gram Table	~51.2 Billion	~82 KB lookup (impl-specific)	Host System RAM (DDR5 via PCIe prefetch)
MTP Head	~4 Billion	Speculative verification	GPU VRAM
Total Checkpoint	~176B–180B	~6 Billion (dense decode)	Hybrid VRAM + Host RAM

Storage, Memory Footprint, and Expert Offloading

Even with low active compute parameters, storage requirements remain substantial:

• At 16-bit precision, the checkpoint exceeds 350 GB.
• Quantized to 4-bit precision (GGUF Q4_K_M or EXL2), the package requires roughly 100 to 110 GB of NVMe storage.

Running Qwen3.8-Flash-Next on a 48 GB GPU setup therefore requires a double offload:

1. Host System RAM: At least 64 GB to 128 GB of RAM is needed to park the quantized 51B n-gram table (~25–35 GB) plus MoE expert weights that exceed VRAM capacity.
2. GPU VRAM: 24 GB to 48 GB holds active non-expert layers, remaining expert weights, and the KV cache.

Firsthand Experience: From Early llama.cpp to Working MTP

My testbed is an open-frame workstation with two 24 GB NVIDIA RTX 3090 GPUs (48 GB VRAM) and 128 GB RAM on Threadripper PRO 3945WX. For hardware details, see my dual RTX 3090 workstation guide.

My first experience, shortly after architecture support arrived in llama.cpp, was approximately 23 tokens per second without MTP. In that tested build, enabling the draft head produced assertion errors or corrupted text. That was the experience behind this article’s original recommendation to stick with Qwen 3.8 27B—not a permanent verdict on Flash-Next or every runtime.

September 27 update: I now have working native MTP in ExLlamaV3, with approximately 100 tok/s at a 65,536-token existing prefix in one configuration, and 81.53 tok/s near the boundary of a fixed 262,144-token total budget in another.

Those are not endpoints of a controlled “23 to 100” speedup test. The original llama.cpp experience, early EXL pilots, held-out coding tests, and latest context sweeps differ in engine, quantization, prompt shape, timing boundaries, and placement. Historical llama-bench kernel timings also must not be treated as equivalent to the EXL worker’s fresh-prompt/generation wall timings.

The Goal Was to Match a Recipe, Not Invent a New Engine

The starting point was the community’s dual-3090 Flash-Next recipe, using a patched vLLM runtime and a W4A16/FP8-PLE checkpoint. Its September 25 release reports 104.5 tok/s after a 131,072-token prompt and up to 103.1 tok/s for 260,096 input plus 2,048 output tokens. Those headline values are best-of-three external results, not measurements from my machine.

I did run its standard, non-P2P path locally: a quick, single measured 8,192-input/256-output check gave an 82.51 tok/s API-observed decode estimate. That was a useful reference, not an apples-to-apples engine ranking: the quantization, cache format, and HTTP/SSE timing boundary differ from EXL.

I also investigated other approaches. Cinference was interesting for CUDA-graph scheduling and block drafting, but that pinned implementation requires Blackwell sm_120a, not the 3090’s Ampere architecture. It was a source investigation, not a benchmark on this workstation. I did not need to port its DFlash2 machinery to get the results below.

The Useful Optimization: Keep the Right Experts on GPU

The first retained successful EXL pilot, with whole-layer expert offload and MTP disabled, produced 31.82 tok/s at zero existing prefix and 32.17 tok/s at 8K, using fresh PP512/TG256 requests. Earlier, smaller-offload fit attempts ran out of VRAM.

The eventual approach was more selective: keep the n-gram table in RAM, split each target MoE layer’s 512 experts between CPU and GPU, and use a frozen routing profile to choose which experts stay on GPU. The model still selects ten experts per token per layer; I did not prune experts or reduce routing top-k. “Hot” means frequently selected, not a smaller model.

This is static placement, not continuous expert swapping. During decode, EXL computes the cold experts on CPU and combines their results with GPU work. That differs from the published vLLM recipe’s GPU expert cache, which copies missing expert weights from RAM to GPU for execution. Large-batch prefill can also take a different execution path from decode.

Profiling made that distinction matter. In a corrected CUDA-event pass, exposed CPU-result collection/wait occupied about 1,321 ms of 3,591 ms of generation. Explicit copies accounted for only about 3.19 ms in a separate, heavier trace. These overlapping, separately measured spans are not additive, and neither proves a pure PCIe-bandwidth bottleneck. The practical lead was reducing CPU expert work and the GPU’s wait for it.

The strongest matched result came from training placement on two separate coding prompts, each with 8,192 input tokens and 512 generated tokens. I counted only target routing during decode verification—not prompt processing, draft routing, or warm-up—and normalized each prompt/layer before giving the two prompts equal weight. Rejected speculative verification positions still count as work. The profile was frozen before evaluation on a separate held-out coding prompt.

On that held-out PP8192/TG256 test, with one warm-up and three measured repetitions per profile, decode rose from 67.93 to 100.83 tok/s (+48.44%). The CPU share of selected target experts fell from 15.44% to 3.64%; fresh-prompt-plus-generation latency fell from 9.46 to 8.07 seconds. This isolates a placement-profile change much better than comparing the first llama.cpp result with the latest EXL number. It is still one held-out coding workload, not a universal speed or quality result.

A Small Draft, Entirely on GPU

The final configuration uses fixed MTP2: two proposed tokens per speculative round, with the target verifying them. All 512 draft experts, the draft’s own embedding, and its compute/cache stay on the second GPU. The target still uses its RAM-backed components.

More drafting was not automatically faster. Directional single-sample checks gave 71.42 tok/s for full-GPU MTP2 versus 64.86 for full-GPU MTP3, before the decode-trained profile. Changing draft depth also changed one target layer’s native autosplit placement, so that comparison is not a perfectly isolated draft-depth experiment. The final sweeps keep MTP2 fixed.

CPU tuning mattered too: testing 8, 12, 16, and 24 worker threads did not reward simply using more threads. The retained configuration is 12 threads, four work slots, and 8,192-token prefill chunks, with dynamic expert swapping and dynamic draft length disabled.

Long Context: Two Different Memory Budgets

Both latest cohorts use the same EXL3 checkpoint, EXL K8/V8 cache with QSA/recurrent state (not llama.cpp Q8_0), one stream, 512 fresh prompt tokens, and 256 generated tokens. Each depth has one warm-up and three measured repetitions.

Existing prefix	256 CPU / 256 GPU	288 CPU / 224 GPU
0	95.67 tok/s	87.11 tok/s
2,048	101.93 tok/s	92.23 tok/s
4,096	97.72 tok/s	89.28 tok/s
8,192	99.62 tok/s	89.32 tok/s
16,384	101.65 tok/s	95.46 tok/s
32,768	100.30 tok/s	93.14 tok/s
65,536	100.57 tok/s	90.39 tok/s
131,072	Not measured	89.90 tok/s
261,376	Not measured	81.53 tok/s

The earlier reload-per-depth cohort establishes the roughly 100 tok/s result at 64K. However, cache allocation grew with depth, and the native loader moved target layer 25 from GPU 0 to GPU 1 at 16K. It is not a fixed-allocation context-scaling experiment.

For the persistent fixed-capacity cohort, I first tried allocating the full 262,144-token budget with the 256/256 target split. Native autosplit could not fit the target, before even loading the draft. Moving 32 more target experts per layer to CPU made room, while leaving the entire draft on GPU.

That second cohort loads the model once, keeping the same process, model/cache objects, expert placement, GPU layer placement, and target/draft capacity across all 36 warm-up/measured iterations. Allocated GPU cache tensors remain approximately 4.789 GiB. Each iteration resets state and rebuilds the prefix; it does not reload weights or reuse a previous answer’s cache. Vision was never loaded—not “offloaded to CPU.”

The exact endpoint matters: 261,376 existing tokens + 512 fresh prompt tokens + 256 generated tokens = 262,144 total. The 81.53 tok/s result is not generation after a full 262,144-token existing prefix. Nor can I join the earlier 100.57 tok/s point to it and call the difference pure context slowdown: the target placement changed.

Prompt Processing Did Not Collapse from 1,500 to 250 Tok/s

There are two different input measurements:

• Bulk prefix prefill: building the existing context in chunks of up to 8,192 tokens.
• Fresh PP512: processing the next 512 tokens after that prefix, inside the measured request.

In the persistent run, bulk prefill averaged 1,436.73 tok/s at 8K, 1,434.64 at 64K, 1,419.74 at 128K, and 1,391.06 at the 261,376-token endpoint. The earlier reload cohort gave 1,480.44 at 8K and 1,476.22 at 64K. Those rates are calculated from each measured repetition’s prefix tokens and prefix-build time.

Fresh PP512 in the persistent cohort ranged from 226.16 to 338.65 tok/s, including 228.25 at 64K and 315.56 at the maximum prefix. It is not the same workload as bulk prefill; comparing those columns as if they were one before/after rate would invent a slowdown.

At the maximum prefix, rebuilding context took 187.90 seconds; prefix-to-completion took 192.66 seconds, excluding model loading and cache reset. Fast decode does not make the first long prompt instantaneous.

The corpus content also changes with depth, which can change routing and speculative acceptance. This is a realistic workload/context sweep, not an isolated attention-length microbenchmark. The generated samples were coherent, but that is a sanity check—not evidence that long-context reasoning quality is unchanged.

Downloads and Runtime Resources

Use these resources to test Qwen3.8-Flash-Next locally:

• Official Model Card: Qwen/Qwen3.8-Flash-Next on Hugging Face — Source weights, architecture metadata, and tokenizer files.
• Official Model Collection: Qwen 3.8 Flash Next Collection — Official preview repository collection.
• Quantized GGUF Builds: Community releases like LM Studio Community GGUFs provide quantized splits ready for llama.cpp.
• Related Local AI Guides:
  • Qwen 3.8 27B: The Practical Sweet Spot for Local AI
  • Speculative Decoding and MTP Explained
  • Beginner’s Guide to LLM Quantization
  • Best GPUs and Devices for Local AI
  • RoPE and YaRN: Long Context Explained

The Verdict on Qwen3.8-Flash-Next

Qwen3.8-Flash-Next demonstrates how future architectures can decouple memory capacity from compute scaling.

By offloading common local n-gram patterns into a 20-million-row learned table, the architecture expands local pattern capacity without proportional increases in GPU compute. The table is not readable text, prompt caching, or an attention replacement; it is an associative memory accelerator that frees the active 6B backbone to focus on high-level reasoning.

The practical lesson changed as the runtime experiments improved. The 51B table was not a reason to abandon this model, and getting useful speed did not require a new speculative architecture: RAM-backed n-grams, a decode-trained static hot-expert placement, and a fully GPU-resident MTP2 draft were enough on this workstation.

Qwen 3.8 27B remains an attractive compact option, but my original “Flash-Next is stuck around 23 tok/s with broken MTP” experience is now historical. Flash-Next is a much more credible local option at roughly 100 tok/s in the 64K reload configuration, or 81.53 tok/s at the boundary of a fixed 262,144-token budget with more CPU experts. The choice is a memory-placement tradeoff—not a free, universal speedup, and not a quality ranking between the models.