Fundamentals / September 8, 2026 / 5 min read

Long-Context and Attention in LLMs Explained (RoPE, YaRN)

Why scaling position embeddings lets models handle 128K or 256K tokens, and why real-world reasoning degrades much sooner.

On this page
  1. The Three Numbers: Trained, Advertised, and Reliable Context
  2. Why Models Need Word Order: The Position Problem
  3. RoPE and the Clock-Hand Analogy
  4. Why Pushing Past Training Causes Quality Loss
  5. How YaRN Makes Clock Hands Turn Appropriately
  6. Why Models Still Soften Before the Limit
  7. Firsthand Observation: Xiaomi MiMo V2.5 at 32K
  8. Practical Advice for Local Inference
  9. See Also and Sources

Modern language models routinely advertise massive context windows: 128K, 256K, or even one million tokens.

For local users running models in tools like llama.cpp, Ollama, or vLLM, those numbers make it tempting to paste an entire codebase, multiple book chapters, or months of chat logs directly into a single prompt.

In practice, there is a big difference between an engine ingesting 256,000 tokens without running out of memory and the model reliably following instructions across that entire span.

Understanding why this gap exists comes down to how transformers track word order using Rotary Position Embeddings (RoPE), why stretching context breaks down, and how clever techniques like YaRN help keep things on track.

The Three Numbers: Trained, Advertised, and Reliable Context

When looking at any long-context model, you need to separate three numbers that often get lumped together:

  1. Trained context: The actual sequence length the model saw during training. Models are trained on specific chunk sizes, and they see far fewer examples at their maximum lengths than at shorter lengths.
  2. Advertised (or supported) context: The maximum sequence length allowed by the model configuration and runtime engine. This is what the software lets you input before refusing or throwing an error.
  3. Practically reliable context: The sequence length where the model still reliably follows instructions, preserves formatting, performs complex reasoning, and retrieves facts without hallucinating.

The advertised context tells you what the engine accepts without crashing, not that output quality remains identical.

Meta’s Llama 3 report is an example. Llama 3 was originally trained on 8K sequences. Meta then ran roughly 800 billion additional training tokens across multiple stages (16K, 32K, 64K, and 128K) to teach the model how to use that expanded space. Adjusting position settings allowed the model to accept the larger window, but extensive training was what actually taught it to reason there.

Why Models Need Word Order: The Position Problem

Under the hood, standard transformer attention has no built-in sense of order. If you feed words into an attention layer without positional information, it processes them like an unordered bag of words. To the raw math, “dog bites man” and “man bites dog” look identical.

To make sense of language, a model needs to know two things:

  • Where a word sits in the sentence.
  • More importantly, how far apart words are from each other (relative distance).

Knowing that two words are right next to each other helps the model understand grammar and phrasing, while knowing they are thousands of words apart helps it manage big-picture context.

RoPE and the Clock-Hand Analogy

Introduced by Jianlin Su et al. in the RoFormer paper (2021), Rotary Position Embedding (RoPE) has become the standard positional method for modern models like Llama, Mistral, Qwen, and MiMo.

Instead of stamping a rigid position number onto each word, RoPE represents position using hands on a clock face.

Imagine that every token has a set of clock hands. As you move forward through a prompt from token 0 to token 10,000, those hands rotate forward:

  • When the model compares two words, it looks at the angle between their clock hands.
  • If two words are right next to each other (like positions 4 and 5, or positions 5,000 and 5,001), the difference between their clock hands is identical.
  • This gives the model a natural sense of relative distance anywhere in the text.

Even better, RoPE uses dozens of hands spinning at different speeds:

  • Fast-spinning hands (second hands) complete full rotations over just a few tokens. They handle local relationships like syntax, punctuation, and adjacent words.
  • Slow-turning hands (hour hands) rotate very gradually across thousands of tokens. They track global structure and long-range connections.

Why Pushing Past Training Causes Quality Loss

What happens if you take a model trained on 8,000 tokens and suddenly feed it 32,000 tokens?

For the fast-spinning second hands, nothing feels unfamiliar. They have already spun around hundreds of times during training, so continuing to spin around is normal.

The slow-turning hour hands, however, run into trouble. During an 8,000-token training run, the slowest hands might have only rotated through a tiny slice of the clock face—say, from 12 o’clock to 2 o’clock.

When you push the sequence to 32,000 tokens, those slow hands swing past 2 o’clock and point toward 5 o’clock or 8 o’clock. The model has never encountered those angles during training.

Because these angles are completely unfamiliar, the model’s internal attention scores get distorted. It loses track of relative distances, output quality degrades rapidly, and the model begins producing rambling text or losing the thread entirely.

How YaRN Makes Clock Hands Turn Appropriately

To prevent clock hands from wandering into unfamiliar territory, researchers developed context scaling techniques:

The Early Fix: Position Interpolation

In 2023, Meta introduced Position Interpolation (PI). The concept was straightforward: if you want to double the context from 4,000 to 8,000 tokens, simply slow down every clock hand by half. Token 8,000 now looks like token 4,000.

This stopped the model from crashing, but it introduced a new issue: slowing down the fast second hands compressed nearby word distances. Adjacent words that used to be distinct started blurring together, which hurt fine-grained grammar and local comprehension.

The Smarter Fix: YaRN

To solve this trade-off, Bowen Peng and collaborators introduced YaRN (Yet another RoPE extensioN) in late 2023. YaRN takes a selective approach:

  • Leave fast hands alone: It does not slow down the rapidly spinning hands. This preserves sharp discrimination between adjacent words and keeps local grammar intact.
  • Slow down the slow hands: It applies interpolation only to the slow-moving hour hands, keeping them within the familiar angles seen during training.
  • Smoothly blend the middle hands: Hands between the fastest and slowest transition gradually.
  • Adjust attention focus: YaRN also gently recalibrates the model’s focus so attention doesn’t get diluted or “washed out” when searching across tens of thousands of tokens at once.

With YaRN, models can comfortably extend their context windows with only a tiny fraction of the fine-tuning data previously required.

Why Models Still Soften Before the Limit

Even with YaRN, RoPE adjustments, or extended post-training, local models often show softer output long before reaching their advertised maximum:

  • Attention dilution: Searching for a single relevant detail among 64,000 background tokens is far harder than finding it among 2,000 tokens. Background distractors naturally pull at the model’s attention.
  • Instruction drift: System instructions placed at the very start of a prompt receive less relative focus as thousands of new tokens accumulate between the prompt and the generation point.
  • Statistical tracking limits: Transformers do not have rigid counter registers. Keeping track of step numbers, list items, or strict JSON structures over 30,000 tokens relies on statistical patterns, and small probabilities compound into dropped constraints.

Firsthand Observation: Xiaomi MiMo V2.5 at 32K

The gap between theoretical context support and real-world behavior is easy to spot when testing models locally.

In my own testing with Xiaomi’s MiMo V2.5—a 310B-total, 15B-active Mixture-of-Experts model—there is a noticeable difference between accepted context and dependable quality.

Xiaomi’s model card notes that the context was progressively extended during post-training from 32K to 256K and then 1M. MiMo received legitimate long-context training, yet practical quality still shifts as prompts grow.

In interactive local sessions, MiMo V2.5 demonstrates sharp reasoning, strict instruction following, and clean formatting up to roughly 16K to 32K tokens. As active context crosses 32K tokens, I consistently observe a qualitative shift:

  • Inconsistent chain-of-thought numbering: Multi-step reasoning steps begin skipping, repeating numbers, or collapsing into unstructured paragraphs.
  • Formatting drift within COT: Headers and body start to blend together. Formatted step by step instructions start becoming paragraphs.
  • Softened instruction adherence: Negative constraints stated earlier in the prompt are more easily overlooked.

The model does not crash or generate incoherent gibberish. Instead, the crisp precision visible at shorter contexts softens into looser, less disciplined output.

The takeaway: never assume a model’s advertised maximum equals uniform quality across the entire window.

Practical Advice for Local Inference

If you run long-context models locally using llama.cpp, Ollama, or vLLM, keep these operational rules in mind:

1. KV Cache VRAM Grows Linearly

The Key-Value (KV) cache stores past token representations so the model does not have to recompute the entire prompt on every new token.

This cache grows directly with sequence length. Depending on the model’s architecture and precision format, running a 64K or 128K context at higher precisions can easily demand 10 to 20 GB of VRAM just for the cache, before counting the model weights themselves.

Use my Tokens-per-Second and Memory Fit Calculator to check whether your target model and context will fit in your available VRAM.

2. Quantize the KV Cache

If you run out of memory when expanding context, quantizing your KV cache is often much better than heavily quantizing your model weights.

In llama.cpp, running with --cache-type-k q8_0 --cache-type-v q8_0 cuts KV-cache memory compared to FP16 while usually preserving more precision than lower-bit cache formats.

For a broader look at precision formats, see my beginner’s guide to LLM quantization.

3. Do Not Rely on Needle-in-a-Haystack Benchmarks

Synthetic “needle-in-a-haystack” (NIAH) tests insert a phrase like “The secret passcode is 94821” into 100,000 tokens of text and ask the model to retrieve it.

Passing a needle test shows that the model can retrieve a distinctive string in that specific test setup, rather than proving a broad capability. It does not mean the model can synthesize complex arguments, follow negative constraints, or maintain disciplined chain-of-thought reasoning over that same distance.

4. Test Your Model’s Effective Horizon

Before building an automated pipeline or agent workflow that dumps tens of thousands of tokens into an LLM, test it empirically:

  1. Use a prompt representing your real workload (a slice of your codebase, multiple documents, or structured instructions).
  2. Test outputs at 8K, 16K, 32K, and 64K tokens.
  3. Identify where instruction following and output formatting begin to soften.

Configuring your runtime context to the model’s reliable horizon—rather than its advertised maximum—saves VRAM, speeds up prompt processing, and gives you far more consistent results.

See Also and Sources

Primary Research Papers

About the author

Harris Oldroyd

Independent self-taught builder and researcher

I learn systems from first principles, build them, and measure them before writing about them. The notebook covers local AI hardware and inference, systematic trading research, and the software that keeps both repeatable.

Related articles