[Paper Review] Attention Sinks and Compression Valleys in LLMs are Two Sides of the Same Coin

Thu, 23 Oct 2025 00:00:00 +0900

Paper

One Narrative Forged by Massive Activations: Bridging Attention Sink and Compression Valley

TL;DR

Massive activations in the residual stream (especially BOS) form a dominant singular value, which simultaneously induces mid-layer representation compression (entropy↓) and attention sink (attention drawn to BOS) in the same layer, and targeted MLP ablation provides causal confirmation of this. The optimal depth for generation, reasoning, and embedding is explained within a single Mix–Compress–Refine framework (source: §Abstract, §3.1–§3.3, Fig.1–4, §4–§5).

Activation Dynamics on Jaehun's Blog

[Paper Review] Attention Sinks and Compression Valleys in LLMs are Two Sides of the Same Coin

One Narrative Forged by Massive Activations: Bridging Attention Sink and Compression Valley

TL;DR