Kv-Cache on Jaehun's Blog

SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking

Tue, 15 Sep 2026 00:00:00 +0900

Paper

SAS: A Simple Attention Sparsification That Learns Context Ranking Directly Without Distillation

TL;DR — In post-training attention sparsification, inject the continuous scores produced by the selector into the attention logits as a log gate. This bypasses the gradients blocked by discrete Top-K selection, allowing the selector to be trained end-to-end with the language modeling loss, without auxiliary distillation. As a result, at a tight budget (1024 tokens), it lifts GPQA-Diamond by up to +15.5 points over SeerAttention-R on Qwen3-4B/8B/14B, and cuts decoding latency by up to 5.6× at 512K context.

VestigeKV: The NoPE-MLA KV Cache Carries Its Own Eviction Signal in a Vestigial Branch

Wed, 09 Sep 2026 00:00:00 +0900

Paper

VestigeKV: The Degenerate RoPE Branch Carries Its Own Cache Eviction Signal

TL;DR

In NoPE-MLA models (Kimi Linear 48B), reading only 11% of each row — the 64-dim decoupled branch that training has repurposed as a salience channel, a “vestige” of RoPE — yields a query-independent importance signal and achieves 8–32× (up to 128× with the retrieval tier) KV cache compression while maintaining retrieval performance. It is a pure cache policy that never touches weights, kernels, or compute (source: Abstract).

ShallowStream: Index Shallow then Answer Deep for Streaming Video Understanding

Tue, 08 Sep 2026 00:00:00 +0900

Paper

ShallowStream: Index Shallowly, Answer Deeply — Unlocking the Computational Bottleneck of Streaming Video Understanding

TL;DR — In streaming video understanding, the most expensive cost is prefilling every frame through the MLLM’s full depth (28–32 layers). ShallowStream builds frame encoding and a retrieval index with only 4–5 shallow layers, and runs full-depth computation on the retrieved evidence only at the moment a question arrives. As a result, it keeps SOTA-level performance (OVO-Bench 69.5 / StreamingBench 78.2) while cutting per-frame prefill by up to 52.1× and 10-second end-to-end latency by up to 11.9× (source: Abstract).

BeaconKV: Key-Value Cache Compression Guided by Beacon Queries for Efficient Large Reasoning Model Inference

Mon, 07 Sep 2026 00:00:00 +0900

Paper

Catch the Moment a Model “Revisits Its Thoughts”: How BeaconKV Compresses Reasoning KV Caches 5.8× with Beacon Queries

TL;DR — The KV cache of a Large Reasoning Model (LRM) grows linearly as it produces longer Chains-of-Thought (CoT). When Qwen3-4B generates 32K tokens at batch size 16, the KV cache alone exceeds 77 GB, approaching the 80 GB GPU limit (source: §1). Existing compression methods predict future important tokens from recent queries, but this assumption breaks down during reasoning because of a phenomenon called Thought Revisiting Token (TRT), in which the model re-reads distant past context (problem definition, solution plan) (source: §3). Building on the geometric insight that the global queries behind TRT form a few clusters in embedding space, BeaconKV selects a beacon query to represent each cluster online via Continual FPS and compresses the KV cache accordingly. This achieves up to 5.8× memory savings (77.0 → 13.3 GB) and 4.3× throughput gains (82.3 → 356.4 tokens/s), while improving accuracy by up to 31.7 pp over RPC and R-KV (source: §5.4, Fig. 8).

Language Models Can Control Their Own Attention

Mon, 07 Sep 2026 00:00:00 +0900

Paper

Language Models Control Their Own Attention: Declarative Attention

TL;DR — Existing sparse attention methods still paid $O(N)$ per decoding step by approximating externally which tokens matter. This paper flips the direction: it lets the model “declare” directly, inside Chain-of-Thought (CoT), where it will look. The inference engine parses this declaration like a tool call and skips most KV cache reads. Without any training (zero-shot), Gemma-4-31B cuts decoding attention cost by 52.0% with only a 1.27pp accuracy drop (source: §1, §5.1).

Same Request, Different Answer: Quantization Amplifies Cache-Induced Divergence in LLM Serving

Mon, 07 Sep 2026 00:00:00 +0900

Paper

Same Request, Different Answer: Prefix Caching Makes Serving Non-Reproducible, and Quantization Amplifies It

TL;DR — Prefix caching is billed as a “transparent optimization,” but it is not transparent. With caching off, repeated runs of the same workload are bit-identical across all 10 configurations × 80 episodes = 800 runs (source: Tab. 2); with caching on, agent trajectories change in 36.2% of episodes at 16 bits and 75.0% under 4-bit quantization (source: §IV-D). Yet average accuracy does not move — this is not “degradation” but “instability,” and the two call for different responses (source: §IV-F, §V).

SGD-KV: Summarization Guided KV Cache Compression

Mon, 07 Sep 2026 00:00:00 +0900

Paper

SGD-KV: Finding ‘heads that are good at summarizing’ cuts the 1M-token KV cache by up to 75%

TL;DR — Attention heads do not all do the same job. SGD-KV pinpoints the summarization heads — the ones responsible for the high-level information compression called ‘summarization’ — with a diagnostic task, then hands out the KV cache budget in proportion to their scores. At contexts of up to 1 million (1M) tokens it cuts KV cache memory by up to 75% while beating existing head-level compression methods.

Random Attention: Rethinking KV Cache Eviction for Efficient Reasoning

Sun, 06 Sep 2026 00:00:00 +0900

Paper

The Scores Were a Waste: The Paradox Where ‘Random Sampling’ Ties the Strongest in KV Cache Eviction — An In-Depth Review of Random Attention

TL;DR — Research on reasoning-specific KV cache eviction has always been the problem of scoring “which tokens will matter later” and picking the top-K. Random Attention forcibly preserves only the prompt with +∞ and evicts the rest without any scores, uniformly at random per head. Across 4 models and 6 reasoning tasks it matches the strongest baseline (TriAttention) in accuracy (statistical advantage in 31 of 60 comparison cells, significantly worse in just 1), and since vLLM serving has no scoring pass it delivers 32–43% higher throughput (source: §1, Fig. 1, Tab. 1, Tab. 4).