TL;DR — The KV cache of a Large Reasoning Model (LRM) grows linearly as it produces longer Chains-of-Thought (CoT). When Qwen3-4B generates 32K tokens at batch size 16, the KV cache alone exceeds 77 GB, approaching the 80 GB GPU limit (source: §1). Existing compression methods predict future important tokens from recent queries, but this assumption breaks down during reasoning because of a phenomenon called Thought Revisiting Token (TRT), in which the model re-reads distant past context (problem definition, solution plan) (source: §3). Building on the geometric insight that the global queries behind TRT form a few clusters in embedding space, BeaconKV selects a beacon query to represent each cluster online via Continual FPS and compresses the KV cache accordingly. This achieves up to 5.8× memory savings (77.0 → 13.3 GB) and 4.3× throughput gains (82.3 → 356.4 tokens/s), while improving accuracy by up to 31.7 pp over RPC and R-KV (source: §5.4, Fig. 8).

2609.04971v1 on Jaehun's Blog

BeaconKV: Key-Value Cache Compression Guided by Beacon Queries for Efficient Large Reasoning Model Inference

Catch the Moment a Model “Revisits Its Thoughts”: How BeaconKV Compresses Reasoning KV Caches 5.8× with Beacon Queries