SlimKV: Joint Token-Feature KV Cache Compression with Reconstruction-Free Beacon Attention

Paper

SlimKV: Compressing Tokens and Features Together, Cutting the KV Cache by up to 32× with Reconstruction-Free Latent-Space Attention

TL;DR

SlimKV is a question-agnostic framework that compresses the KV cache — the bottleneck of long-context LLM inference — simultaneously along the token axis and the feature axis. The key idea is to remove RoPE only from the Keys of beacon memory tokens, so attention is performed directly in the latent space without a reconstruction step. It achieves the best performance at 16×/32× compression on LongBench (source: §5.2), and at a 128K token length it delivers up to 7.34× attention speedup and 3.38× end-to-end decoding speedup over no compression (source: §5.4).

Core Idea

The central claim of this paper can be summed up in one sentence.

The authors hypothesize that by training the beacon state in low-rank form from the start and removing RoPE only from the beacon Key, one can achieve 16–32× KV cache compression that simultaneously overcomes the limits of single-axis token/feature compression and the Key reconstruction latency bottleneck.

Three lines of contribution support this claim (source: §1).

1. Combined token·feature compression framework — It trains low-rank beacon KV representations within a fixed feature budget and allocates adaptive ranks to match per-layer heterogeneity.
2. Discovery of positional asymmetry — It empirically shows that removing RoPE on the Key side (K-RoPE) damages ordinary tokens and beacon tokens differently, with beacons being far less sensitive.
3. Reconstruction-free latent-space attention (LSA) — Using this asymmetry, it removes RoPE only from the beacon Key, thereby skipping the restoration to full dimensionality during decoding.

Key Numbers at a Glance

Background: The Problem They Solved

LLMs need long context for document understanding, multi-turn dialogue, and complex reasoning, but during inference the KV cache grows in proportion to context length, causing memory pressure and memory-access latency (source: §1). The KV cache size is roughly

where $L$ is the number of layers, $H$ is the number of KV heads, $d_{\text{head}}$ is the per-head dimension, seq is the sequence length, and batch is the batch size. The longer the context, the more this term dominates memory and bandwidth.

Existing compression strategies fall broadly into two axes, each with clear limits (source: §2).

• Token-wise: reduces the number of cache states. But explicit eviction loses a lot of information under aggressive compression (KVZip), and most eviction methods must first build the entire KV before eviction, which limits prefill compute savings (SnapKV, H2O). Soft compression (Activation Beacon) mitigates information loss but did not further explore the redundancy of the beacon state.
• Feature-wise: reduces the KV dimension per token (MQA, GQA, MLA, Eigen Attention, PALU, MHA2MLA). However, the compressed Key often has to be reconstructed to full dimensionality before RoPE is applied, leaving a decoding latency bottleneck.

The central question the authors pose is this (source: §1).

“Can combined token·feature compression go beyond the limits of a single axis while also avoiding the reconstruction bottleneck?”

The New Approach: SlimKV

SlimKV is a framework that combines feature-axis compression on top of an Activation Beacon-style progressive token-compression pipeline (source: §3.1). The workflow is as follows.

1. Split the input into chunks at fixed intervals, and insert a beacon token between each chunk.
2. Let each chunk attend to the previous beacon memories so the beacon state absorbs context information.
3. When a chunk’s prefill finishes, discard the KV of the raw tokens and cache only the compressed beacon memory.

SlimKV differs in two ways.

(1) Low-Rank Beacon Training — Instead of post-hoc SVD decomposition, it trains the beacon KV projection in the target low-rank form from the start (source: §3.2). The backbone is frozen and only the beacon branch is updated. The beacon hidden state $h_b^{(l)}$ (at layer $l$) is projected as

into latent states $C_K^{(l)}, C_V^{(l)}$ smaller than the original KV dimension $D$, and the full-dimensional form is reconstructed as $K_b^{(l)} = C_K^{(l)} U_K^{(l)\top}$, $V_b^{(l)} = C_V^{(l)} U_V^{(l)\top}$. At inference time, only the latent states $C_K, C_V$ are stored in the cache.

(2) Layer-Adaptive Rank Allocation — The low-rank structure of transformer projection matrices varies by layer. Since the beacon projection is unknown before training, the authors use the pretrained original KV projection weights as a proxy (source: §3.3). The two exhibit per-layer rank patterns with a Spearman correlation of about 0.99 or higher and a per-layer rank error within 2, matching almost exactly (source: Appx.A, Tab.SVD).

Specifically, for the singular values $\sigma_{l,1} \ge \cdots \ge \sigma_{l,D}$ of the original KV projection, they define the cumulative spectral energy

and pick the initial rank $r_l^{0} = \min\{ r \mid E_l(r) \ge \rho \}$ with an energy threshold $\rho$. They then introduce a “floor” $r_{\min} = \lceil \lambda \bar r \rceil$ to keep layers with too-small ranks from becoming a training bottleneck, and choose the largest $\rho$ that satisfies the total budget $B = LD/k$. This paper uses $\lambda = 0.5$ (source: §5.1).

How It Works: A Concrete Example

Let us walk through the whole flow, from compression to speedup, with a small example. To make this easier, assume the following setting (source: §4.4, Appx.C).

• KV heads $H_{kv}=8$, query heads $H_q=32$, GQA group $g = H_q/H_{kv} = 4$
• Per-head dimension $d=128$, total KV dimension $D = H_{kv} \cdot d = 1024$
• Feature compression ratio $k=4$ → latent KV dimension $D/k = 256$

Step 1 — Token compression. Summarizing a 16K-token document into one beacon every 4 tokens yields 4K beacons (4× token compression).

Step 2 — Feature compression. Each beacon’s K/V is projected from $D=1024$ dimensions into a $256$-dimensional latent space (4× feature compression). Multiplying the two gives 16× compression, and the cache retains only $C_K, C_V \in \mathbb{R}^{4000 \times 256}$.

Step 3 — Reconstruction-free attention. Now suppose we compute attention for a single new token during decoding. If RoPE remains on the beacon Key, RoPE is position-dependent and cannot be absorbed into the latent space in advance, so we must first reconstruct the full dimensions via $K_b = C_K U_K^\top$ and then apply RoPE.

In contrast, SlimKV has removed RoPE from the beacon Key, so using the associativity of matrix multiplication it computes the score directly in the latent space (source: §4.3).

The Value path is handled the same way, $A V_b = A (C_V U_V^\top) = (A C_V) U_V^\top$. In other words, the two steps of “reconstruction → attention” become the single step of “query projection → latent attention.”

How much does the cost change? Comparing the dominant FLOPs of one decoding step (source: §4.4, Appx.C)

where $N_b$ is the number of cached beacon tokens. The reconstruction approach incurs a full-dimensional reconstruction cost of $\mathcal{O}(N_b D^2/k)$ proportional to $N_b$, whereas SlimKV turns that term into $\mathcal{O}(gD^2/k)$, which applies only to the current step’s query and output projections, independent of $N_b$. Plugging in the setting above,

so for long contexts ($N_b \gg 128$) the theoretical arithmetic is reduced by about 32.6×. Of course, this accounts only for arithmetic cost and excludes kernel overhead and memory bandwidth, so the measured speedup is more conservative, as shown below (source: Appx.C).

Performance Validation: Main Results

LongBench — The More Compression, the Better

In the high-compression 16×/32× regime, SlimKV set the best average score on both backbones (source: §5.2, Tab.1). Notably, SlimKV-f4 (t4f4, 4× token · 4× feature) at 16× compression reaches Llama-3.1-8B 46.36 and Qwen2.5-14B 46.23, beating SnapKV, Activation Beacon, PALU, and KVZip. At 32× compression (t8f4) it is also best at 44.75 / 45.09 respectively.

Comparing feature compression ratios of 2×/4×/8×, 4× was the sweet spot giving the best average in most high-compression settings. This is due to the tradeoff: too little feature compression makes token compression excessive, while too much leaves insufficient latent capacity (source: §5.2).

Comparing the gap from uncompressed against Activation Beacon, SlimKV-f4 narrows the gap by 18.5%–28.2% (source: Tab.2). At low compression (4×/8×) it uses 2× feature compression (SlimKV-f2) to retain over 96% of the uncompressed score, and on Qwen2.5-14B at 8× reaches 49.94 (uncompressed 51.52), far ahead of KVZip and PALU (source: Tab.3).

Question-Independent vs Question-Aware: The Strength Revealed in Multi-Turn

Question-aware methods like SnapKV benefit from knowing the question in advance. SlimKV is question-independent, so it shines in scenarios that reuse a single compressed context across multiple questions (source: §5.2). In the QMSum multi-request experiment, against an uncompressed ROUGE-L F1 of 25.17, SnapKV plummets to 10.39 at 16× compression, while SlimKV-f4 holds 24.47, 97.2% of uncompressed.

Needle-in-a-Haystack — Positional Robustness Survives Even Without K-RoPE

Removing RoPE from the beacon Key raises concerns about losing positional information. The NIAH results put this concern to rest (source: §5.2, Fig.7). SlimKV-f4 showed stable retrieval performance across needle depth, and maintained performance even when the answer lay beyond Qwen2.5-14B’s context window (32K). This is because long context is compressed into a small number of beacon tokens, keeping the cache within the usable range, and the positional cues absorbed during RoPE-aware prefill remain valid even without K-RoPE.

Efficiency — The Real Benefit of Eliminating Reconstruction

Single-step attention and end-to-end latency were measured on a single A100 80GB, BF16, batch size 1, with Llama-3.1-8B (source: §5.4, Tab.5). Compared with the reconstruction-based variant (Recon, which keeps beacon K-RoPE and reconstructs full dimensions), SlimKV’s speedup grows as context lengthens.

Also, at the maximum length of 11.5K where the uncompressed model can run without OOM, SlimKV improves TTFT by 1.03–1.37× and reduces peak memory by 2.71–2.82× (source: §5.4). Interestingly, the peak memory of Recon and SlimKV is nearly the same. Removing reconstruction changes the compute flow, not the storage size (source: Appx.F).

Scalability — It Transfers Well Even to a 30B MoE

On Qwen3-30B-A3B-Instruct (MoE), SlimKV-f4 maintained competitive performance at 46.08 for 16× and 45.60 for 32× (uncompressed 50.30) (source: §5.3, Tab.4). Its trainable parameters are only 434M, just 1.45% of the full model, demonstrating the potential for parameter-efficient scaling.

Ablation — The Fact That “No Reconstruction” Is Not Free

The 32× compression ablation is especially telling (source: §5.5, Tab.6). Simply removing beacon K-RoPE drops performance from 43.67 to 41.28. Removing RoPE alone is not enough — one must train directly in low-rank form (41.95 → 44.38) and add layer-adaptive rank allocation (44.75) to surpass Activation Beacon.

Our Perspective: Strengths, Limits, and Why This Research Matters

Strengths. The most valuable contribution of this work is that it redraws the problem not as a single axis of “accuracy vs efficiency” but as a structure of two axes of compression × the reconstruction bottleneck. In particular, (1) the empirical discovery of beacon K-RoPE asymmetry and (2) the logical development that connects it to eliminating reconstruction are clean. Even on the attention heatmap similarity metric, removing beacon K-RoPE gives a Pearson correlation of 0.9707 with the original pattern, far closer than 0.7570 for removing original tokens (source: §4.2, Tab.5). The fact that the cost analysis honestly separates theory (32.6×) from measurement (up to 7.34×) also inspires confidence.

Limits. The authors themselves acknowledge two (source: §Limitations). First, validation is confined to RoPE-based LLMs. Second, it is text-only, and extension to multimodal/cross-modal KV caches is unverified. From a reader’s perspective, we would add that (a) requiring training is a barrier to adoption compared with training-free methods like PALU and KVZip (though the number of trainable parameters is small), (b) having erased the RoPE of the beacon Key, there is considerable backbone-to-backbone variation on order-sensitive tasks (especially code and format-dependent tasks) (source: Appx.G), and (c) the NIAH judgment relies on automatic GPT-5.4-mini grading, which caps the robustness of the measurement.

Why this research matters. In long-context serving, the KV cache is already the main memory and bandwidth bottleneck, and compression techniques so far have failed to simultaneously address “how well is accuracy preserved” and “does it actually get faster.” SlimKV proposes a design principle that answers both questions at once — beacons do not need to express position via RoPE — and this is likely to become a baseline reference for future low-rank KV design.

What’s Next?: The Road Ahead

The authors themselves suggest two directions (source: §Limitations). One is verifying generalization to positional encodings other than RoPE (e.g., ALiBi, YaRN, learned positions). The paper theoretically shows that ALiBi is algebraically compatible as $S = Q K_b^\top + B = (Q U_K) C_K^\top + B$ (source: Appx.J), so empirical validation is the natural next step. The other is extension to visual/cross-modal KV caches.

Adding further, the reasonably expected follow-up work can be summarized as follows.

• Shoring up order-sensitive tasks: Injecting a subtle positional cue (e.g., a lightweight ALiBi-style bias) to offset the positional information loss of the beacon Key could reduce backbone-to-backbone variation on code and format tasks.
• Quantitative comparison with MLA: Although the appendix gives a conceptual comparison (source: Appx.I), a head-to-head comparison against MLA-style low-rank designs at the same backbone and same compression ratio is still lacking.
• System-level optimization: A kernel implementation for latent-space attention is the key lever that could narrow the gap between the measured speedup (7.34×) and the theoretical speedup (32.6×).
• Reducing training cost: Currently training takes hours to tens of hours per backbone, so research on initialization and regularization that converges with less corpus would lower the deployment barrier.

SlimKV is a study that combines practicality and insight, turning the conventional wisdom that “compression means loss” into “where you compress determines the loss.”

Tables from the paper

Tables converted mechanically from the arXiv e-print LaTeX source. The numbers are the paper’s own and did not pass through a model.

Table 1. Layer-wise rank consistency between Beacon and raw K/V projections.

Table 2. Detailed decoding latency on Llama-3.1-8B-Instruct over 128 generated tokens. Attn. and E2E denote cumulative attention and end-to-end decoding latency. All values are in seconds.

Table 3. TTFT and peak allocated memory on Llama-3.1-8B-Instruct. TTFT is reported in seconds and memory in GB.

Table 4. Similarity to the w/ K-RoPE setting.

Table 5. LongBench scores under high KV-cache compression. SlimKV t$n$f$m$: $n\times$ token-wise, $m\times$ feature-wise. The bold and underlined numbers denote the first and second rankings, respectively.

Table 6. LongBench gap reduction over Activation Beacon. $\Delta$ denotes the drop from Full KV.

Table 7. LongBench averages at low compression. The averages of the uncompressed models are in Table .

Table 8. LongBench category scores on Qwen3-30B-A3B-Instruct-2507.