TL;DR — Attention heads do not all do the same job. SGD-KV pinpoints the summarization heads — the ones responsible for the high-level information compression called ‘summarization’ — with a diagnostic task, then hands out the KV cache budget in proportion to their scores. At contexts of up to 1 million (1M) tokens it cuts KV cache memory by up to 75% while beating existing head-level compression methods.

2609.03235v1 on Jaehun's Blog

SGD-KV: Summarization Guided KV Cache Compression

SGD-KV: Finding ‘heads that are good at summarizing’ cuts the 1M-token KV cache by up to 75%