Tailoring the Quantization Space for 1-Bit KV Cache Compression
TaSQ: 'Tailored Design' of the Quantization Space for a 1-bit KV Cache TL;DR — TaSQ (Tailored Space Vector Quantization) redesigns the very target space to which vector…
Posts
All posts on technology, daily life, and thoughts.
118 pages
5 /
20
Prefill-Free Cross-Family KV Cache Transfer for Heterogeneous Multi-Agent LLMs
How to Hand Off KV Caches As-Is Between Different LLMs: HeteroFold TL;DR: HeteroFold, a method that reuses the KV cache between models with different tokenizers and…
On-Policy or Off-Policy Learning? A Systematic Study of Distillation Dynamics
On-Policy Is Not a Panacea: The Real Protagonists of Rollout Policy, KL Direction, and Learning Rate in Distillation Dynamics TL;DR — When we separate on-policy (OnPD)…
The KV Cache Is the New Memory Wall
The KV Cache Is the New Memory Wall: Three Regimes and Five Domains of Long-Context LLM Inference TL;DR — Long-context LLM decode inference is bound by memory bandwidth,…
WavePP: High-Throughput Pipeline Parallel LLM Prefill under Prefix Reuse
WavePP — A Pipeline-Parallel Prefill Runtime That Burns In Prefix Reuse TL;DR Built on top of TensorRT-LLM, WavePP overlaps request admission (reuse consensus, cache…
Periodic Weak Spots: Phase Sensitivity from Chunked KV-Cache Compression
Phase Sensitivity: The 'Periodic Weakness' Created by Chunked KV-Cache Compression TL;DR — In models that use chunked KV-cache compression (such as the DeepSeek-V4…