Unlocking Lossless Speedups in LLMs via Discrete Diffusion
Uno: AR and Diffusion in One Model — Lossless Parallel Generation for LLM Acceleration TL;DR — LLMs are slow because next-token prediction (NTP) can emit only one token…
Posts
All posts on technology, daily life, and thoughts.
119 pages
14 /
20
BeaconKV: Key-Value Cache Compression Guided by Beacon Queries for Efficient Large Reasoning Model Inference
Catch the Moment a Model "Revisits Its Thoughts": How BeaconKV Compresses Reasoning KV Caches 5.8× with Beacon Queries TL;DR — The KV cache of a Large Reasoning Model…
Hardware-Aware FP4 FlashAttention-4
Why Don't FP4 Tensor Cores Make Attention Faster? Blackwell FlashAttention-4, Solved with Direct-P and Quantized Backprop TL;DR — Blackwell's FP4 tensor cores handle…
Language Models Can Control Their Own Attention
Language Models Control Their Own Attention: Declarative Attention TL;DR — Existing sparse attention methods still paid $O(N)$ per decoding step by approximating…
Same Request, Different Answer: Quantization Amplifies Cache-Induced Divergence in LLM Serving
Same Request, Different Answer: Prefix Caching Makes Serving Non-Reproducible, and Quantization Amplifies It TL;DR — Prefix caching is billed as a "transparent…
SGD-KV: Summarization Guided KV Cache Compression
SGD-KV: Finding 'heads that are good at summarizing' cuts the 1M-token KV cache by up to 75% TL;DR — Attention heads do not all do the same job. SGD-KV pinpoints the…