Rethinking Sequence Compression with Test-time Regression
Introduction In models that handle long contexts, the compression operation that reduces multiple tokens to a single vector becomes a performance bottleneck. This paper…
Long Context
As context grows, attention cost and the KV cache grow with it. These posts generally take one of four routes:
- Make attention sparse — hardware-aligned, natively trainable sparsity
- Compress, then recompute — shrink the context and rebuild what is needed
- Change the architecture — state-space models and their relation to Transformers
- Absorb it in serving — parallelism for million-token requests
34 pages
Cache Engineering for Agents
The Cost and Accuracy Dilemma of Long Agent Trajectories LLM agents accumulate long trajectories by repeating reasoning and tool calls for hundreds of rounds. To make…
DLite: History-free Speculative Decoding for Agentic AI
DLite explained through a candidate-verification example, cache ownership, and a careful comparison of acceptance length and decoding speed.
FlashMask-3: Efficient and Expressive Mask-Aware Distributed Attention
An analysis of FlashMask-3’s Hopper kernels, mask localization, load balancing, communication overlap, and long-context training results and limitations.
HLA-WM: Hybrid Linear Attention for Long-Horizon Video World Models
HLA-WM retrieves past chunks using camera geometry to restore scene memory. At 60 seconds, its history memory is about 12× smaller than Full KV and 6× larger than GDN,…
Mechanics of Long-Context Hybrid Models Part 1.1: From Hybrid Attention to Hybrid Position
Mechanics of Long-Context Hybrid Models: From Hybrid Attention to Hybrid Position TL;DR — These days open-source LLMs are rapidly shifting from models that "use full…
SlimKV: Joint Token-Feature KV Cache Compression with Reconstruction-Free Beacon Attention
SlimKV: Compressing Tokens and Features Together, Cutting the KV Cache by up to 32× with Reconstruction-Free Latent-Space Attention TL;DR SlimKV is a question-agnostic…
Tailoring the Quantization Space for 1-Bit KV Cache Compression
TaSQ: 'Tailored Design' of the Quantization Space for a 1-bit KV Cache TL;DR — TaSQ (Tailored Space Vector Quantization) redesigns the very target space to which vector…
Prefill-Free Cross-Family KV Cache Transfer for Heterogeneous Multi-Agent LLMs
How to Hand Off KV Caches As-Is Between Different LLMs: HeteroFold TL;DR: HeteroFold, a method that reuses the KV cache between models with different tokenizers and…
The KV Cache Is the New Memory Wall
The KV Cache Is the New Memory Wall: Three Regimes and Five Domains of Long-Context LLM Inference TL;DR — Long-context LLM decode inference is bound by memory bandwidth,…
WavePP: High-Throughput Pipeline Parallel LLM Prefill under Prefix Reuse
WavePP — A Pipeline-Parallel Prefill Runtime That Burns In Prefix Reuse TL;DR Built on top of TensorRT-LLM, WavePP overlaps request admission (reuse consensus, cache…
Periodic Weak Spots: Phase Sensitivity from Chunked KV-Cache Compression
Phase Sensitivity: The 'Periodic Weakness' Created by Chunked KV-Cache Compression TL;DR — In models that use chunked KV-cache compression (such as the DeepSeek-V4…
MassAlloc Attention: Let Attention Allocate Its Own Compute
Paper MassAlloc Attention (MALA): Letting Attention Allocate Its Own Compute TL;DR — Full softmax attention (FullAttn) performs all post-score computation after QK score…
WorldAttention: An Efficient Attention Architecture for Interactive Video World Models
WorldAttention: System Co-Design of an Interactive Video World Model That Drops the Sliding Window TL;DR — Text-controlled interactive video world models are trapped…
Block Sparse Attention with Log-Linear Complexity
PISA: Pyramid Top-$K$ Selection Lowers the Cost of Block-Sparse Attention to $O(N\log N)$ TL;DR — Block-sparse attention makes attention computation linear, but block…
GDN Tree-Scan: Served Tree Verification for Recurrent-Hybrid Language Models
LumoTree: Path-Parallel Speculative Verification for Hybrid Language Models TL;DR — Tree speculative decoding in hybrid models that mix Gated DeltaNet with full attention…
Adapting Tree-Structured Speculative Decoding to DeepSeek-V4 for Efficient Inference
Branching on Compressed Memory: Adding Tree Reasoning to DeepSeek-V4 TL;DR Adding tree-structured speculative decoding to DeepSeek-V4-Flash yields longer acceptance…
DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression
DeepSeek-V4.1-Flash: A Multimodal MoE That Cuts the KV Cache to 1/4 While Raising Performance Further TL;DR. DeepSeek-V4.1-Flash, released by DeepSeek-AI, is a multimodal…
Fathom: Per-Query Read Depth for Sparse Decoding over Offloaded KV Caches
Fathom: Accelerating 1M-Token Decoding by 1.67x with Query-Adaptive Depth Scanning TL;DR When agent sessions stretch to 1M tokens and dozens of sessions reside…
ZGCM-1: A Fully Open and Extremely Efficient Foundation Model for Math and Agentic Search
ZGCM-1: How a 7B Dense Model Trades Blows with 235B Frontiers — A Fully Open, Ultra-Efficient Foundation Model Design Report TL;DR — With a single 7.39B dense model,…
SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking
SAS: A Simple Attention Sparsification That Learns Context Ranking Directly Without Distillation TL;DR — In post-training attention sparsification, inject the continuous…
Kalman Delta Networks: Uncertainty-aware Associative Memory
Linear Attention Rewritten with a Kalman Filter: Kalman Delta Networks TL;DR The fixed-size recurrent memory update of linear attention is reinterpreted as a…
Deadline-Aware Adaptive Prefill Chunking for Efficient Large Language Model Serving
Deadline-Filling Prefill Chunks: SLOWeave — An Adaptive Chunking Scheduler for LLM Serving TL;DR — Instead of a fixed prefill chunk size, we propose SLOWeave, an online…
Why Is Video Still So Expensive? A Survey of Inference-Efficiency Mechanisms in Video and Audiovisual LLMs
Why Are Video LLMs Still So Expensive? — A Four-Stage Guide to Inference Efficiency Mechanisms TL;DR — This paper classifies 125 VideoLLM inference efficiency studies…
Online Draft Co-Training for Speculative Decoding in Large-Scale, Long-Context RL Post-Training
Two Designs That Make Speculative Decoding Practical for Large-Scale, Long-Context RL Post-Training TL;DR : Rollout generation dominates the wall-clock time of RL…
Unlocking Lossless Speedups in LLMs via Discrete Diffusion
Uno: AR and Diffusion in One Model — Lossless Parallel Generation for LLM Acceleration TL;DR — LLMs are slow because next-token prediction (NTP) can emit only one token…
Language Models Can Control Their Own Attention
Language Models Control Their Own Attention: Declarative Attention TL;DR — Existing sparse attention methods still paid $O(N)$ per decoding step by approximating…
SGD-KV: Summarization Guided KV Cache Compression
SGD-KV: Finding 'heads that are good at summarizing' cuts the 1M-token KV cache by up to 75% TL;DR — Attention heads do not all do the same job. SGD-KV pinpoints the…
![[Paper Review] Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models](https://paper-assets.alphaxiv.org/figures/2601.07372v1/img-0.jpeg)
[Paper Review] Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models
Engram: The Second Sparsity Axis After Conditional Computation (MoE), Conditional Memory One-Line Summary (TL;DR) Engram adds a "knowledge lookup primitive" to the…
![[Paper Review] Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://icml.cc/media/PosterPDFs/ICML%202024/32613.png)
[Paper Review] Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
Structured State Space Duality: Unifying SSMs and Attention with Mamba-2 for 2–8× Acceleration TL;DR Structured State-Space Duality (SSD) reveals that SSMs and masked…
![[Paper Review] KIMI K2: OPEN AGENTIC INTELLIGENCE](https://github.com/MoonshotAI/Kimi-K2/raw/main/figures/kimi-logo.png){kind=logo}
[Paper Review] KIMI K2: OPEN AGENTIC INTELLIGENCE
Kimi K2: An Open-Source LLM’s Leap Toward Agentic Intelligence TL;DR With a 3-stage pipeline consisting of MuonClip pretraining + large-scale agentic tool-use data +…
![[Paper Review] Qwen 3 Technical Report](https://qianwen-res.oss-accelerate-overseas.aliyuncs.com/logo_qwen3.png){kind=logo}
[Paper Review] Qwen 3 Technical Report
Qwen 3: The Evolution of a Giant MoE Language Model with Adjustable Reasoning Depth TL;DR (in one line) Qwen 3 couples a user-controllable Thinking Budget with a…
Code I/O: Condensing Reasoning Patterns via Code Input-Output Prediction
CODE I/O: From Code I/O + Natural-Language CoT to General-Purpose Reasoning — Lifting 7B-30B LLMs by +2 Points on Average with Data Design Alone TL;DR A single data…
Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
Native Sparse Attention (NSA) — 11× faster even at 64k tokens, accuracy intact One-line summary (TL;DR) NSA combines a three-branch sparse attention — “compress → select…