Rethinking Sequence Compression with Test-time Regression
Introduction In models that handle long contexts, the compression operation that reduces multiple tokens to a single vector becomes a performance bottleneck. This paper…
With-Deepseek-V4-Pro
53 pages
Large Flow Language Models
The sequential bottleneck of sampling one token at a time Large language models can compute an entire sentence in parallel during training, but when generating text they…
Cache Engineering for Agents
The Cost and Accuracy Dilemma of Long Agent Trajectories LLM agents accumulate long trajectories by repeating reasoning and tool calls for hundreds of rounds. To make…
Speculative Sampling in Interactive Video Generation
Speculative sampling reuses future video predictions to reduce response latency. How it works, recomputation costs, and the limits of the serving experiments.
MemT-1.0: Towards Memory-Native Evolution
MemT-1.0 stores conversation history in a fixed-size matrix state. We examine memory updates, information loss, and how the approach compares with full-context and…
DLite: History-free Speculative Decoding for Agentic AI
DLite explained through a candidate-verification example, cache ownership, and a careful comparison of acceptance length and decoding speed.
FlashMask-3: Efficient and Expressive Mask-Aware Distributed Attention
An analysis of FlashMask-3’s Hopper kernels, mask localization, load balancing, communication overlap, and long-context training results and limitations.
HLA-WM: Hybrid Linear Attention for Long-Horizon Video World Models
HLA-WM retrieves past chunks using camera geometry to restore scene memory. At 60 seconds, its history memory is about 12× smaller than Full KV and 6× larger than GDN,…
Contextual Correction of Parallel Drafts for Speculative Decoding
DCOCO reselects existing parallel draft candidates using contextual scores. This review distinguishes accepted length from actual throughput and examines correction…
Towards Looped Models Done Right, Part II: Rethinking at Fixed Points
Fixed-point behavior connects KV sharing, state reuse, and distilled prefill in looped language models. We review the savings and accuracy trade-offs demonstrated on…
SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inference
SparseDecoding calibrates pruning on model-generated tokens and adds a decoding-specific sparse kernel. We examine the conditions behind its maximum 1.48× speedup on A100…
MC-Sparse: Deconstructing and Closing the Dense-Sparse Attention Gap in Diffusion Transformers
MC-Sparse separates three sources of quality and efficiency loss in sparse DiT attention. This review explains why token selection and kernel execution costs must be…
TokenRouter: Efficient Serving System for Token-Level LLM Routing
TokenRouter: An Efficient Serving System for Token-Level LLM Routing TL;DR — Routing an LLM at the token level rather than the session/query level can substantially push…
Mechanics of Long-Context Hybrid Models Part 1.1: From Hybrid Attention to Hybrid Position
Mechanics of Long-Context Hybrid Models: From Hybrid Attention to Hybrid Position TL;DR — These days open-source LLMs are rapidly shifting from models that "use full…
TRACE: Rollout-Guided Quantization-Aware Training for FP4 Reinforcement Learning of MoE Language Models
TRACE: Rollout-Aligned FP4 Quantization-Aware Training — MoE LLM RL at BF16 Quality, up to 5.4x Faster TL;DR In reinforcement learning (RL) for MoE LLMs, quantizing…
SlimKV: Joint Token-Feature KV Cache Compression with Reconstruction-Free Beacon Attention
SlimKV: Compressing Tokens and Features Together, Cutting the KV Cache by up to 32× with Reconstruction-Free Latent-Space Attention TL;DR SlimKV is a question-agnostic…
Tailoring the Quantization Space for 1-Bit KV Cache Compression
TaSQ: 'Tailored Design' of the Quantization Space for a 1-bit KV Cache TL;DR — TaSQ (Tailored Space Vector Quantization) redesigns the very target space to which vector…
On-Policy or Off-Policy Learning? A Systematic Study of Distillation Dynamics
On-Policy Is Not a Panacea: The Real Protagonists of Rollout Policy, KL Direction, and Learning Rate in Distillation Dynamics TL;DR — When we separate on-policy (OnPD)…
The KV Cache Is the New Memory Wall
The KV Cache Is the New Memory Wall: Three Regimes and Five Domains of Long-Context LLM Inference TL;DR — Long-context LLM decode inference is bound by memory bandwidth,…
WavePP: High-Throughput Pipeline Parallel LLM Prefill under Prefix Reuse
WavePP — A Pipeline-Parallel Prefill Runtime That Burns In Prefix Reuse TL;DR Built on top of TensorRT-LLM, WavePP overlaps request admission (reuse consensus, cache…
Periodic Weak Spots: Phase Sensitivity from Chunked KV-Cache Compression
Phase Sensitivity: The 'Periodic Weakness' Created by Chunked KV-Cache Compression TL;DR — In models that use chunked KV-cache compression (such as the DeepSeek-V4…
MassAlloc Attention: Let Attention Allocate Its Own Compute
Paper MassAlloc Attention (MALA): Letting Attention Allocate Its Own Compute TL;DR — Full softmax attention (FullAttn) performs all post-score computation after QK score…
WorldAttention: An Efficient Attention Architecture for Interactive Video World Models
WorldAttention: System Co-Design of an Interactive Video World Model That Drops the Sliding Window TL;DR — Text-controlled interactive video world models are trapped…
Block Sparse Attention with Log-Linear Complexity
PISA: Pyramid Top-$K$ Selection Lowers the Cost of Block-Sparse Attention to $O(N\log N)$ TL;DR — Block-sparse attention makes attention computation linear, but block…
Who Pays for the KV Cache? Attributing Shared AI Inference Spend Across Kubernetes and LLM Provider Bills
Who pays for the KV cache? — Kubernetes, gateway, and provider bills in one ledger TL;DR — The cost of a single AI feature is scattered across three disconnected ledgers:…
GDN Tree-Scan: Served Tree Verification for Recurrent-Hybrid Language Models
LumoTree: Path-Parallel Speculative Verification for Hybrid Language Models TL;DR — Tree speculative decoding in hybrid models that mix Gated DeltaNet with full attention…
SPECTRA: Adaptive Execution of Speculative Decoding on a Runtime-Reconfigurable Tiled Architecture
SPECTRA: A Reconfigurable LLM Accelerator That Changes Its Hardware With Every Inference Stage TL;DR Speculative decoding oscillates between prefill (compute-bound GEMM),…
ActionPiece: Rethinking Action Tokenization for Autoregressive Vision-Language-Action Models
ActionPiece: Rethinking Action Tokenization Beyond Reconstruction Error, via Physical Ranking TL;DR — In autoregressive VLA (vision-language-action) models, the action…
DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression
DeepSeek-V4.1-Flash: A Multimodal MoE That Cuts the KV Cache to 1/4 While Raising Performance Further TL;DR. DeepSeek-V4.1-Flash, released by DeepSeek-AI, is a multimodal…
StepAudio 3 Gen Technical Report
StepAudio 3 Gen: Unifying Speech, Song, Music, and Sound Effects with a Single Discrete Autoregressive (RVQ) Model TL;DR StepAudio 3 Gen, released by StepFun, is a fully…
How Lossless Is Lossless Speculative Decoding? The Role of Numerical Precision in Orthrus
Is "Lossless" Speculative Decoding Really Lossless? — The Numerical-Precision Trap Exposed by Reproducing Orthrus TL;DR — Orthrus attaches a diffusion view to a frozen AR…
ZGCM-1: A Fully Open and Extremely Efficient Foundation Model for Math and Agentic Search
ZGCM-1: How a 7B Dense Model Trades Blows with 235B Frontiers — A Fully Open, Ultra-Efficient Foundation Model Design Report TL;DR — With a single 7.39B dense model,…
Programmable World Model
Programmable World Model — Embedding "Executable State" into Generative Worlds One-Line Summary (TL;DR) Video world models are producing increasingly realistic…
SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking
SAS: A Simple Attention Sparsification That Learns Context Ranking Directly Without Distillation TL;DR — In post-training attention sparsification, inject the continuous…
CoVeR: Coverage-Based Token Pruning for Multi-View 3D Reasoning in VLMs
CoVeR: Coverage-Based Pruning That Cuts Visual Tokens of Multi-View 3D Reasoning by up to 92% Using Geometry Alone One-Line Summary (TL;DR) Rendering a 3D scene into…
Marigold V2: Revisiting Diffusion Transformers for Monocular Depth Estimation
Marigold V2: Reviving an Image-Editing Diffusion Transformer (DiT) as a Monocular Depth Estimator TL;DR This study fine-tunes an image-editing diffusion transformer…
SQS: Bayesian DNN Compression through Sparse Quantized Sub-distributions
SQS: Fusing Pruning and Quantization into One Bayesian Learning — Where Spike-and-Slab Meets GMM TL;DR — Doing pruning and low-bit quantization separately leaves…
Kalman Delta Networks: Uncertainty-aware Associative Memory
Linear Attention Rewritten with a Kalman Filter: Kalman Delta Networks TL;DR The fixed-size recurrent memory update of linear attention is reinterpreted as a…
X-AuT: Progressive Audio-Encoder Compression for Speech LLMs with Cross-Scale Distillation
X-AuT: Pruning Audio Encoders in Speech LLMs with Behavioral Probes and Cross-Scale Distillation TL;DR X-AuT is a framework that progressively prunes the audio encoder…
Deadline-Aware Adaptive Prefill Chunking for Efficient Large Language Model Serving
Deadline-Filling Prefill Chunks: SLOWeave — An Adaptive Chunking Scheduler for LLM Serving TL;DR — Instead of a fixed prefill chunk size, we propose SLOWeave, an online…
Why Is Video Still So Expensive? A Survey of Inference-Efficiency Mechanisms in Video and Audiovisual LLMs
Why Are Video LLMs Still So Expensive? — A Four-Stage Guide to Inference Efficiency Mechanisms TL;DR — This paper classifies 125 VideoLLM inference efficiency studies…
Miles v0.1: Production-Level Post-Training
Miles v0.1: A "verified, clean, and scalable" full-stack system for frontier RL post-training TL;DR: Miles is a full-stack RL post-training framework that resolves the…
Online Draft Co-Training for Speculative Decoding in Large-Scale, Long-Context RL Post-Training
Two Designs That Make Speculative Decoding Practical for Large-Scale, Long-Context RL Post-Training TL;DR : Rollout generation dominates the wall-clock time of RL…
VestigeKV: The NoPE-MLA KV Cache Carries Its Own Eviction Signal in a Vestigial Branch
VestigeKV: The Degenerate RoPE Branch Carries Its Own Cache Eviction Signal TL;DR In NoPE-MLA models (Kimi Linear 48B), reading only 11% of each row — the 64-dim…
ShallowStream: Index Shallow then Answer Deep for Streaming Video Understanding
ShallowStream: Index Shallowly, Answer Deeply — Unlocking the Computational Bottleneck of Streaming Video Understanding TL;DR — In streaming video understanding, the most…
Unlocking Lossless Speedups in LLMs via Discrete Diffusion
Uno: AR and Diffusion in One Model — Lossless Parallel Generation for LLM Acceleration TL;DR — LLMs are slow because next-token prediction (NTP) can emit only one token…
BeaconKV: Key-Value Cache Compression Guided by Beacon Queries for Efficient Large Reasoning Model Inference
Catch the Moment a Model "Revisits Its Thoughts": How BeaconKV Compresses Reasoning KV Caches 5.8× with Beacon Queries TL;DR — The KV cache of a Large Reasoning Model…
Hardware-Aware FP4 FlashAttention-4
Why Don't FP4 Tensor Cores Make Attention Faster? Blackwell FlashAttention-4, Solved with Direct-P and Quantized Backprop TL;DR — Blackwell's FP4 tensor cores handle…
Language Models Can Control Their Own Attention
Language Models Control Their Own Attention: Declarative Attention TL;DR — Existing sparse attention methods still paid $O(N)$ per decoding step by approximating…
Same Request, Different Answer: Quantization Amplifies Cache-Induced Divergence in LLM Serving
Same Request, Different Answer: Prefix Caching Makes Serving Non-Reproducible, and Quantization Amplifies It TL;DR — Prefix caching is billed as a "transparent…
SGD-KV: Summarization Guided KV Cache Compression
SGD-KV: Finding 'heads that are good at summarizing' cuts the 1M-token KV cache by up to 75% TL;DR — Attention heads do not all do the same job. SGD-KV pinpoints the…
SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers
SMELT: Scaling Laws for Compute-Matched MoE Loop Transformers TL;DR Loop transformers scale depth by executing a layer block repeatedly, but most prior work fixed only…
Why Gated DeltaNet Survives 4-Bit Quantization: NVFP4 W4A4 for the Recurrent Half of a Hybrid 27B LLM
The Recurrent Half Is the Easy-to-Quantize Half: Why Gated DeltaNet Survives at 4 Bits TL;DR — The 48 recurrent Gated DeltaNet (GDN) layers of a hybrid 27B LLM…