TAO: Tool-Aware Joint Scheduling for LLM-Based Agentic Serving
TAO: Tool-Aware Joint Scheduling for LLM-Based Agentic Serving Original: Shuhuai Li et al., "TAO: Tool-Aware Joint Scheduling for LLM-Based Agentic Serving"…
Efficient Inference
49 pages
On the SPOT: Enhancing Speculative Decoding for Diffusion Language Models via Commit-Order Prediction
Diffusion language models must decide 'what' and 'where' at the same time Diffusion language models generate sentences by repeatedly filling masked tokens. To decode…
Channel Selection for Mixture-of-Experts Decoding Needs No Calibration Data
Introduction In settings where the batch size is close to 1, such as conversational services, the latency of a mixture-of-experts language model comes mostly from reading…
Extreme Compression of Per-Layer Embeddings via Cross-Layer Latent Vector Quantization
Recent language models keep increasing the amount of information packed into a single vocabulary entry. Per-layer embeddings (PLE), first introduced in Gemma 3n, carried…
Intern Lumina U2: A Multi-Codebook Diffusion Large Language Model for Omni-Visual Understanding and Image Generation
Problem: Why It's Hard to Unify Understanding and Generation in One Representation There have been many attempts to merge image understanding and image generation into a…
Matrix-Game 4.0: Harnessing Video Models for Real-Time Playable Game Worlds
Introduction A game world built by a video generation model alone stores off-screen state nowhere. Every time a corridor you have already passed through is redrawn, it…
Rethinking Sequence Compression with Test-time Regression
Introduction In models that handle long contexts, the compression operation that reduces multiple tokens to a single vector becomes a performance bottleneck. This paper…
Large Flow Language Models
The sequential bottleneck of sampling one token at a time Large language models can compute an entire sentence in parallel during training, but when generating text they…
Cache Engineering for Agents
The Cost and Accuracy Dilemma of Long Agent Trajectories LLM agents accumulate long trajectories by repeating reasoning and tool calls for hundreds of rounds. To make…
Speculative Sampling in Interactive Video Generation
Speculative sampling reuses future video predictions to reduce response latency. How it works, recomputation costs, and the limits of the serving experiments.
DLite: History-free Speculative Decoding for Agentic AI
DLite explained through a candidate-verification example, cache ownership, and a careful comparison of acceptance length and decoding speed.
Trajectory-Level Speculation with Semantic Verification for Tool-Using Agents
An analysis of SpecAgent, where a weaker model drafts tool calls and a stronger model verifies them, covering its design, experimental results, and limitations.
HLA-WM: Hybrid Linear Attention for Long-Horizon Video World Models
HLA-WM retrieves past chunks using camera geometry to restore scene memory. At 60 seconds, its history memory is about 12× smaller than Full KV and 6× larger than GDN,…
Contextual Correction of Parallel Drafts for Speculative Decoding
DCOCO reselects existing parallel draft candidates using contextual scores. This review distinguishes accepted length from actual throughput and examines correction…
Towards Looped Models Done Right, Part II: Rethinking at Fixed Points
Fixed-point behavior connects KV sharing, state reuse, and distilled prefill in looped language models. We review the savings and accuracy trade-offs demonstrated on…
SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inference
SparseDecoding calibrates pruning on model-generated tokens and adds a decoding-specific sparse kernel. We examine the conditions behind its maximum 1.48× speedup on A100…
MC-Sparse: Deconstructing and Closing the Dense-Sparse Attention Gap in Diffusion Transformers
MC-Sparse separates three sources of quality and efficiency loss in sparse DiT attention. This review explains why token selection and kernel execution costs must be…
TokenRouter: Efficient Serving System for Token-Level LLM Routing
TokenRouter: An Efficient Serving System for Token-Level LLM Routing TL;DR — Routing an LLM at the token level rather than the session/query level can substantially push…
Mechanics of Long-Context Hybrid Models Part 1.1: From Hybrid Attention to Hybrid Position
Mechanics of Long-Context Hybrid Models: From Hybrid Attention to Hybrid Position TL;DR — These days open-source LLMs are rapidly shifting from models that "use full…
TRACE: Rollout-Guided Quantization-Aware Training for FP4 Reinforcement Learning of MoE Language Models
TRACE: Rollout-Aligned FP4 Quantization-Aware Training — MoE LLM RL at BF16 Quality, up to 5.4x Faster TL;DR In reinforcement learning (RL) for MoE LLMs, quantizing…
SlimKV: Joint Token-Feature KV Cache Compression with Reconstruction-Free Beacon Attention
SlimKV: Compressing Tokens and Features Together, Cutting the KV Cache by up to 32× with Reconstruction-Free Latent-Space Attention TL;DR SlimKV is a question-agnostic…
Tailoring the Quantization Space for 1-Bit KV Cache Compression
TaSQ: 'Tailored Design' of the Quantization Space for a 1-bit KV Cache TL;DR — TaSQ (Tailored Space Vector Quantization) redesigns the very target space to which vector…
Prefill-Free Cross-Family KV Cache Transfer for Heterogeneous Multi-Agent LLMs
How to Hand Off KV Caches As-Is Between Different LLMs: HeteroFold TL;DR: HeteroFold, a method that reuses the KV cache between models with different tokenizers and…
The KV Cache Is the New Memory Wall
The KV Cache Is the New Memory Wall: Three Regimes and Five Domains of Long-Context LLM Inference TL;DR — Long-context LLM decode inference is bound by memory bandwidth,…
WavePP: High-Throughput Pipeline Parallel LLM Prefill under Prefix Reuse
WavePP — A Pipeline-Parallel Prefill Runtime That Burns In Prefix Reuse TL;DR Built on top of TensorRT-LLM, WavePP overlaps request admission (reuse consensus, cache…
Periodic Weak Spots: Phase Sensitivity from Chunked KV-Cache Compression
Phase Sensitivity: The 'Periodic Weakness' Created by Chunked KV-Cache Compression TL;DR — In models that use chunked KV-cache compression (such as the DeepSeek-V4…
MassAlloc Attention: Let Attention Allocate Its Own Compute
Paper MassAlloc Attention (MALA): Letting Attention Allocate Its Own Compute TL;DR — Full softmax attention (FullAttn) performs all post-score computation after QK score…
WorldAttention: An Efficient Attention Architecture for Interactive Video World Models
WorldAttention: System Co-Design of an Interactive Video World Model That Drops the Sliding Window TL;DR — Text-controlled interactive video world models are trapped…
Block Sparse Attention with Log-Linear Complexity
PISA: Pyramid Top-$K$ Selection Lowers the Cost of Block-Sparse Attention to $O(N\log N)$ TL;DR — Block-sparse attention makes attention computation linear, but block…
GDN Tree-Scan: Served Tree Verification for Recurrent-Hybrid Language Models
LumoTree: Path-Parallel Speculative Verification for Hybrid Language Models TL;DR — Tree speculative decoding in hybrid models that mix Gated DeltaNet with full attention…
SPECTRA: Adaptive Execution of Speculative Decoding on a Runtime-Reconfigurable Tiled Architecture
SPECTRA: A Reconfigurable LLM Accelerator That Changes Its Hardware With Every Inference Stage TL;DR Speculative decoding oscillates between prefill (compute-bound GEMM),…
Adapting Tree-Structured Speculative Decoding to DeepSeek-V4 for Efficient Inference
Branching on Compressed Memory: Adding Tree Reasoning to DeepSeek-V4 TL;DR Adding tree-structured speculative decoding to DeepSeek-V4-Flash yields longer acceptance…
Flash-dLLM: IO-Aware KV Caching and Parallel Decoding for Fast, Memory-Efficient Diffusion LLMs
Flash-dLLM: I/O-First Diffusion LLM Acceleration, Unlocking 210.6 tokens/s with Fused Cache and Self-Verification One-Line Summary (TL;DR) Diffusion LLMs (dLLMs) support…
H-Spec: Parallel Speculative Decoding Without a Drafter-Side KV Cache
Parallel Inference without Drafter-Side KV Cache: H-Spec's Hybrid Speculative Decoding TL;DR shows that the bottleneck in concurrent serving is that a block-diffusion…
Analytical Power-Aware Provisioning for Prefill-Decode Disaggregated AI Inference
Analytical PD-inference provisioning uses instance counts and KV-cache reservations to find the lowest-power deployment for a target capacity.
VC-Attention: Value Smoothing and Softmax Casting for Low-bit Attention
VC-Attention is a training-free kernel that removes value-quantization and softmax bottlenecks in low-bit video-DiT attention, speeding it up 1.59× on B200.
DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression
DeepSeek-V4.1-Flash: A Multimodal MoE That Cuts the KV Cache to 1/4 While Raising Performance Further TL;DR. DeepSeek-V4.1-Flash, released by DeepSeek-AI, is a multimodal…
Fathom: Per-Query Read Depth for Sparse Decoding over Offloaded KV Caches
Fathom: Accelerating 1M-Token Decoding by 1.67x with Query-Adaptive Depth Scanning TL;DR When agent sessions stretch to 1M tokens and dozens of sessions reside…
The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction
Waking the MoE on SSD: Serving a 35B MoE at 20 tok/s on a 24 GB Desktop with Edge0 One-line summary (TL;DR) Even at 4-bit, a 35B-class MoE occupies 19.5 GB and cannot…
How Lossless Is Lossless Speculative Decoding? The Role of Numerical Precision in Orthrus
Is "Lossless" Speculative Decoding Really Lossless? — The Numerical-Precision Trap Exposed by Reproducing Orthrus TL;DR — Orthrus attaches a diffusion view to a frozen AR…
ZGCM-1: A Fully Open and Extremely Efficient Foundation Model for Math and Agentic Search
ZGCM-1: How a 7B Dense Model Trades Blows with 235B Frontiers — A Fully Open, Ultra-Efficient Foundation Model Design Report TL;DR — With a single 7.39B dense model,…
SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking
SAS: A Simple Attention Sparsification That Learns Context Ranking Directly Without Distillation TL;DR — In post-training attention sparsification, inject the continuous…
CoVeR: Coverage-Based Token Pruning for Multi-View 3D Reasoning in VLMs
CoVeR: Coverage-Based Pruning That Cuts Visual Tokens of Multi-View 3D Reasoning by up to 92% Using Geometry Alone One-Line Summary (TL;DR) Rendering a 3D scene into…
SQS: Bayesian DNN Compression through Sparse Quantized Sub-distributions
SQS: Fusing Pruning and Quantization into One Bayesian Learning — Where Spike-and-Slab Meets GMM TL;DR — Doing pruning and low-bit quantization separately leaves…
Kalman Delta Networks: Uncertainty-aware Associative Memory
Linear Attention Rewritten with a Kalman Filter: Kalman Delta Networks TL;DR The fixed-size recurrent memory update of linear attention is reinterpreted as a…
X-AuT: Progressive Audio-Encoder Compression for Speech LLMs with Cross-Scale Distillation
X-AuT: Pruning Audio Encoders in Speech LLMs with Behavioral Probes and Cross-Scale Distillation TL;DR X-AuT is a framework that progressively prunes the audio encoder…
Deadline-Aware Adaptive Prefill Chunking for Efficient Large Language Model Serving
Deadline-Filling Prefill Chunks: SLOWeave — An Adaptive Chunking Scheduler for LLM Serving TL;DR — Instead of a fixed prefill chunk size, we propose SLOWeave, an online…
Why Is Video Still So Expensive? A Survey of Inference-Efficiency Mechanisms in Video and Audiovisual LLMs
Why Are Video LLMs Still So Expensive? — A Four-Stage Guide to Inference Efficiency Mechanisms TL;DR — This paper classifies 125 VideoLLM inference efficiency studies…
![[Paper Review] Llama-Nemotron: Efficient Reasoning Models](https://cdn-thumbnails.huggingface.co/social-thumbnails/collections/nvidia/llama-nemotron-67d92346030a2691293f200b.png)
[Paper Review] Llama-Nemotron: Efficient Reasoning Models
Hydragen: The Secret Weapon for Decoding Large Batches with Shared Prefixes up to 32× Faster TL;DR By decomposing the prefix and suffix using softmax denominator…