TAO: Tool-Aware Joint Scheduling for LLM-Based Agentic Serving
TAO: Tool-Aware Joint Scheduling for LLM-Based Agentic Serving Original: Shuhuai Li et al., "TAO: Tool-Aware Joint Scheduling for LLM-Based Agentic Serving"…
Large Language Models
29 pages
Extreme Compression of Per-Layer Embeddings via Cross-Layer Latent Vector Quantization
Recent language models keep increasing the amount of information packed into a single vocabulary entry. Per-layer embeddings (PLE), first introduced in Gemma 3n, carried…
Rethinking Sequence Compression with Test-time Regression
Introduction In models that handle long contexts, the compression operation that reduces multiple tokens to a single vector becomes a performance bottleneck. This paper…
Large Flow Language Models
The sequential bottleneck of sampling one token at a time Large language models can compute an entire sentence in parallel during training, but when generating text they…
Trajectory-Level Speculation with Semantic Verification for Tool-Using Agents
An analysis of SpecAgent, where a weaker model drafts tool calls and a stronger model verifies them, covering its design, experimental results, and limitations.
Contextual Correction of Parallel Drafts for Speculative Decoding
DCOCO reselects existing parallel draft candidates using contextual scores. This review distinguishes accepted length from actual throughput and examines correction…
Towards Looped Models Done Right, Part II: Rethinking at Fixed Points
Fixed-point behavior connects KV sharing, state reuse, and distilled prefill in looped language models. We review the savings and accuracy trade-offs demonstrated on…
SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inference
SparseDecoding calibrates pruning on model-generated tokens and adds a decoding-specific sparse kernel. We examine the conditions behind its maximum 1.48× speedup on A100…
TokenRouter: Efficient Serving System for Token-Level LLM Routing
TokenRouter: An Efficient Serving System for Token-Level LLM Routing TL;DR — Routing an LLM at the token level rather than the session/query level can substantially push…
Mechanics of Long-Context Hybrid Models Part 1.1: From Hybrid Attention to Hybrid Position
Mechanics of Long-Context Hybrid Models: From Hybrid Attention to Hybrid Position TL;DR — These days open-source LLMs are rapidly shifting from models that "use full…
TRACE: Rollout-Guided Quantization-Aware Training for FP4 Reinforcement Learning of MoE Language Models
TRACE: Rollout-Aligned FP4 Quantization-Aware Training — MoE LLM RL at BF16 Quality, up to 5.4x Faster TL;DR In reinforcement learning (RL) for MoE LLMs, quantizing…
SlimKV: Joint Token-Feature KV Cache Compression with Reconstruction-Free Beacon Attention
SlimKV: Compressing Tokens and Features Together, Cutting the KV Cache by up to 32× with Reconstruction-Free Latent-Space Attention TL;DR SlimKV is a question-agnostic…
Tailoring the Quantization Space for 1-Bit KV Cache Compression
TaSQ: 'Tailored Design' of the Quantization Space for a 1-bit KV Cache TL;DR — TaSQ (Tailored Space Vector Quantization) redesigns the very target space to which vector…
Prefill-Free Cross-Family KV Cache Transfer for Heterogeneous Multi-Agent LLMs
How to Hand Off KV Caches As-Is Between Different LLMs: HeteroFold TL;DR: HeteroFold, a method that reuses the KV cache between models with different tokenizers and…
On-Policy or Off-Policy Learning? A Systematic Study of Distillation Dynamics
On-Policy Is Not a Panacea: The Real Protagonists of Rollout Policy, KL Direction, and Learning Rate in Distillation Dynamics TL;DR — When we separate on-policy (OnPD)…
Periodic Weak Spots: Phase Sensitivity from Chunked KV-Cache Compression
Phase Sensitivity: The 'Periodic Weakness' Created by Chunked KV-Cache Compression TL;DR — In models that use chunked KV-cache compression (such as the DeepSeek-V4…
Block Sparse Attention with Log-Linear Complexity
PISA: Pyramid Top-$K$ Selection Lowers the Cost of Block-Sparse Attention to $O(N\log N)$ TL;DR — Block-sparse attention makes attention computation linear, but block…
GDN Tree-Scan: Served Tree Verification for Recurrent-Hybrid Language Models
LumoTree: Path-Parallel Speculative Verification for Hybrid Language Models TL;DR — Tree speculative decoding in hybrid models that mix Gated DeltaNet with full attention…
Flash-dLLM: IO-Aware KV Caching and Parallel Decoding for Fast, Memory-Efficient Diffusion LLMs
Flash-dLLM: I/O-First Diffusion LLM Acceleration, Unlocking 210.6 tokens/s with Fused Cache and Self-Verification One-Line Summary (TL;DR) Diffusion LLMs (dLLMs) support…
H-Spec: Parallel Speculative Decoding Without a Drafter-Side KV Cache
Parallel Inference without Drafter-Side KV Cache: H-Spec's Hybrid Speculative Decoding TL;DR shows that the bottleneck in concurrent serving is that a block-diffusion…
DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression
DeepSeek-V4.1-Flash: A Multimodal MoE That Cuts the KV Cache to 1/4 While Raising Performance Further TL;DR. DeepSeek-V4.1-Flash, released by DeepSeek-AI, is a multimodal…
StepAudio 3 Gen Technical Report
StepAudio 3 Gen: Unifying Speech, Song, Music, and Sound Effects with a Single Discrete Autoregressive (RVQ) Model TL;DR StepAudio 3 Gen, released by StepFun, is a fully…
SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking
SAS: A Simple Attention Sparsification That Learns Context Ranking Directly Without Distillation TL;DR — In post-training attention sparsification, inject the continuous…
SQS: Bayesian DNN Compression through Sparse Quantized Sub-distributions
SQS: Fusing Pruning and Quantization into One Bayesian Learning — Where Spike-and-Slab Meets GMM TL;DR — Doing pruning and low-bit quantization separately leaves…
X-AuT: Progressive Audio-Encoder Compression for Speech LLMs with Cross-Scale Distillation
X-AuT: Pruning Audio Encoders in Speech LLMs with Behavioral Probes and Cross-Scale Distillation TL;DR X-AuT is a framework that progressively prunes the audio encoder…
Why Is Video Still So Expensive? A Survey of Inference-Efficiency Mechanisms in Video and Audiovisual LLMs
Why Are Video LLMs Still So Expensive? — A Four-Stage Guide to Inference Efficiency Mechanisms TL;DR — This paper classifies 125 VideoLLM inference efficiency studies…
Miles v0.1: Production-Level Post-Training
Miles v0.1: A "verified, clean, and scalable" full-stack system for frontier RL post-training TL;DR: Miles is a full-stack RL post-training framework that resolves the…
[Paper Review] Peri-LN: Revisiting Normalization Layer in the Transformer Architecture
Peri-LayerNorm: A Third Option Beyond Post-LN and Pre-LN TL;DR By simply adding another LayerNorm right after the residual connection—"Peri-LayerNorm (Peri-LN)"—the…
Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling
Janus-Pro 7B: Dual-Encoder Multimodal LLM That Outsmarts Bigger Models One-line summary (TL;DR) By fully separating the SigLIP understanding encoder from the VQ…