TAO: Tool-Aware Joint Scheduling for LLM-Based Agentic Serving
TAO: Tool-Aware Joint Scheduling for LLM-Based Agentic Serving Original: Shuhuai Li et al., "TAO: Tool-Aware Joint Scheduling for LLM-Based Agentic Serving"…
LLM Inference
25 pages
On the SPOT: Enhancing Speculative Decoding for Diffusion Language Models via Commit-Order Prediction
Diffusion language models must decide 'what' and 'where' at the same time Diffusion language models generate sentences by repeatedly filling masked tokens. To decode…
Channel Selection for Mixture-of-Experts Decoding Needs No Calibration Data
Introduction In settings where the batch size is close to 1, such as conversational services, the latency of a mixture-of-experts language model comes mostly from reading…
Extreme Compression of Per-Layer Embeddings via Cross-Layer Latent Vector Quantization
Recent language models keep increasing the amount of information packed into a single vocabulary entry. Per-layer embeddings (PLE), first introduced in Gemma 3n, carried…
Large Flow Language Models
The sequential bottleneck of sampling one token at a time Large language models can compute an entire sentence in parallel during training, but when generating text they…
Cache Engineering for Agents
The Cost and Accuracy Dilemma of Long Agent Trajectories LLM agents accumulate long trajectories by repeating reasoning and tool calls for hundreds of rounds. To make…
DLite: History-free Speculative Decoding for Agentic AI
DLite explained through a candidate-verification example, cache ownership, and a careful comparison of acceptance length and decoding speed.
Trajectory-Level Speculation with Semantic Verification for Tool-Using Agents
An analysis of SpecAgent, where a weaker model drafts tool calls and a stronger model verifies them, covering its design, experimental results, and limitations.
Contextual Correction of Parallel Drafts for Speculative Decoding
DCOCO reselects existing parallel draft candidates using contextual scores. This review distinguishes accepted length from actual throughput and examines correction…
SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inference
SparseDecoding calibrates pruning on model-generated tokens and adds a decoding-specific sparse kernel. We examine the conditions behind its maximum 1.48× speedup on A100…
TokenRouter: Efficient Serving System for Token-Level LLM Routing
TokenRouter: An Efficient Serving System for Token-Level LLM Routing TL;DR — Routing an LLM at the token level rather than the session/query level can substantially push…
Prefill-Free Cross-Family KV Cache Transfer for Heterogeneous Multi-Agent LLMs
How to Hand Off KV Caches As-Is Between Different LLMs: HeteroFold TL;DR: HeteroFold, a method that reuses the KV cache between models with different tokenizers and…
WavePP: High-Throughput Pipeline Parallel LLM Prefill under Prefix Reuse
WavePP — A Pipeline-Parallel Prefill Runtime That Burns In Prefix Reuse TL;DR Built on top of TensorRT-LLM, WavePP overlaps request admission (reuse consensus, cache…
Who Pays for the KV Cache? Attributing Shared AI Inference Spend Across Kubernetes and LLM Provider Bills
Who pays for the KV cache? — Kubernetes, gateway, and provider bills in one ledger TL;DR — The cost of a single AI feature is scattered across three disconnected ledgers:…
SPECTRA: Adaptive Execution of Speculative Decoding on a Runtime-Reconfigurable Tiled Architecture
SPECTRA: A Reconfigurable LLM Accelerator That Changes Its Hardware With Every Inference Stage TL;DR Speculative decoding oscillates between prefill (compute-bound GEMM),…
Adapting Tree-Structured Speculative Decoding to DeepSeek-V4 for Efficient Inference
Branching on Compressed Memory: Adding Tree Reasoning to DeepSeek-V4 TL;DR Adding tree-structured speculative decoding to DeepSeek-V4-Flash yields longer acceptance…
Flash-dLLM: IO-Aware KV Caching and Parallel Decoding for Fast, Memory-Efficient Diffusion LLMs
Flash-dLLM: I/O-First Diffusion LLM Acceleration, Unlocking 210.6 tokens/s with Fused Cache and Self-Verification One-Line Summary (TL;DR) Diffusion LLMs (dLLMs) support…
Analytical Power-Aware Provisioning for Prefill-Decode Disaggregated AI Inference
Analytical PD-inference provisioning uses instance counts and KV-cache reservations to find the lowest-power deployment for a target capacity.
Fathom: Per-Query Read Depth for Sparse Decoding over Offloaded KV Caches
Fathom: Accelerating 1M-Token Decoding by 1.67x with Query-Adaptive Depth Scanning TL;DR When agent sessions stretch to 1M tokens and dozens of sessions reside…
The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction
Waking the MoE on SSD: Serving a 35B MoE at 20 tok/s on a 24 GB Desktop with Edge0 One-line summary (TL;DR) Even at 4-bit, a 35B-class MoE occupies 19.5 GB and cannot…
How Lossless Is Lossless Speculative Decoding? The Role of Numerical Precision in Orthrus
Is "Lossless" Speculative Decoding Really Lossless? — The Numerical-Precision Trap Exposed by Reproducing Orthrus TL;DR — Orthrus attaches a diffusion view to a frozen AR…
CoVeR: Coverage-Based Token Pruning for Multi-View 3D Reasoning in VLMs
CoVeR: Coverage-Based Pruning That Cuts Visual Tokens of Multi-View 3D Reasoning by up to 92% Using Geometry Alone One-Line Summary (TL;DR) Rendering a 3D scene into…
Deadline-Aware Adaptive Prefill Chunking for Efficient Large Language Model Serving
Deadline-Filling Prefill Chunks: SLOWeave — An Adaptive Chunking Scheduler for LLM Serving TL;DR — Instead of a fixed prefill chunk size, we propose SLOWeave, an online…
[Paper Review] Towards Efficient and Practical GPU Multitasking in the Era of LLM
"Let's put an OS on the GPU": A proposal for a GPU multitasking OS layer for the LLM era One-line summary (TL;DR) In LLM serving, GPUs often stay at ≤10% utilization…
![[Paper Review] SGLang: Efficient Execution of Structured Language Model Programs](https://cdn.bytez.com/mobilePapers/v2/neurips/94872/images/20-0.png)
[Paper Review] SGLang: Efficient Execution of Structured Language Model Programs
SGLang & RadixAttention: How Execution Optimization for "LM Programs" Achieved a 6.4x Speedup TL;DR By combining a language-runtime co-design with RadixAttention (radix…