Speculative Sampling in Interactive Video Generation
Speculative sampling reuses future video predictions to reduce response latency. How it works, recomputation costs, and the limits of the serving experiments.
Inference Acceleration
9 pages
DLite: History-free Speculative Decoding for Agentic AI
DLite explained through a candidate-verification example, cache ownership, and a careful comparison of acceptance length and decoding speed.
Trajectory-Level Speculation with Semantic Verification for Tool-Using Agents
An analysis of SpecAgent, where a weaker model drafts tool calls and a stronger model verifies them, covering its design, experimental results, and limitations.
Contextual Correction of Parallel Drafts for Speculative Decoding
DCOCO reselects existing parallel draft candidates using contextual scores. This review distinguishes accepted length from actual throughput and examines correction…
SPECTRA: Adaptive Execution of Speculative Decoding on a Runtime-Reconfigurable Tiled Architecture
SPECTRA: A Reconfigurable LLM Accelerator That Changes Its Hardware With Every Inference Stage TL;DR Speculative decoding oscillates between prefill (compute-bound GEMM),…
Deadline-Aware Adaptive Prefill Chunking for Efficient Large Language Model Serving
Deadline-Filling Prefill Chunks: SLOWeave — An Adaptive Chunking Scheduler for LLM Serving TL;DR — Instead of a fixed prefill chunk size, we propose SLOWeave, an online…
Online Draft Co-Training for Speculative Decoding in Large-Scale, Long-Context RL Post-Training
Two Designs That Make Speculative Decoding Practical for Large-Scale, Long-Context RL Post-Training TL;DR : Rollout generation dominates the wall-clock time of RL…
![[Paper Review] Hardware-Efficient Attention for Fast Decoding](https://moonlight-paper-snapshot.s3.ap-northeast-2.amazonaws.com/arxiv/hardware-efficient-attention-for-fast-decoding-1.png)
[Paper Review] Hardware-Efficient Attention for Fast Decoding
GTA & GLA: Hardware-Efficient Attention That Breaks the 'Memory-Dominated' Decode TL;DR GTA (key–value tying) and GLA (latent head sharding) raise FLOPs/byte and shrink…
![[paper review] SageAttention3: Microscaling FP4 Attention for Inference and An Exploration of 8-bit Training](https://cdn-uploads.huggingface.co/production/uploads/66c0a08bac74db25de8427ec/Tb20E3IJSV6PjcD9Nkvfg.png)
[paper review] SageAttention3: Microscaling FP4 Attention for Inference and An Exploration of 8-bit Training
\[Paper Review] SageAttention 3 & SageBwd — FP4-Powered Inference and 8-bit Training Paper link: https://arxiv.org/abs/2505.11594v1 📝 TL;DR The SageAttention 3…