Channel Selection for Mixture-of-Experts Decoding Needs No Calibration Data
Introduction In settings where the batch size is close to 1, such as conversational services, the latency of a mixture-of-experts language model comes mostly from reading…
GPU Acceleration
9 pages
Matrix-Game 4.0: Harnessing Video Models for Real-Time Playable Game Worlds
Introduction A game world built by a video generation model alone stores off-screen state nowhere. Every time a corridor you have already passed through is redrawn, it…
Speculative Sampling in Interactive Video Generation
Speculative sampling reuses future video predictions to reduce response latency. How it works, recomputation costs, and the limits of the serving experiments.
SparseDecoding: Decoding-Aware Pruning for Accurate and Efficient LLM Inference
SparseDecoding calibrates pruning on model-generated tokens and adds a decoding-specific sparse kernel. We examine the conditions behind its maximum 1.48× speedup on A100…
MC-Sparse: Deconstructing and Closing the Dense-Sparse Attention Gap in Diffusion Transformers
MC-Sparse separates three sources of quality and efficiency loss in sparse DiT attention. This review explains why token selection and kernel execution costs must be…
Analytical Power-Aware Provisioning for Prefill-Decode Disaggregated AI Inference
Analytical PD-inference provisioning uses instance counts and KV-cache reservations to find the lowest-power deployment for a target capacity.
VC-Attention: Value Smoothing and Softmax Casting for Low-bit Attention
VC-Attention is a training-free kernel that removes value-quantization and softmax bottlenecks in low-bit video-DiT attention, speeding it up 1.59× on B200.
![[Paper Review] Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://icml.cc/media/PosterPDFs/ICML%202024/32613.png)
[Paper Review] Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
Structured State Space Duality: Unifying SSMs and Attention with Mamba-2 for 2–8× Acceleration TL;DR Structured State-Space Duality (SSD) reveals that SSMs and masked…
Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
Native Sparse Attention (NSA) — 11× faster even at 64k tokens, accuracy intact One-line summary (TL;DR) NSA combines a three-branch sparse attention — “compress → select…