FlashMask-3: Efficient and Expressive Mask-Aware Distributed Attention
An analysis of FlashMask-3’s Hopper kernels, mask localization, load balancing, communication overlap, and long-context training results and limitations.
FlashAttention
8 pages
Flash-dLLM: IO-Aware KV Caching and Parallel Decoding for Fast, Memory-Efficient Diffusion LLMs
Flash-dLLM: I/O-First Diffusion LLM Acceleration, Unlocking 210.6 tokens/s with Fused Cache and Self-Verification One-Line Summary (TL;DR) Diffusion LLMs (dLLMs) support…
VC-Attention: Value Smoothing and Softmax Casting for Low-bit Attention
VC-Attention is a training-free kernel that removes value-quantization and softmax bottlenecks in low-bit video-DiT attention, speeding it up 1.59× on B200.
Hardware-Aware FP4 FlashAttention-4
Why Don't FP4 Tensor Cores Make Attention Faster? Blackwell FlashAttention-4, Solved with Direct-P and Quantized Backprop TL;DR — Blackwell's FP4 tensor cores handle…
![[Paper Review] Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://icml.cc/media/PosterPDFs/ICML%202024/32613.png)
[Paper Review] Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
Structured State Space Duality: Unifying SSMs and Attention with Mamba-2 for 2–8× Acceleration TL;DR Structured State-Space Duality (SSD) reveals that SSMs and masked…
![[Paper Review] Llama-Nemotron: Efficient Reasoning Models](https://cdn-thumbnails.huggingface.co/social-thumbnails/collections/nvidia/llama-nemotron-67d92346030a2691293f200b.png)
[Paper Review] Llama-Nemotron: Efficient Reasoning Models
Hydragen: The Secret Weapon for Decoding Large Batches with Shared Prefixes up to 32× Faster TL;DR By decomposing the prefix and suffix using softmax denominator…
![[Paper Review] Helix Parallelism: Rethinking Sharding Strategies for Interactive Multi-Million-Token LLM Decoding](https://www.storagereview.com/wp-content/uploads/2025/07/image2-2-png-e1752234784623.webp)
[Paper Review] Helix Parallelism: Rethinking Sharding Strategies for Interactive Multi-Million-Token LLM Decoding
Helix Parallelism: Breaking the Latency-Throughput Wall of Ultra-Long LLM Decoding TL;DR Helix Parallelism schedules Attention and FFN with different parallelism…
Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
Native Sparse Attention (NSA) — 11× faster even at 64k tokens, accuracy intact One-line summary (TL;DR) NSA combines a three-branch sparse attention — “compress → select…