Hardware-Aware FP4 FlashAttention-4
Paper Why Don’t FP4 Tensor Cores Make Attention Faster? Blackwell FlashAttention-4, Solved with Direct-P and Quantized BackpropTL;DR — …
34 min
FlashAttention
Machine Learning
FlashAttention
5 pages
![[Paper Review] Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://icml.cc/media/PosterPDFs/ICML%202024/32613.png)
[Paper Review] Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
Paper Link Structured State Space Duality: Unifying SSMs and Attention with Mamba-2 for 2–8× AccelerationTL;DRStructured State-Space Duality …
19 min
Mamba
SSM
Transformer
Long Context
Efficient Training
FlashAttention
GPU Acceleration
![[Paper Review] Llama-Nemotron: Efficient Reasoning Models](https://cdn-thumbnails.huggingface.co/social-thumbnails/collections/nvidia/llama-nemotron-67d92346030a2691293f200b.png)
[Paper Review] Llama-Nemotron: Efficient Reasoning Models
Paper Link Hydragen: The Secret Weapon for Decoding Large Batches with Shared Prefixes up to 32× FasterTL;DRBy decomposing the prefix and …
25 min
Prefix Caching
Efficient Inference
Attention Optimization
FlashAttention
vLLM
![[Paper Review] Helix Parallelism: Rethinking Sharding Strategies for Interactive Multi-Million-Token LLM Decoding](https://www.storagereview.com/wp-content/uploads/2025/07/image2-2-png-e1752234784623.webp)
[Paper Review] Helix Parallelism: Rethinking Sharding Strategies for Interactive Multi-Million-Token LLM Decoding
Paper Link Helix Parallelism: Breaking the Latency-Throughput Wall of Ultra-Long LLM DecodingTL;DRHelix Parallelism schedules Attention and …
17 min
Tensor Parallelism
Mixture of Experts
FlashAttention
Long Context Inference
Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
Paper Native Sparse Attention (NSA) — 11× faster even at 64k tokens, accuracy intactOne-line summary (TL;DR)NSA combines a three-branch …
36 min
Long Context
Transformer Optimization
GPU Acceleration
FlashAttention
Memory Efficiency
DeepSeek