Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
Paper Native Sparse Attention (NSA) — 11× faster even at 64k tokens, accuracy intactOne-line summary (TL;DR)NSA combines a three-branch …
36 min
Long Context
Transformer Optimization
GPU Acceleration
FlashAttention
Memory Efficiency
DeepSeek