Transformer Optimization

2 pages

[paper review] SageAttention3: Microscaling FP4 Attention for Inference and An Exploration of 8-bit Training

[paper review] SageAttention3: Microscaling FP4 Attention for Inference and An Exploration of 8-bit Training

[Paper Review] SageAttention 3 & SageBwd — FP4-Powered Inference and 8-bit TrainingPaper link: https://arxiv.org/abs/2505.11594v1

Transformer Optimization Inference Acceleration TrainingEfficiency CUDA Triton

Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention

Paper Native Sparse Attention (NSA) — 11× faster even at 64k tokens, accuracy intactOne-line summary (TL;DR)NSA combines a three-branch …

Long Context Transformer Optimization GPU Acceleration FlashAttention Memory Efficiency DeepSeek