Hardware-Aware FP4 FlashAttention-4

Mon, 07 Sep 2026 00:00:00 +0900

Paper

Why Don’t FP4 Tensor Cores Make Attention Faster? Blackwell FlashAttention-4, Solved with Direct-P and Quantized Backprop

TL;DR — Blackwell’s FP4 tensor cores handle matrix multiplication far faster than BF16, but attention does not automatically inherit that benefit, because a “middle operation” — softmax — sits between the two matrix multiplications. This paper shortens the critical path with Direct-P, which reframes softmax probability generation not as a sequential “accurate exponential → round” path but as a problem of directly classifying scores into E2M1 codes, achieving up to 2.13× forward throughput over BF16 on an NVIDIA GB200 (source: §Abstract). For training, backward reuses the quantization state that forward produced to speed a single step of an 8B model by up to 1.14×, but lowering P/V to MXFP4 makes every trajectory diverge, so P/V must be kept in FP8 (source: §7.6).

2609.04105v1 on Jaehun's Blog

Hardware-Aware FP4 FlashAttention-4

Why Don’t FP4 Tensor Cores Make Attention Faster? Blackwell FlashAttention-4, Solved with Direct-P and Quantized Backprop