Hardware-Aware FP4 FlashAttention-4

Mon, 07 Sep 2026 00:00:00 +0900

Paper

Why Don’t FP4 Tensor Cores Make Attention Faster? Blackwell FlashAttention-4, Solved with Direct-P and Quantized Backprop

TL;DR — Blackwell’s FP4 tensor cores handle matrix multiplication far faster than BF16, but attention does not automatically inherit that benefit, because a “middle operation” — softmax — sits between the two matrix multiplications. This paper shortens the critical path with Direct-P, which reframes softmax probability generation not as a sequential “accurate exponential → round” path but as a problem of directly classifying scores into E2M1 codes, achieving up to 2.13× forward throughput over BF16 on an NVIDIA GB200 (source: §Abstract). For training, backward reuses the quantization state that forward produced to speed a single step of an 8B model by up to 1.14×, but lowering P/V to MXFP4 makes every trajectory diverge, so P/V must be kept in FP8 (source: §7.6).

Why Gated DeltaNet Survives 4-Bit Quantization: NVFP4 W4A4 for the Recurrent Half of a Hybrid 27B LLM

Mon, 07 Sep 2026 00:00:00 +0900

Paper

The Recurrent Half Is the Easy-to-Quantize Half: Why Gated DeltaNet Survives at 4 Bits

TL;DR — The 48 recurrent Gated DeltaNet (GDN) layers of a hybrid 27B LLM (Qwen3.8-27B) have until now been protected at 8–16 bits, on the intuition that “error accumulates across the context.” This paper shows that intuition is exactly backwards. Minima, which quantizes all 496 layers — GDN gates included — to NVFP4 W4A4, matches BF16 within seed noise on 6 benchmarks (5-task average −0.52) while also being the smallest (17.5 GiB) and the fastest at prefill (+14–19%). On top of that, a four-stage mechanistic study delivers an architecture-level account of why — block scaling localizes outliers, gate nonlinearity compresses noise, and the delta rule actively erases state error.

Quantization on Jaehun's Blog

Hardware-Aware FP4 FlashAttention-4

Why Don’t FP4 Tensor Cores Make Attention Faster? Blackwell FlashAttention-4, Solved with Direct-P and Quantized Backprop

Why Gated DeltaNet Survives 4-Bit Quantization: NVFP4 W4A4 for the Recurrent Half of a Hybrid 27B LLM

The Recurrent Half Is the Easy-to-Quantize Half: Why Gated DeltaNet Survives at 4 Bits