Hardware-Aware FP4 FlashAttention-4

Paper

Why Don’t FP4 Tensor Cores Make Attention Faster? Blackwell FlashAttention-4, Solved with Direct-P and Quantized Backprop

TL;DR — Blackwell’s FP4 tensor cores handle matrix multiplication far faster than BF16, but attention does not automatically inherit that benefit, because a “middle operation” — softmax — sits between the two matrix multiplications. This paper shortens the critical path with Direct-P, which reframes softmax probability generation not as a sequential “accurate exponential → round” path but as a problem of directly classifying scores into E2M1 codes, achieving up to 2.13× forward throughput over BF16 on an NVIDIA GB200 (source: §Abstract). For training, backward reuses the quantization state that forward produced to speed a single step of an 8B model by up to 1.14×, but lowering P/V to MXFP4 makes every trajectory diverge, so P/V must be kept in FP8 (source: §7.6).

Core idea

The paper’s central claim fits in one sentence.

Attention’s bottleneck is no longer the matrix multiplications but the softmax middle stage that turns scores into probabilities, and fixing it requires not more accurate approximations but removing work from the critical path. FP4 probabilities should therefore be produced by “code classification” rather than “compute the exponential, then round”, and the normalization should use the very rounded values that PV actually consumes (source: §3, §4).

The core insight boils down to two linked problems (source: §3).

1. A timing problem — P is produced inside the kernel, so the first valid PV operation must wait on the sequential chain S → max → scale → probability → E2M1 pack → expose. FP4 makes QK and PV fast but does not accelerate score reduction, exponentiation, synchronization, or scale exposure.
2. A numeric-range problem — softmax probabilities live in a range that is awkward to carry in a 4-bit payload with a block scale. Preserving the range is expensive; cheapening it underflows whole blocks to zero.

The paper’s attitude is to not hide the fact that “the fastest FP4 operating point accepts larger error” and to measure the speed–accuracy tradeoff honestly (source: §5.1, §8.5).

Background: the problem they set out to solve

Attention is “two matrix multiplications + one softmax”

Attention is built from queries $Q$, keys $K$, and values $V$:

$$ S = QK^T / \sqrt{d}, \qquad P = \mathrm{softmax}(S), \qquad O = PV \tag{1} $$

That is, a softmax sits between the two matrix multiplications, the QK product and the PV product. FlashAttention evaluates these quadratic-size matrices $S$ and $P$ tile by tile instead of storing them in HBM (source: §1). FlashAttention-4 (FA4) re-targeted that tiled algorithm at Blackwell’s asynchronous matrix hardware (source: §1).

On Blackwell, FP4 only accelerates matrix multiplication

Blackwell’s FP4 matrix multiplication is much faster than BF16. But softmax must still reduce each tile of scores, evaluate the exponential, build a scaled tile of probabilities, and get it ready for the second product. The faster the matrix multiplication gets, the more this middle stage becomes the bottleneck (source: §1).

The key observation the authors cite is this: “removing work that happens after the publication does not reduce latency. You must remove work that happens before the first valid P tile” (source: §3.1).

Two FP4 formats and their tradeoffs

FP4 uses an E2M1 (2 exponent bits, 1 mantissa bit) payload. Ignoring the sign, the representable magnitudes are:

$$ F_{E2M1} = {0,\ \tfrac12,\ 1,\ \tfrac32,\ 2,\ 3,\ 4,\ 6} \tag{10} $$

How this payload is scaled is what separates the formats (source: §3.2, Table 2):

The key difference is range. When NVFP4’s E4M3 scale rounds to zero, the whole block vanishes (underflow). MXFP4’s E8M0 has power-of-two amplitudes, so even tiny probability blocks can be represented without a separate row shift (source: §3.2). Running precision diagnostics on Gaussian softmax probabilities in Table 3, the authors show that stabilized NVFP4 gives the highest fidelity but needs a per-row range correction, while MXFP4 gives power-of-two scales that align exactly with the N32 production fragments but lays down probabilities more coarsely (source: §3.2).

So Direct-P chooses NVFP4 for Q/K and MXFP4 for P/V.

The new approach: Direct-P

Direct-P’s boundary is narrow and precise. It keeps the external schedule that HAO AI Lab’s FP4 FA4 implementation provides (two-query pipeline, TMEM lifetime, issue protocol) as is, and changes only the segment between “a finished FP32 tile of scores” and “a legal FP4 probability operand for PV to consume” (source: §4.1, Table 4).

Direct-P consists of three linked choices (source: §4.1):

1. Map normalized scores directly to an E2M1 payload → shortens the critical path (fixes the timing problem)
2. Compute the normalization denominator from the very rounded payload that PV consumes → the numerator and denominator describe a single approximation operator (numeric consistency)
3. Apply range guards only on layers with extreme logits → most layers keep the shiftless path

Fix 1: classify scores directly into E2M1 codes

The standard path computes a relatively accurate exponential and then rounds to one of the 8 E2M1 magnitudes. That intermediate precision never reaches PV. Direct-P inverts this, treating probability generation as a code-classification problem: it only decides which E2M1 bin each normalized score falls into (source: §4.2).

The E2M1 positive codes change value at seven rounding boundaries ${\frac14,\frac34,\frac54,\frac74,\frac52,\frac72,5}$. The authors therefore fit an affine classifier in value space:

$$ \hat u(x) = \max(0,, Ax + B), \qquad \hat q(x) = Q_{E2M1}(\hat u(x)) \tag{17} $$

The fitting objective is not the accuracy of the real-valued exponential but the E2M1 code-match rate. The score transform and the $e_B$ and $\log_2 6$ terms are folded into packed FMA coefficients (source: §4.2). The general fast fit uses $A{=}1.50,\ B{=}1.20$; the Wan-model activation evaluation uses $A{=}1.60,\ B{=}0.95$ (source: §4.2). Two-lane FMAs (FFMA2) and the native conversion (F2FP) emit the E2M1 payload, and selected locations can use Blackwell’s native base-2 exponent instruction EX2 (source: §4.2).

Fix 2: normalize the “represented probabilities”

If the numerator (the output) uses rounded FP4 probabilities, then using an independently approximated FP32 sum of exponentials as the denominator mixes two different operators. Direct-P accumulates the denominator from the same codes and block scale that PV actually consumes (source: §4.3):

$$ \tilde N_{iB} = \frac{\alpha_B}{6}\sum_{j\in B} q_{ij}\hat V_j, \qquad \tilde L_{iB} = \frac{\alpha_B}{6}\sum_{j\in B} q_{ij}, \qquad \tilde O_i = \frac{\sum_B \tilde N_{iB}}{\sum_B \tilde L_{iB}} \tag{19–21} $$

In other words, the numerator and denominator describe exactly the same represented operator. Four packed payload words are reduced with byte permutes and DP4A (4-way integer dot product) (source: §4.3).

Fix 3: guard only the extreme-logit layers

The fast shiftless path stays finite on synthetic grids and on most layers, but some late Wan layers see BF16 logits exceeding 500, even 1000. Scanning and reloading every score would erase the speed advantage. So only those layers are routed through Algorithm 3 (source: §4.4).

The real cause of the original layer-39 failure was not a missed anchor: the expression $(s/6)\sum_i c_i$ produced a subnormal intermediate at E8M0 code 1 and was flushed to zero. Algorithm 3 re-associates it as $s,(\sum_i c_i/6)$, resolving it without a second scan, a new barrier, or a stable-softmax fallback (source: §4.4).

Two operating points

(source: Table 5)

How it works: a concrete example

Let’s trace the core algorithm through one small example (source: §4.2, Algorithm 2). For clarity, we assume a block of 4 keys.

Standard path vs. Direct-P path

The standard path is serial:

  flowchart LR
    Z[score z] --> M[row max m]
    M --> E[exponent exp z-m]
    E --> A[block max aB]
    A --> S[encode scale s]
    S --> D[division]
    D --> Q[E2M1 pack]
    Q --> P[expose to PV]

Direct-P compresses that chain into a single classification:

  flowchart LR
    Z[score z] --> X["x = (z-m)·log2e − eB + log2 6"]
    X --> U["u = max(0, Ax+B)"]
    U --> Q["q = Q_E2M1(u)"]
    Q --> P[expose to PV]
    Q --> L["denominator = αB·Σq/6  (uses the same q)"]

A worked numeric example

For the 4 keys of one query row, suppose the exponents of the scores normalized by subtracting the row max $m$ are:

Here the probabilities land exactly on the E2M1 grid, so the error is 0. The block magnitude is $\alpha_B = 1$ ($e_B = 0$) and the reconstruction step is $\delta_B = 1/6$.

The standard path computes $\exp$ accurately and then rounds to obtain $q$; Direct-P instead forms $x = (z_j - m)\log_2 e + \log_2 6$, pushes it through the affine map $u = \max(0, Ax+B)$, and directly emits $q = Q_{E2M1}(u)$. The “accurate exponential” intermediate never reaches PV anyway, so Direct-P simply never builds it (source: §4.2).

The denominator also accumulates with the same $q$: $\sum_j q_j = 6 + 4 + 2 + 1 = 13$, hence $\tilde L = 13/6$. This matches the exact denominator $\sum \exp = 1 + 2/3 + 1/3 + 1/6 = 13/6$. Because the numerator and denominator use the same rounded values, the approximate operator is a single internally consistent operator (source: §4.3).

Why the critical path shrinks

The crux is that in HAO’s two-query schedule, PV must wait until two adjacent N32 fragments (i.e., K64) are complete (source: §2.5). Direct-P reduces the upstream work needed to produce those two fragments, so the first legal K64 operand appears sooner. Conversely, removing any amount of work that happens after that exposure does not reduce latency (source: §3.1, §8.3).

Performance validation: key results

The results split into two branches: forward inference and causal training. Forward trades error for speed; training decides whether that error breaks convergence.

Forward: up to 2.13×, with honest error accounting

Across 9 GB200 D128 shapes, fast is 2.023× faster on a geometric mean than HAO BF16 and peaks at 2998 TFLOP/s; accurate reaches 1.669× and 2416 TFLOP/s (source: §6.1). At S32768/H24, fast runs 4.400448 ms (2998 TFLOP/s), beating HAO’s published 2018 TFLOP/s on B200 and 2677 TFLOP/s on GB300 (source: §6.1).

The price of speed is error. HAO NV/FP8 has a mean cosine of 0.9899, versus 0.9438 for fast and 0.9517 for accurate (source: §6.1). Figure 3 shows this tradeoff: FP8 PV needs no E2M1 payload or block-scale page and is more accurate, while all-FP4 is faster but pays a larger error.

On B300 (SM103), D128 latency drops a further 5.6–7.7% in the S4096–S8192 range. The standard S8192/H64 shape hits 3116 TFLOP/s and the wave-aligned S9472/H64 shape hits 3159 TFLOP/s (source: §6.1).

How error propagates in real models

The practical question is how operator error propagates to the end of a model. On ViT at S4096, fast keeps BF16 top-1 accuracy (88.5%) with a 95.5% prediction-agreement rate, while accurate achieves 89.0% and 98.5% (source: §6.3, Table 8). Of 2272 classification examples, fast changes 32 predictions, and 31 of those are concentrated in the bottom quartile of the BF16 top-2 logit margin — only low-confidence predictions move (source: §6.3).

On Wan2.1 video diffusion, fast is 1.75× faster at 1.3B and 2.09× faster at 14B. Over 20 steps, though, the 14B cosine decays to 0.8496 — more accumulated drift than HAO NV/NV (0.9036). The error is bounded, but it does accumulate (source: §6.4, Table 9).

On ViT-MAE image reconstruction, fast lands at PSNR −0.020±0.026 dB and reconstruction cosine 0.99973, staying close to BF16 even after all 12 layers. The residual only becomes visible when amplified 8× (source: §6.5, Table 10).

Training: backward reuses forward’s quantization

Training adds three constraints: causal masking, GQA, and recovering gradients without materializing the $S\times S$ matrix (source: §7). Forward stores the NVFP4 Q/K payload bytes, block scales, and the LSE normalizer; backward reconstructs the probabilities by recomputing $QK^T$ from those (source: §7.1).

The results are split by measurement boundary (source: §7.3–7.7):

(source: Table 12–14)

The lesson is clear here. The reconstruction core alone shows a flashy 1.405×, but the E5M2 dO and row-statistics emitters that training safety demands swallow almost all of that saving (0.986×). That is why one “fast backward kernel” cannot predict end-to-end gains (source: §7.3).

The 8B full update gains more as the GPU fills up: B1 1.09× → B4 1.14×. On the FP8 P/V path, B4 throughput rises from 19,173 → 21,795 tokens/s and FLOP utilization from 41.12% → 46.74% (source: §7.5).

Training stability picks FP8 P/V

MXFP4 P/V was fast in the forward pass. But in long training, every tested MXFP4 P/V trajectory diverges. In the 4-arm cross diagnosis (projection format × P/V format), the two MXFP4 arms split from the FP8 control by update 500 (131.1M tokens): loss 7.97 vs. 5.41, with pre-clipping gradient norms spiking into the millions (source: §7.6, Appendix G.8). The two FP8 arms, meanwhile, descend without divergence through the shared observation horizon of 55.5B tokens.

The training path therefore keeps P/V in FP8 — rather than running “fully FP4 training,” it takes the gains that come from handing forward quantization to backward, conceding numeric ground only on P/V (source: §8.6).

Matched distributed training: 100B tokens

On 64 GPUs with global batch 1024 and a 100B-token schedule, the BF16 and FP8 paths complete the same course on identical data coordinates. Final training loss is 2.3095 (BF16) vs. 2.3613 (FP8); held-out validation at the same update is 2.3048 vs. 2.3948 (gap 0.0900) (source: §7.7).

Throughput rises 1.112×, from a median 21,853 → 24,303 tokens/s/GPU over 874 aligned observations (source: §7.7, Figure 11). The FP8 trajectory is stable and descending but not numerically identical to BF16 — and since this gap changes both projection and attention, it cannot be attributed to attention alone (source: §7.7).

Hardware diagnosis: the real bottleneck isn’t arithmetic

Three hardware conclusions are backed by measurements (source: §8, Table 15):

1. TMEM ownership caps the overlap. Two FP32 score banks plus two FP32 output accumulators occupy all $4\times128 = 512$ columns. Cutting shared memory from 209,920 to 163,840 bytes did not raise CTA occupancy (source: §8.1).
2. Low tensor activity is a symptom of “not ready.” Even executing the same 98,304 tensor instructions, the real probability path keeps tensor-pipe activity at just 18.8% (source: §8.2).
3. Faster arithmetic doesn’t speed up the whole kernel. A fixed-P diagnostic that removes almost all probability generation cuts latency by only 5.23% (source: §8.3).

Key Numbers (summary)

• Params: 8.03B (Llama-3.1 style, 32 layers, 32 Q-heads / 8 KV-heads / D128) — for the training experiments
• Context/Seq: S4096 (training), S7680 (Wan inference), up to S32768 (benchmarks)
• Architecture: FlashAttention-4 attention kernel (tiled + online softmax), GQA 32:8
• Positional: RoPE (applied at the projection-inclusive boundary) | Attention: Flash/online softmax, two-query two-CTA schedule
• Forward performance: GB200 fast 2998 TFLOP/s (geometric-mean 2.023× over BF16, up to 2.13×) | B300 S8192/H64 3116, S9472/H64 3159 TFLOP/s
• Forward error: fast cosine 0.9438 / rel-L2 0.3366, accurate 0.9517 / 0.3272 (vs. HAO NV/FP8 0.9899)
• Training speedup: 8B full update B4 1.137× (FP8 P/V), attention incl. projections 1.245×, distributed throughput 1.112×
• Training scale: 64 GPUs, global batch 1024, 100B tokens completed, validation gap +0.0900
• HW: GB200 (SM100, 152 SMs) / B300 (SM103, 148 SMs), TMEM 256 KB/SM (identical)
• Cost / Energy: $/1M tokens, kWh/1M tokens — not reported in the paper

$$ \text{KV-Cache(GB)} \approx \frac{2 \cdot L \cdot H \cdot d_\text{head} \cdot \text{seq} \cdot \text{batch} \cdot \text{bytes/elt}}{10^9} $$

Terminology: TPOT = Time Per Output Token (written alongside as “TBT = TPOT” where relevant). Because this is a kernel/operator-level study, the paper reports performance in TFLOP/s, ms, and tokens/s rather than TPOT/TTFT.

SOTA comparison (same setting, forward D128)

(source: Table 7, Figure 4)

Our take: strengths, limitations, and why it matters

Strengths — honest boundary separation and a sharp view of the critical path

The paper’s greatest strength is being explicit about what it does not claim. By separating measurement boundaries cell by cell — forward kernel / isolated backward / attention incl. projections / full 8B update / distributed trajectory — it forecloses the “fast kernel = fast model” simplification from the start (source: §5, Table 6). The fact that the backward reconstruction core’s 1.405× is eaten by the E5M2 emitters down to 0.986×, for example, would never have surfaced without that separation (source: §7.3).

The second is the crisp framing of “work on the critical path” vs. “work after it.” The measurement that a fixed-P diagnostic leaves only 5.23% shows that the next bottleneck for FP4 attention is not a more accurate polynomial approximation but a larger overlap window (source: §8.3, §8.4). This is data that breaks the naive expectation of “just make the arithmetic faster.”

Third is the scientific value of the divergence experiments. Showing that MXFP4 P/V is fast in forward but diverges in training — and using a factorial design that crosses it with both projection formats to pin the common factor on the P/V representation — is, together with the lesson “don’t mistake a fast timing bracket for a convergence result,” a very valuable result (source: §7.6, Appendix G.8).

Limitations — an honest scope

Limitations the authors explicitly acknowledge (source: §8.5):

• Boundary of the evidence. Training comparisons run one trajectory per route, so run-to-run variance and statistical equivalence cannot be estimated. Projection precision (E4M3 vs. NVFP4) is also a separate boundary, so the 0.0900 validation gap cannot be attributed to attention alone.
• Shape dependence. The hardware conclusions are limited to D128. D64 uses entirely different tile sizes, CTA ownership, and TMEM overlap strategies and should not be extrapolated (source: §8.5, Appendix A.6).
• Fully FP4 training remains incomplete. P/V and backward still concede to FP8. The fully FP4 training path merely proposes UE5M3 block scales as separate research and does not evaluate them here (source: §8.6).

Adding a potential limitation: this is a single-author technical report, and whether fast’s ~0.34 relative-L2 level of error stays harmless on real downstream tasks has only been verified on a limited set of fixed inputs for ViT/BERT/Wan. The margin mechanism — “31 of 2272 examples fall in the low-margin quartile” — is persuasive, but the sample is too small to establish universal inference or training safety (source: §6.3).

Why this work matters

• Practically: it shows that when you “make attention fast with FP4,” the bottleneck is not the matrix multiplication but the softmax middle stage — and it supplies both the fix (code classification + represented normalization + selective guards) and its cost (error).
• As an engineering warning: the two 256× scale bugs (the LSE lift +8 and the dO epilogue 1/256 correction) and the E4M3 issue of rounding 97% of dO to zero are traps anyone implementing low-precision attention training will step on (source: Appendix G.2).
• As a hardware pointer: “one more allocatable score bank,” “K32 scaled-FP4 PV,” and “scales outside TMEM” are concrete candidates derived from the measured dependencies (source: §8.4).

What’s next: the road ahead

1. Fully FP4 training. A separate study shows that UE5M3 (unsigned E5M3) block scales keep the E2M1 payload while providing a far wider scale range, enabling stable FP4 language-model pretraining. Applying this to the P/V product and the backward gradient product is a promising path to fully FP4 training, but it needs an efficient hardware implementation (source: §8.6, [6]).
2. Widen the overlap window. Once Direct-P shortens probability generation, the measured conclusion is that a larger allocatable overlap window is worth more than yet another polynomial approximation. A score destination that “the next QK can use while PV is still consuming” is the leading candidate design (source: §8.4).
3. Shape generalization. D64 and other regimes need their own tile, ownership, and overlap strategies, so the D128 schedule should not be extrapolated. Extending to those shapes is follow-up work (source: §8.5).
4. Statistical robustness. The single-trajectory distributed experiments should be extended to multiple seeds to quantify the run-to-run variance of the 0.0900 validation gap and whether the FP8 path is statistically equivalent (source: §8.5).

Bottom line: on Blackwell, FP4 makes matrix multiplication 4× faster, but attention’s bottleneck has already shifted to the softmax side. Direct-P shortens the critical path with the idea “don’t compute an accurate exponential — classify the E2M1 code directly,” gaining over 2× forward throughput; for training, it takes the gains of handing forward quantization to backward while showing that P/V must remain in FP8. The point is not arithmetic speed but the overlap window (source: §8.6).

Reference: reproduction checklist

• Code/commits/license: github.com/MrHuff/fp4-fa4 (TK forward/backward kernels, CuTe-DSL comparison kernels, experiment setup, evidence artifacts)
• Generate measurement graphs: python3 tools/plan_fa4_measurements.py list / print --family noncausal-forward|downstream|...
• Required operand contract: --nv-qk-fold-k64-scales both --nv-qk-fold-scale-select mse (K64 Q/K scale folding); accurate additionally requires a fixed 32-row K/V permutation
• Hardware/software: GB200 (SM100)/B300 (SM103), CUDA 13.0, CUTLASS DSL, seed 20260814
• Evaluation metrics: cosine / relative-L2 / RMSE, 300 ms warm-up + 3000 ms median window (some B300 runs use repeated windows)
• Training setup: fused AdamW, dense cross entropy (CCE disabled, torch.compile), B∈{1,2,4}, S4096, 10 warm-up + 21 measurement steps
• Distributed: 64 GPUs, global batch 1024 (local 4 × 4 accum), 100B tokens, checkpoint resume supported
• Evidence integrity: JSON manifest + SHA256 receipts (receipts/…json), W&B history frozen read-only

Tables from the paper

Tables converted mechanically from the arXiv e-print LaTeX source. The numbers are the paper’s own and did not pass through a model.

Table 1. Selected B300 D64 format diagnostics. NV/MX is the retained finite route. NV/NV timings are shown only to explain the rejected control and must not be treated as production performance when the status is non-finite.

Table 2. Shiftless TK NV/NV failure on model activations. Overflow is the fraction of N32 P scales above E4M3’s maximum before encoding.

Table 3. Measured TK NV/MX tuning points on B300. Every row reports speed and error from the same output.

Table 4. Major rejected directions. A timing tie is not promoted when it adds synchronization, storage, or numerical risk.

Table 5. Major rejected directions (continued).

Table 6. Complete fixed-schedule format matrix. Each row reports latency and output error together; no timing is paired with accuracy from another run.

Table 7. Accuracy-matched B300 control. Superscript p marks HAO-published cross-run values.

Table 8. Internal backward version map.

Table 9. Historical isolated D64 causal backward on GB200. The low-precision route is a generated CuTe kernel; both columns include required output clears.

Table 10. Historical isolated D128 causal backward at S4096/Hq32/Hkv8 on GB200. The native column is the v454/v482 predecessor family.

Table 11. Historical saturated single-GPU brackets. Each speedup is valid within its row group but must not be transferred to the final training recipe.

Table 12. Initial frozen rolling-log cutoff, retained for provenance. Jobs began at different times, so these are status observations rather than aligned loss or throughput comparisons.

Table 13. Reader’s guide to the main notation and Blackwell hardware terms.

Table 14. Block-scaled FP4 formats used in this work.

Table 15. Probability-format range at D128. Zero scales'' is the fraction of blocks whose scale encodes as zero; lost mass’’ is exact probability mass mapped to zero. This is a numerical diagnostic, not a kernel timing.

Table 16. Inherited structure and changes made in this work.

Table 17. Retained policies. Both use NVFP4 QK, MXFP4 P/V, K64 PV issue, and the two-query HAO layout.

Table 18. Measurement boundaries used in the paper.

Table 19. Primary forward comparison across Blackwell systems. Bold marks independently confirmed B300 results above 3 PFLOP/s. Published HAO columns provide cross-run context. ``ms / TF’’ means milliseconds and TFLOP/s; TK error is cosine / relative-$L_2$ against BF16.

Table 20. Downstream fixed-input trade-off. Task score is provider / BF16 accuracy; final error is cosine / relative-$L_2$ against BF16. Speedup belongs to the physical attention shape, not the complete model. HAO NV/NV is the identical-input control because no NV/FP8 task evaluation is published.

Table 21. Wan2.1 quality and warmed GB200 kernel speed at S7680/D128. Speedup is relative to HAO CuTe-DSL BF16. Quality is cosine / relative-$L_2$ of the final latent against the paired BF16 run.

Table 22. Paired ViT-MAE reconstruction. PSNR $\Delta$ is FP4 minus BF16 with a paired 95% interval. Reconstruction and layer cells are cosine / relative-$L_2$ against BF16.

Table 23. Hardware takeaways and the evidence that supports them.

Table 24. Matched historical profile with identical tensor work. This diagnostic predates the final Direct-P binary and is used only to identify the stall mechanism.

Table 25. Final-kernel ceilings relative to the 0.092448-ms valid record. These rows deliberately remove required work and do not compute valid attention.

Table 26. Hardware properties relevant to the measured kernels. TMEM capacity does not increase from GB200 to the tested B300.

Table 27. Precision contract for causal training. Learned projections and attention operands are separate boundaries.

Table 28. Isolated D128 causal backward at B1/S4096. Latency is the median of warmed runs; speedup is BF16 latency divided by the latency in each row.

Table 29. Projection-inclusive attention on one GB200. Backward-only runs the same prepared forward outside the timing interval; the second row times both forward and backward. Values are medians of warmed runs.

Table 30. Complete 8B updates at S4096 on one GB200. Each P/V route has its own adjacent BF16 timing bracket. Times are medians in milliseconds.

Figures in this post are taken from the original arXiv:2609.04105 (CC BY 4.0). Only size and format were changed.