Paper

NVFP4 4-bit Pretraining, Made Practical: 12B up to 10T Tokens, Effectively On Par with FP8

TL;DR

Pretraining a 12B hybrid Mamba-Transformer on 10T tokens at NVFP4 (4-bit) closely tracks FP8, with a loss gap of <1% in the stable regime and ~1.5% in the final phase, while downstream performance is also mostly on par (some math and multilingual tasks +0.9–+3.7pp). Matching MXFP4’s loss requires +36% more tokens (1.36T vs 1.0T), confirming NVFP4’s token-efficiency advantage. (source: Fig.2, Tab.2, Fig.6)

Core idea

• NVFP4 format: stores FP4 (E2M1) values in 1×16 blocks with an FP8 (E4M3) block scale (plus a global FP32 scale), reducing zero-flush/saturation errors and outlier loss. (source: §NVFP4, Fig.1)
• Recipe (training method): (1) keep roughly 15% of sensitive layers in BF16 (mostly the last 8 blocks), (2) apply a 2D 16×16 block scale to weights to guarantee forward/backward consistency, (3) apply RHT (Hadamard) to Wgrad only, (4) apply SR (stochastic rounding) to gradients only (Dgrad + Wgrad). (source: §4, Fig.4, Appx E.2–E.4)

Background: the problem they set out to solve

4-bit training offers hardware advantages — 2× throughput (GB200)/3× (GB300) and ~1/2 the memory (vs FP8) — but it frequently diverges during large-scale pretraining because of chain-rule inconsistency, outliers, and quantization bias. NVFP4 improves numerical accuracy over MXFP4 with smaller blocks (16) and finer-grained scales (E4M3), and the paper complements the remaining limits with a training recipe. (source: Tab.1, §NVFP4)

The new approach: the NVFP4 training recipe

1. Partial high precision: quantizing every linear layer to FP4 diverges. Keeping the last 4–8 blocks in BF16 yields stable convergence (a common trend at 12B/1.2B). (source: Appx E.2 Fig.9)
2. 2D weight scaling (16×16): forward and backward scale along different axes, so $(w_{\text{fprop}}\neq w_{\text{bprop}})$ → chain-rule violation. Applying a 2D block scale to the weights keeps the same quantized representation. (source: §4.3, Fig.14)
3. RHT only for Wgrad: applying it to fprop/Dgrad actually hurts; applying it only to the Wgrad input improves loss. Matrix size d=16 balances compute and accuracy. (source: Appx E.4 Fig.11–12)
4. SR only for gradients: SR on gradients is the key to convergence; SR on activations/weights triggers divergence. At 12B, both Dgrad and Wgrad need SR. (source: §4.4, Appx E.3 Fig.10)

Summary: the four must work as a set to converge stably over the 10T horizon; removing any one of them degrades convergence and loss. (source: Fig.4, Fig.8)

How it works: walking through a concrete example

Consider a single tile (block) of a heavily simplified linear layer (input 3, output 3).

• Quantization scaling: first remap the whole tensor with an FP32 scale $(s_\mathrm{tensor})$; then for each 1×16 (assumed 1×3 here) block, multiply an FP8 (E4M3) scale $(s_\mathrm{block})$ to squeeze values into the representable FP4 range. This reduces zero-flushing of small values and saturation of large ones. (source: §NVFP4)
• Chain-rule consistency: forward (fprop) scales along rows and backward (bprop) along columns — the axes flip. With a 1D scale, the weight’s quantized representation differs between forward and backward $((w_{\text{fprop}}\neq w_{\text{bprop}}))$ → a 2D 16×16 scale applies the same scale to the same tile, making $(w_{\text{fprop}}=w_{\text{bprop}})$ approximately hold. (source: §4.3, Fig.14)
• Outlier mitigation (RHT): mixing activations/gradients with a Hadamard×sign transform spreads values so the in-block variance grows, which mitigates FP4 error. Applying it to fprop/Dgrad, however, enlarges quantization inconsistency and backfires → Wgrad-only is the safe choice. (source: Fig.11)
• Debiasing (SR): near FP4, round-to-nearest creates bias. Applying stochastic rounding (SR) to gradients only — to cut that bias — was the key to convergence. (source: §4.4, Fig.10)

Validation: key results

• Pretraining loss curves (12B, 10T): tracks FP8 closely, with a relative gap <1% in the stable regime and ~1.5% through the lr-decay phase. (source: Fig.2)
• Downstream (BF16 eval): MMLU-Pro 62.58 vs 62.62 (−0.04pp), GSM8K-CoT +3.19pp, MGSM +3.66pp, ARC-C tie, coding family −2.5~−3.2pp. (source: Tab.2)
• NVFP4 vs MXFP4 (8B): NVFP4 relative gap ~1.5%, MXFP4 ~2.5%. MXFP4 needs +36% more tokens (1.36T vs 1.0T) to match NVFP4’s loss. (source: Fig.6)
• End-of-training precision switch (12B, 10T): switching only fprop to BF16 from 8.2T brings the loss gap from 1.5% to 0.5%, adding just ~6% of high-precision compute. (source: Fig.7)

Our take: strengths, limitations, and why it matters

Strengths

• Better accuracy–cost Pareto: at the same accuracy, a 36% saving in token budget (vs MXFP4) → potential savings in time, energy, and $. (source: Fig.6)
• Reliability at scale: simultaneous stable convergence and downstream parity over the 12B·10T horizon. (source: Fig.2, Tab.2)
• Methodological transparency: ablations (Fig.4/8/10/11/12/14) systematically establish each component as a necessary condition. (source: §4, Appx E)

Limitations

• Weaker on coding: −2.5~−3.2pp on HE+/MBPP+. (source: Tab.2)
• Reliance on partial high precision: currently needs the last 8 blocks (~15%) kept in BF16. (source: §5 setup)
• No system metrics: the paper leaves end-to-end runtime/cost/energy measurements out of scope. (source: §Intro/Conclusions scope note)

Why it matters

• The optimization format of the Blackwell generation (Tensor-Core-native, FP4 2×/3× throughput, 1/2 memory): the larger the model/data scale, the sooner it crosses the economic viability threshold. (source: Tab.1)

What’s next? The road ahead

• Full FP4 (all layers): explore dynamic BF16↔FP4 gating driven by layer-sensitivity metrics (e.g., Wgrad error). (source: Appx E.2)
• Automating the forward-only precision switch: detect change points such as the start of lr decay to decide the switch around 8.2T automatically. (source: Fig.7)
• FP4 for attention/communication paths: extend FP4 beyond GEMM and report TTFT/TPOT/Throughput/$·kWh per 1M tok under identical settings. (source: Tab.1)
• Format-specific scaling laws: quantitatively model NVFP4 vs MXFP4 over P·T horizons to pre-design the optimal format and token budget. (source: Fig.6 comment)

Reference table (summary)

One-line conclusion: the NVFP4 recipe is an empirical demonstration that FP8-equivalent accuracy can be realized within FP4’s cost structure, and refining format, scale, and precision switching will accelerate the move to full FP4 and end-to-end cost savings. (source: Fig.2/6/7, Tab.1/2)

Click the toggle to view the detailed LLM Q&A about the paper.

Prompt 1.1.1 (research gap)

Analyze the paper's 'Introduction' and 'Related Work' sections and explain what core research gap, critical limitation of prior work, or unanswered question this study explicitly sets out to address. Also summarize what the state of the art was at the time of publication, as described by the authors.

Bottom line: the gap this paper targets is the absence of a method for stably pretraining token-trillion-scale LLMs at FP4 (4-bit) precision. Combining the NVFP4 format + 2D weight scaling + Random Hadamard Transform (RHT) + stochastic rounding, the authors converge a 12B model on 10T tokens at a loss gap of <~1.5% vs FP8 with on-par performance, presenting the first public evidence of multi-trillion-token FP4 pretraining. (source: §Intro, §3 Fig.2, Tab.2, §6)

1) What is the ‘research gap’?

• The FP8→FP4 transition gap: industry SOTA had widely adopted FP8 training, but FP4 (especially at large scale and long token horizons) was bottlenecked by convergence stability, chain-rule inconsistency, quantization bias, and implementation constraints. (source: §Intro)
• Enough hardware motivation, but an algorithmic gap: Blackwell Tensor Cores deliver 2–3× math throughput (vs FP8) and ~1/2 memory, but the lack of methodology kept these real benefits from reaching pretraining. (source: Tab.1/§NVFP4 Format)
• No evidence at pretraining scale: earlier FP4 work/formats (MXFP4 and others) existed, but public results systematically showing FP8-level convergence and accuracy at multi-trillion-token scale were missing. (source: §Intro, §6)
• A precise diagnosis of the problem: (i) forward/backward axis mismatch gives the same tensor different quantized representations, breaking the chain rule (for weights); (ii) gradient quantization bias (more severe in FP4); (iii) the need to handle block-level outliers. (source: §4.3, §4.4, Appx E)

2) The solution the authors propose (= filling the gap)

• NVFP4 format refinement: smaller micro-blocks (16) and two-level scaling — an FP8 fractional-precision scale plus an FP32 tensor scale — capture local dynamic range and improve precision. (source: §Intro/NVFP4 Format)
• 2D weight scaling: 16×16 blocks keep the same quantized representation in both fprop and bprop, mitigating chain-rule inconsistency (improves loss). (source: §4.3, Fig.4, Fig.14)
• RHT (Random Hadamard Transform): applied to spread block-outlier variance, but not to weights — restricted to Wgrad to avoid damaging consistency. (source: §4.3, Appx E.4.1)
• Stochastic rounding (required for grads): reduces gradient quantization bias and is shown to be essential for convergence at 12B. (source: §4.4, Fig.4)
• Mixed-precision layers: keep sensitive layers at higher precision (partly BF16) to stabilize overall convergence. (source: §Training Methodology, Fig.4)

3) The ‘SOTA’ context this paper set at publication time

4) Open questions & limitations (per the authors)

• Widening coverage: quantizing all linear layers, extending to attention/communication paths, and generalizing to larger models, longer token horizons, and MoE architectures remain as homework. (source: §Conclusions/Future Work)
• Precision-switching strategy: raising precision (BF16/MXFP8) during the final (lr decay) phase helps close the loss gap — most training can stay in FP4 with only a minimal amount of high precision. (source: Fig.2 note / Appendix D mention)
• A general solution to chain-rule/consistency: the generality and optimal parameters (e.g., Hadamard size, randomization scheme) of the 2D scaling/RHT design across diverse architectures and kernels still need further investigation. (source: §4.3/“Hadamard size & randomization” experiments)

5) Key numbers (SOTA context)

• Scale: 12B parameters, 10T tokens pretrained in FP4 (the longest publicly reported). (source: §Abstract/§3)
• Loss tracking: relative gap <1% in the stable regime, ~1.5% in the final phase. (source: Fig.2)
• Downstream: MMLU-Pro 62.58 vs FP8 62.62, on par across a broad set of tasks (slightly behind only on code). (source: §Abstract, Tab.2)
• Format comparison: NVFP4 reaches the same loss as MXFP4 with −36% tokens (1.36T→1T). (source: §5 Fig.6)

Takeaway: at publication time, SOTA consisted of standardized FP8 training plus Blackwell’s FP4 hardware advantage, but stable, accurate convergence for large-scale FP4 pretraining was the gap. This paper fills it with NVFP4 + its algorithmic bundle, demonstrating FP8 parity at 12B·10T. (source: §Intro, §3, Fig.2, Tab.2, §6)

Prompt 1.1.2 (core hypothesis)

What is the central hypothesis or core claim of this paper? State it in one clear, concise sentence, in a form such as 'the authors assume that by using [proposed technique] they can achieve [concrete result] that overcomes [existing limitation]'.

Hypothesis (one sentence): the authors claim that combining the NVFP4 format (16-element FP4 blocks + E4M3 block scales + FP32 tensor scale) with 2D weight scaling (16×16), RHT restricted to Wgrad, stochastic rounding of gradients, and keeping sensitive linear layers in BF16 overcomes the chain-rule inconsistency, quantization bias, and outlier problems of FP4 training, allowing a 12B LLM to be pretrained over 10T tokens with a validation loss gap ≤≈1.5% vs FP8 (below 1% in the stable regime and ~1.5% in the final phase) and on-par downstream accuracy. (source: §NVFP4 Format/§4.3–4.4, Fig.2–3, Tab.2, §6)

Prompt 1.2.1 (identifying novelty)

Based on the full paper, list the 1–3 most important and original contributions as distinct items. Clearly classify each one as a new architectural component, a new training technique, a new theoretical insight, a new dataset, or a new application of an existing methodology.

Bottom line: the contributions are (1) the NVFP4 4-bit microscaling format (16-element blocks, E4M3 block scale + FP32 tensor scale), (2) a stable FP4 training recipe (16×16 2D weight scaling + Wgrad-only RHT + gradient SR + keeping a few layers high-precision), and (3) a large-scale 12B·10T demonstration with a 36% token saving vs MXFP4. (source: §2 Fig.1, §4 Fig.4, §3 Fig.2/Tab.2, §5 Fig.6)

Novel contribution 1 — the NVFP4 data format (a new number representation/format)

• Core: instead of MXFP4 (block 32, UE8M0 scale), it introduces a two-level microscaling scheme combining blocks of 16 (E2M1) with an E4M3 block scale and an FP32 tensor scale. The ≥6.25% of values in a block that reach the max (amax) are represented at near-FP8 precision, while the rest is stored as FP4, preserving dynamic range. (source: §2 Fig.1)
• Significance (quantitative): shrinking the block from 32→16 reduces in-block variance; moving UE8M0→E4M3 scales with fractional precision; and it avoids the power-of-two scale-rounding loss (possible ±4, ±6 binade loss), reducing saturation and zero-flushing. (source: §2/Appx B.4)

Classification: a new data format (numerical representation)

Novel contribution 2 — a training methodology for stabilizing FP4 (a new training technique/recipe)

• 2D weight scaling (16×16): forces the same quantized representation in fprop/bprop, mitigating chain-rule inconsistency and improving loss at 12B. (source: §4.3 Fig.4/14)
• Restricting RHT’s scope: the Hadamard transform is applied only to Wgrad (size 16×16) and not to weights, preventing the fprop/bprop inconsistency from worsening. (source: §4.3/Appx E.4.1)
• Stochastic rounding (SR): restricted to gradients, where it is essential for convergence; applying it in the forward pass actually hurts. (source: §4.4 Fig.4/Appx E.3)
• Keeping mixed precision: keeping ≤~15–16% of sensitive linear layers in BF16/MXFP8 yields stable convergence even over the 10T-token horizon (keeping just the last 4 blocks high-precision also works). (source: §4 Fig.4, §4.1)

Classification: a new training technique (methodological recipe)

Novel contribution 3 — a public demonstration of large-scale FP4 pretraining and a format comparison (new empirical evidence/application)

• The 12B·10T demonstration: when 12B parameters are pretrained on 10T tokens with NVFP4, the validation loss tracks FP8 at a <1% gap in the stable regime and ~1.5% in the final phase; downstream accuracy is on par (Tab.2, slightly behind only on some code tasks). (source: §3 Fig.2/3, Tab.2)
• NVFP4 vs MXFP4 efficiency: MXFP4 needs +36% more tokens (1.36T vs 1.0T) to reach the same loss → quantifies the added token and time cost. (source: §5 Fig.6)
• Significance: the first public evidence of sustained multi-trillion-token FP4 pretraining, plus a strategy to shrink the late-stage loss gap via precision switching (BF16 etc.). (source: §6, Appx D Fig.7)

Classification: new empirical evidence / scaling application and format-comparison insight

Prompt 1.2.2 (strengths from the authors’ perspective)

From the authors' perspective, why is their approach superior to earlier methods? Quote or explain in plain terms the key arguments they use to support the originality and strengths of their work.

Bottom line: the authors argue that the NVFP4 numeric format (block 16, E4M3, FP32 tensor scale) plus the training recipe (16×16 2D weight scaling, Wgrad-only 16×16 RHT, SR on gradients, ~15% of sensitive layers high-precision) suppresses chain-rule inconsistency, quantization bias, and block outliers, achieving a loss gap of <1% (stable regime) / ~1.5% (final) vs FP8 in 12B@10T pretraining with on-par downstream results. (source: §2, §3 Fig.2–3, §4 Fig.4, Tab.2)

Core strengths from the authors’ perspective (numbers first)

1. The numeric format itself: NVFP4 → block 16, E4M3, FP32 scale (two-level MX)

• Compared with MXFP4 (block 32, UE8M0), the block size 32→16 shrinks the in-block dynamic range, the scale UE8M0→E4M3 raises effective mantissa precision, and the FP32 tensor scale preserves global range. The ≥6.25% of values (amax) in a block are kept at near-FP8 precision, reducing saturation and zero-flushing. (source: §2 Fig.1, Appx B.4)
• The result avoids the 1-binade loss (±4, ±6) risk from MXFP4’s power-of-two scale rounding. (source: §2, Appx B.4)

2. Solving chain-rule consistency: 2D weight scaling (16×16)

• In FP4, fprop/bprop scale/transform along different axes; when the same weight tensor ends up with different quantized representations, the chain rule breaks → worse loss. A 16×16 weight scale forces fprop=bprop representation agreement and is shown to improve loss at 12B. (source: §4.3, Fig.4/14)

3. Outlier mitigation follows a ’no weights’ rule: RHT applied to Wgrad only, at 16×16

• Applying the Hadamard transform to weights worsens the fprop/bprop inconsistency, so it is not applied to weights and is restricted to Wgrad at 16×16, reconciling effectiveness and consistency. (source: §4.2–4.3, Appx E.4.1)
• Matrix size d=16 offers the best convergence–performance balance at 12B over long horizons (too small a d degrades convergence; too large gives diminishing returns). (source: Appx E.4.2 Fig.13)

4. Removing quantization bias: stochastic rounding (SR) on gradients only

• Deterministic rounding raises bias in FP4; SR on grads is essential for 12B convergence, while SR on fprop tensors actually hurts. (source: §4.4, Appx E.3, Fig.4)

5. Minimal mixed precision: only ≤~15% of sensitive linear layers stay high-precision

• The GEMMs stay in FP4 overall; only some sensitive linear layers (mostly at the output end) are kept in BF16/MXFP8 to prevent divergence and stabilize convergence. In the 12B run, the first 2 blocks + last 8 blocks (≈16%) are high-precision, but keeping just the last 4 blocks high-precision is already stable. (source: §4 Fig.4–5, Appx E.2)

6. Scale and accuracy demonstrated: FP8 parity at 12B@10T

• Validation loss: <1% in the stable regime and ~1.5% in the final (lr-decay) phase. Downstream: MMLU-Pro 5-shot 62.58 vs FP8 62.62, on par across general/math/multilingual, only slightly behind on code. (source: §3 Fig.2–3, Tab.2)

7. Quantitative evidence of format efficiency: NVFP4 is more token-efficient than MXFP4

• For the same final loss, MXFP4 needs +36% more tokens (1.36T vs 1.0T) → higher training time and cost. NVFP4 reaches the same loss with fewer tokens. (source: §5 Fig.6)

8. Realizing the hardware-aligned benefit: Blackwell FP4 math throughput↑, memory↓

• Blackwell FP4 math throughput: 2× on GB200 / 3× on GB300 (vs FP8), with ~1/2 memory. The authors show their recipe can carry this advantage into pretraining. (source: Tab.1, §6 conclusion)

9. Practical tip (precision switch): switching to BF16 late in training closes the loss gap from 1.5% to ~0.5%

• Switching mainly the forward pass to BF16 from ~8.2T onward (18% of training) recovers most of the gap, with only ~6% of total compute high-precision → minimal runtime impact. (source: Appx D Fig.7)

Summary: the combination of format (NVFP4) × methodology (2D weight scaling, Wgrad-RHT, grad-SR, partial mixed precision) strikes directly at the sources of numerical instability (consistency, bias, outliers) and quantitatively demonstrates FP8-level accuracy at 12B·10T, including token and time efficiency. (source: §2–§5, Fig.2/4/6, Tab.1/2)

Prompt 1.3.1 (step-by-step algorithm walkthrough)

Explain the core algorithm, model architecture, or main methodology step by step. Assume the reader is a graduate student in AI. In particular, construct a very simple toy example — simple sentences, a 3×3 pixel image, a small state space — with sample inputs, and walk each step through the example to show how the input is transformed into the output. Define every key term and variable as soon as it appears.

Bottom line: one training step runs as follows: (1) NVFP4 quantization (global FP32 scale → block E4M3 scale → FP4 conversion), (2) 2D weight scaling (16×16) keeps fprop/bprop consistent, (3) RHT (16×16) only on the Wgrad input, (4) stochastic rounding on gradients only, (5) a few final layers stay in BF16. With this recipe, 12B@10T pretraining keeps a loss gap of <1% in the stable regime and ~1.5% in the final phase vs FP8, with on-par downstream results. (source: §2 Fig.1/Tab.1, §3 Fig.2–3, §4.1–4.4, Appx E)

0) Prep: key terms

• NVFP4: FP4 (elements) + E4M3 block scale (8-bit) + FP32 global scale — a two-level microscaling format with block size 16. (source: §2)
• 2D scale (weights): scales the weight tensor in 16×16 blocks (input channel × output channel), so fprop and bprop share the same quantized representation. (source: §4.3)
• RHT: random Hadamard transform, applied only to the Wgrad input, size d=16; a fixed seed is sufficient. (source: §4.2, Appx E.4.1–E.4.3)
• Stochastic rounding (SR): must be applied only to gradients for convergence; using it on fprop tensors hurts. (source: §4.4, Appx E.3)
• Mixed precision: embeddings/attention/normalization/optimizer states stay high-precision; only some final linear layers (≤~15–16%) are kept in BF16. (source: §4.1, Appx E.2)

1) The NVFP4 quantization pipeline (for one tensor)

Input: a real tensor $(T\in\mathbb{R}^{m\times n})$. Output: the tensor stored as NVFP4 $(\widehat{T})$ (FP4 values + block scale (E4M3) + global scale (FP32)).

Step 1 — global FP32 scale: multiply by a global scale $(s_\text{glob}\in\mathbb{R}_{+})$ to remap the tensor into a range the block scale (E4M3) can represent. (source: §2)

Step 2 — E4M3 block scale (block = 16 elements): set an E4M3 scale $(s_b)$ per block so each block’s amax maps near the FP4 maximum, shifting the block into the representable FP4 interval:

(source: §2 Fig.1)

Step 3 — FP4 conversion & decode rule: round to FP4: weights/activations use round-to-nearest-even; gradients use SR (probability inversely proportional to the distance to the two nearest values). (source: §4.4) Reconstruction descales by $((s_b\cdot s_\text{glob})^{-1})$. (source: §2)

Why NVFP4? Compared with MXFP4 (block 32, UE8M0), block 16 + E4M3 + global FP32 keeps ≥6.25% of values (each block’s amax) at near-FP8 precision and avoids the binade loss (±4, ±6) from power-of-two scale rounding. (source: §2)

2) Ensuring chain-rule consistency with 2D weight scaling (16×16)

Problem: fprop scales/transforms along rows (input channels) and bprop along columns (output channels), so the same weight takes different quantized representations → chain-rule violation. (source: §4.3)

Solution: only the weights use a 16×16 block scale, forcing the same quantized representation in fprop/bprop. Activations/gradients keep the existing 1×16 scheme. (source: §4.3, Fig.14)

3) RHT (random Hadamard) — on the Wgrad input only, size d=16

• Why Wgrad only? RHT on fprop/dgrad enlarges transform inconsistency and degrades quality; applying it only to the Wgrad input improves the loss. (source: Appx E.4.1)
• Choosing the size: d=16 is better than 4 and similar to 128, the best cost–accuracy tradeoff. (source: Appx E.4.2)
• Randomization: a fixed seed is enough; no randomization is inferior. (source: Appx E.4.3 Fig.13)

4) Stochastic rounding — on gradients only

• Required target: SR on gradients is essential for 12B convergence; SR on other tensors hurts. (source: §4.4, Appx E.3)

5) Mixed precision — only a few final layers in BF16

• FP4 for every linear layer diverges. Keeping just the last 4 blocks in BF16 is already stable; the 12B run adopts the more conservative setting of keeping the first 2 + last 8 blocks (≈16%) in BF16. (source: §4.1, Appx E.2)

6) One Linear-layer training step — procedure summary (pseudocode)

Input: $(x)$ (BF16 activation), $(W)$ (FP32 optimizer weight) Output: updated $(W)$

1. Fprop GEMM

• $(x\rightarrow)$ quantize to NVFP4 (global FP32 → 1×16 E4M3 → FP4) [round-to-nearest-even],
• $(W\rightarrow)$ quantize to NVFP4 (global FP32 → 16×16 E4M3 → FP4),
• $(y = \text{GEMM}(x_\text{FP4}, W_\text{FP4}))$ → accumulate in BF16/FP32. (source: Fig.5, §4.3)

2. Dgrad GEMM

• compute $(\partial L/\partial x)$ from $(\partial L/\partial y)$ (BF16) and $(W_\text{FP4})$. (source: Fig.5)

3. Wgrad GEMM (+RHT, SR)

• apply RHT (d=16) to the Wgrad input $((x, \partial L/\partial y))$ only → NVFP4 conversion,
• use gradient SR to mitigate the FP4-conversion bias,
• compute $(\partial L/\partial W)$ (accumulate in BF16/FP32). (source: §4.2, §4.4, Fig.5)

4. Optimizer update

• $(W)$ and the optimizer state stay in FP32; update, then move to the next step. (source: §4.1)

7) The transform flow on a toy example (4×4 weight block = 16 elements)

weight block $(W\in\mathbb{R}^{4\times4})$:

• Global scale $(s_\text{glob}=1)$ (example).

• Block amax = 6.2 → set the E4M3 block scale $(s_b=\frac{6}{6.2}\approx0.968)$, mapping amax to the FP4 maximum 6. (illustrative example)

• Scaled sample elements: $(6.2\times0.968=6.00)$, $(-4.3\times0.968=-4.16)$, $(0.48\times0.968=0.465)$, …

• FP4 rounding (round-to-nearest-even) value set $({\pm0,\pm0.5,\pm1,\pm1.5,\pm2,\pm3,\pm4,\pm6})$:

  • $(6.00\to 6)$, $(-4.16\to -4)$, $(0.465\to 0.5)$, $(1.7\to 1.5)$, $(-0.6\to -0.5)$, $(3.2\to 3)$, $(-1.1\to -1)$, …

• Decoding multiplies by $(1/(s_b\cdot s_\text{glob}))$ to reconstruct an approximation. → Sharing the same 16×16 scale across both fprop and bprop means the same FP4 representation is reused, which guarantees chain-rule consistency. (source: §4.3, Fig.14)

SR mini-example (scalar g=1.03, FP4 neighbors = {1,1.5}): $(P(1)=\frac{1.5-1.03}{0.5}=0.94)$, $(P(1.5)=0.06)$ → $(\mathbb{E}[\text{SR}(g)]=1.03)$ (unbiased). SR must be applied only to gradients for overall convergence to stay stable. (source: §4.4, Appx E.3)

8) Performance and accuracy context (key numbers)

• Convergence curve (12B@10T): relative loss <1% in the stable regime, ~1.5% in the final (lr-decay) phase. (source: Fig.2)
• Downstream: on par with FP8 across the board; only slightly behind on code. (source: Fig.3/Tab.2)
• NVFP4 vs MXFP4: NVFP4 shows consistently better training behavior, avoiding binade loss. (source: §2 summary, §5 Fig.6)
• HW advantage (Blackwell): FP4 math throughput 2× (GB200)/3× (GB300) vs FP8, ~1/2 memory. (source: Tab.1)

9) The ’linear-layer compute flow’ at a glance (figure mapping)

• All GEMM inputs are quantized to NVFP4; the Wgrad input gets RHT, gradients get SR, and weights/optimizer states stay in FP32. (source: Fig.5, §4.1–4.4)

10) (Appendix) Formula summary

• Global + block scaling:

  $$ \widehat{T}=\operatorname{FP4}!\left(\underbrace{S_b}_{\text{E4M3}}\cdot\underbrace{s_\text{glob}}_{\text{FP32}}\cdot T\right),\quad S_b=\operatorname{diag}({s_b^{(k)}}) $$

  (blocks of 16 elements; 16×16 2D scale for weights) (source: §2, §4.3)

• SR (gradients):

  $$ \operatorname{SR}(v)= \begin{cases} a, & \text{w.p. } \dfrac{b-v}{b-a} b, & \text{w.p. } \dfrac{v-a}{b-a} \end{cases}\quad (a\le v\le b) $$

  (source: §4.4)

Summary: the sequential pipeline — NVFP4 (global FP32 + block E4M3) × 2D weight scaling (16×16) × Wgrad-RHT (16) × grad-SR × mixed precision at the output end — addresses numerical consistency, bias, and outliers simultaneously, and demonstrates FP8-level accuracy even at scale (12B·10T). (source: §2–§5)

Prompt 1.3.2 (spotting the ‘secret weapon’)

Pick one core component and, for the case where it is removed, replaced, or its scale is changed, present Δ(metric) in a table and explain the mechanism behind the change (e.g., gating load balance, rotary vs ALiBi, sparse attention half-window replacement).

Prompt 1.3.2 — spotting the ‘secret weapon’ (the NVFP4 format)

Bottom line: the core component chosen is NVFP4 (block 16, E4M3 block scale + FP32 tensor scale). Replacing it with MXFP4 under identical settings worsens the final validation-loss gap by ~+1.0pp (1.5%→2.5%), and matching NVFP4’s level needs +36% more tokens (1.0T→1.36T). The mechanism: two-level scaling with block 16 + E4M3 + FP32 reduces dynamic-range loss (one binade) and saturation/zero-flushing, while keeping ≥6.25% of in-block values at near-FP8 precision. (source: Fig.6, §2)

Δ(metric) — replacing NVFP4 with MXFP4

• Numbers from: Fig.6 — NVFP4 shows ~1.5% and MXFP4 ~2.5% relative gap (left); MXFP4 reaches the same loss as NVFP4 (1.0T) at 1.36T (=+36%) (right). (source: Fig.6 note/conclusion)

Interpretation: under the same hardware/throughput assumptions, +36% tokens scales time and compute proportionally → a meaningful increase in training time. (source: Fig.6 comment)

Why this Δ? — the mechanism (numeric/structural)

1. Block size 32→16 (localizing dynamic range): NVFP4’s block of 16 lowers in-block variance, fitting FP4’s representable range better. MXFP4 (block 32) faces a larger saturation/zero-flushing risk under the same scale constraints. (source: §2)
2. Scale representation: UE8M0 (power-of-two) → E4M3 (fractional mantissa): MXFP4’s UE8M0 (2^k) rounding can lose a binade (±4, ±6). NVFP4’s E4M3 block scale + FP32 tensor scale provides precise scaling and avoids scale-rounding loss. (source: §2, Appx B.4 point)
3. Near-FP8 preservation ratio: NVFP4 encodes each block’s (16) maximum (amax) in E4M3, so ≥6.25% of in-block values stay at near-FP8 precision while the rest is stored as FP4. [ \text{rate}_{\text{near-FP8}}=\frac{1}{16}=6.25%\ (\text{per block}) ] (source: §2 Fig.1 note)

Summary: NVFP4 = (block 16) × (E4M3 block scale) × (FP32 tensor scale) two-level microscaling → minimal dynamic-range/precision loss → the same loss with fewer tokens. Conversely, MXFP4’s scale rounding and block size increase the loss → it needs more tokens. (source: Fig.6, §2)

(Bonus) two other axes in the same ‘secret weapon’ context — qualitative Δ

• Precision switch (fprop-only BF16 after 8.2T): recovers the relative gap from 1.5% to 0.5%; only ~6% of total compute is high-precision, so the overhead is small. (source: Fig.7)
• Where/size of the Hadamard application: Wgrad-only helps; applying it to fprop/Dgrad hurts; d=16 beats d=4 and matches d=128 (cost–benefit balance). (source: Fig.11, Fig.12)

Overall context: with the NVFP4 format as the core, combining 2D weight scaling (16×16), Wgrad-RHT (d=16), grad-SR, and some end-of-training BF16 achieves stable convergence and FP8-close loss at the 12B·10T scale. (source: Fig.4 summary/recommendation)

Prompt 1.4.1 (analysis of key results)

Analyze the main results, including the tables/figures in 'Experiments' or 'Results'. What are the key performance metrics? On which benchmarks were they reported? Summarize the results the authors emphasize most as evidence of success.

Prompt 1.4.1 — analysis of key results

Bottom line: pretraining a 12B hybrid (Mamba-Transformer) model on 10T tokens at NVFP4 precision keeps the validation loss within <1% of FP8 in the stable regime and holds a ~1.5% relative gap even in the final (lr-decay) phase, with downstream accuracy mostly on par (MMLU-Pro 62.58 vs 62.62, etc.). Also, reaching the same loss under MXFP4 needs −36% fewer tokens for NVFP4 (1.36T→1.0T), i.e., higher training efficiency. Switching only the forward pass to BF16 (after 8.2T) late in training narrows the loss gap from 1.5% to 0.5%, with high-precision compute taking just ~6% of the total. (source: Fig.2/3/6/7, Tab.2)

Key performance metrics (points from Experiments/Results)

• Pretraining scale: 12B params × 10T tokens, evaluated against an FP8 baseline. (source: §3)
• Validation-loss curves: NVFP4 tracks FP8 closely across the whole run — <1% in the stable regime, widening to ~1.5% during lr decay; the slope change around 8T tokens is the lr decay, and the jump near 9T tokens comes from a data-blend change. (source: Fig.2)
• Downstream accuracy: on par with FP8 on MMLU-Pro, MMLU, GSM8K-CoT, MATH, Global-MMLU, MGSM, HumanEval+, MBPP+, ARC-C, HellaSwag, and other multi-domain tasks (only the coding family is slightly behind). (source: Fig.3, Tab.2)
• Format comparison: in NVFP4 vs MXFP4, MXFP4 shows ≈2.5% relative gap vs NVFP4’s 1.5%, and needs +36% more tokens (1.36T) for the same loss. (source: Fig.6)
• End-of-training precision switch: switching only the forward pass to BF16 at 8.2T recovers the gap from 1.5% to 0.5%, with only ~6% of total compute high-precision (about a third of the final 18% stretch). (source: Fig.7, Appx D)
• Contribution of the essential techniques: removing any of SR (grad), Wgrad-RHT, 2D weight scaling, or some BF16 layers degrades convergence (ablations). (source: Fig.4)

Benchmarks & key numbers (from Tab.2)

(source: Tab.2)

Interpretation: in the language, math, and multilingual domains NVFP4 is on par or ahead in places (e.g., GSM8K +3.19pp, MGSM +3.66pp); the coding family (HE+, MBPP+) is slightly behind — the authors mention eval noise/checkpoint choice as a possibility. (source: §3 discussion)

NVFP4 vs MXFP4 — token efficiency

• Under the same 8B setup, reaching the loss that NVFP4 hits at 1.0T requires MXFP4 to use 1.36T (+36%) → directly more time, compute, and energy. (source: Fig.6)
• On a relative-gap basis, MXFP4 is ~2.5% vs NVFP4’s ~1.5%: NVFP4 converges with better quality. (source: Fig.6)

Effect of the end-of-training precision switch (‘a small amount of high precision’)

• Switch point: at 8.2T tokens, switch NVFP4→BF16 (forward only). (source: Fig.7)
• Effect: relative gap 1.5%→0.5%, with high-precision compute a ~6% share (a tiny fraction of the total) → minimal runtime impact. (source: Fig.7/Appx D)

The ’evidence of success’ the authors emphasize

1. Loss stability and FP8-close tracking (<1%~1.5%) even over a long 10T-token horizon, including an explanation of the data/schedule switch points. (source: Fig.2)
2. Broad downstream parity (language, math, multilingual, commonsense, code) — the headline number is MMLU-Pro 62.58 vs 62.62. (source: Fig.3, Tab.2)
3. Format-level efficiency evidence: NVFP4 saves 36% of tokens vs MXFP4 for the same loss → time/cost savings. (source: Fig.6)
4. Each recipe component is necessary: SR (grad), RHT (Wgrad), 2D weight scaling, and some BF16 layers must all be present to converge without divergence (removing any degrades results). (source: Fig.4)

Supporting context: Blackwell Tensor Cores provide FP4 math throughput of 2× (GB200)/3× (GB300) and ~1/2 memory (vs FP8), so the convergence and accuracy results above support realizing the hardware advantage. (source: Tab.1)

Summary: the paper’s experimental results show that NVFP4 maintains FP8-level accuracy in large-scale (12B, 10T) pretraining while the format choice alone also yields a training-efficiency (token, time) benefit. In addition, a small amount of high precision at the end can shrink the loss gap to the 0.5% level, making the practical tradeoff clear. (source: Fig.2/3/6/7, Tab.2)

Prompt 1.4.2 (critical comparison)

Relative to the main baselines and SOTA models mentioned in the paper, how does the proposed method perform? Identify the specific comparison point that most strongly supports the superiority claim. Conversely, summarize any results where it did not beat, or barely improved on, the baselines and give the reasons.

Bottom line: against the FP8 baseline under the same architecture, data, and schedule, NVFP4 tracks closely — <1% validation loss (stable regime) and ~1.5% in the final phase (source: Fig.2) — and downstream is broadly on par; on some tasks (e.g., GSM8K, MGSM) it is even +3.2~+3.7pp ahead (BF16 eval) (source: Tab.2). The coding family (HE+, MBPP+), however, comes in −2.5~−3.2pp behind (source: Tab.2). Within the same FP4 family, NVFP4 vs MXFP4 shows losses of 1.5% vs 2.5%, and MXFP4 needs +36% more tokens (1.36T) for the same loss — an efficiency edge for NVFP4 (source: Fig.6).

Quantitative comparison against SOTA/baselines

1) NVFP4 vs FP8 (same 12B, 10T pretrain, BF16 eval)

(source: Tab.2, BF16 eval)

Key point: the loss curve tracks FP8 tightly — <1% error in the stable regime, ~1.5% in the lr-decay phase (source: Fig.2) — and the figure/table repeatedly confirm broad parity across domains (knowledge, math, commonsense, multilingual) (source: Fig.3, Tab.2).

2) NVFP4 vs MXFP4 (same 8B, 1T~1.5T pretrain)

• Relative loss (vs BF16): NVFP4 ≈ 1.5%, MXFP4 ≈ 2.5% (at 1T). (source: Fig.6a)
• Token efficiency: MXFP4 needs +36% more tokens (1.36T vs 1.0T) to reach NVFP4’s loss → directly higher training time, compute, and energy. (source: Fig.6b)

Supporting evidence (hardware implications): on Blackwell, FP4 gives 2× (GB200)/3× (GB300) compute throughput and ~1/2 memory (vs FP8) → at equal loss, FP4’s system-level performance/cost edge is latent (the paper explicitly leaves runtime and system optimization out of scope). (source: Tab.1; Intro note)

The comparison points that most strongly support the ‘superiority’ claim

1. Token efficiency (NVFP4 vs MXFP4): the +36% token difference is strong evidence that directly compares the cost of reaching the same loss — a controlled format experiment with identical model, data, and schedule. (source: Fig.6)
2. Long-horizon convergence stability (12B, 10T): tracking FP8 at <1% (stable) / ~1.5% (final) error, with the multi-stage data-blend/lr switch events explicitly annotated to establish causality. (source: Fig.2)
3. Downstream parity: MMLU-Pro 62.58 vs 62.62, ARC-C tie at 91.81, and other headline metrics on par, with some math/multilingual tasks ahead (+0.9~+3.7pp). (source: Tab.2)

Where it did not beat, or barely improved (and possible reasons)

• Coding-family drop (HE+, MBPP+): −2.5~−3.2pp. The authors raise eval noise / checkpoint-selection effects as a possibility (MBPP+ dips at the final checkpoint). (source: Tab.2, main text)
• Slightly behind on MATH: −1.84pp — possibly a subtle precision loss in long math reasoning or sensitivity to precision/scale at the end. (source: Tab.2)
• No absolute SOTA (cross-model) competition: the paper focuses on precision-format/recipe comparison within the same architecture (hybrid Mamba-Transformer); it does not provide a head-to-head performance table against large external SOTA (different models/data). (source: §3 overview)
• No system measurements: the paper states that quantifying actual training time/energy/cost (e.g., TFLOP utilization, $/1M tok, kWh/1M tok) is beyond its scope — it focuses on format and algorithm. (source: §1 summary)

Context: why each ‘recipe’ element is necessary (vs internal baselines)

• Ablations: removing any of grad-SR, Wgrad-RHT, 2D weight scaling, or some BF16 layers worsens loss — each is essential for stable convergence over the 10T horizon. (source: Fig.4)
• SR target: effective only on gradients (counterproductive elsewhere) → the key to bias reduction. (source: §4.4, Fig.10)
• RHT location/size: Wgrad-only, with d=16 closest to the optimal compute–accuracy balance. (source: §4.2, Fig.11–12)
• 2D weight scaling: mitigates the quantization mismatch → chain-rule violation between fprop/bprop, improving loss. (source: §4.3)

Overall interpretation

The strong claim (supporting evidence): “achieving FP8-level accuracy at FP4 precision” is jointly supported by the same-setting comparison (12B·10T) loss/downstream numbers and the +36% token-efficiency over MXFP4. (source: Fig.2/3/6, Tab.2)

• Limitations (interpretation caveats): the coding deficit, the absence of direct cross-comparison with external SOTA, and the lack of measured system-level gains (time/cost/energy) remain to be validated. Even so, the theoretical hardware advantage and the controlled format experiments point to a promising path for production FP4. (source: Tab.2, Tab.1)

Summary: the NVFP4 recipe enables FP4 training with no (or negligible) accuracy loss vs FP8 and shows a clear token/time-efficiency advantage over the same FP4 family (MXFP4). (source: Fig.2/6, Tab.2)

Prompt 1.5.1 (acknowledged limitations and potential ones)

What limitations, weaknesses, or failure cases do the authors explicitly acknowledge? Based on your analysis, what do you see as potential limitations (strong assumptions, scalability, computational cost, generalization limits, social impact, and so on)?

Bottom line: the authors acknowledge that quantizing every linear layer to FP4 diverges early in training, and that stability requires some layers (≈the last 8 blocks, ~15%) kept in BF16 together with SR (gradient-only), 2D weight scaling, and Wgrad-only Hadamard, all used as a ‘set’ (numbers: loss gap <1% in the stable regime and ~1.5% in the final phase; switching only the forward pass to BF16 at 8.2T recovers 1.5%→0.5%, with high-precision compute ≈6%) (source: Fig.2, Fig.7, Fig.8/E.1). In addition, system/runtime evaluation is out of scope, so the actual speed, energy, and cost gains are not reported (source: §1 end).

1) Limitations/weaknesses the authors explicitly acknowledge

• FP4 alone diverges: “full-layer FP4 without extra techniques” diverges early in training, so keeping part of the final blocks in BF16 is key (e.g., keeping the last 4 blocks in BF16 is stable) (source: Appx E.1/E.2, Fig.9).
• Constraints on SR’s scope: SR must be applied only to gradients to stay stable (at 12B, both Dgrad and Wgrad need it). SR on activations/weights triggers divergence (error larger than nearest rounding) (source: §4.4, Appx E.3, Fig.10).
• Chain-rule inconsistency risk: forward/backward use different scale/transform axes, so the same weight can take ‘different quantized representations’, violating the chain rule → must be mitigated with 2D weight scaling (16×16) (source: §4.3).
• The forward path is the main driver of the loss gap: switching only the forward pass to BF16 from 8.2T recovers the gap from 1.5% to 0.5%, with high-precision compute ≈6% → small overhead (source: Fig.7, Appx D).
• Some coding tasks degrade: HumanEval+/MBPP+ −2.5~−3.2pp; the authors mention eval noise / checkpoint choice as possible causes (source: Fig.3·Tab.2 discussion, Tab.2).
• Scope limits (system/runtime): this report is algorithm- and methodology-centered and does not cover runtime efficiency or system optimization (source: §1 end).
• Unfinished tasks (future work): (i) refining the recipe toward quantizing all linear layers to FP4, (ii) reducing the remaining BF16 layers, (iii) extending to attention and communication paths, (iv) validation on larger models/longer token horizons/MoE, (v) applications in post-training (source: Conclusions).
• No format-specific scaling laws yet: a scaling law for NVFP4 vs MXFP4 across parameter/token horizons needs further study (source: Fig.6 comment).

2) Potential limitations (analysis-based)

• The structural constraint of ‘partial high precision’: currently ≈15% (the last 8 blocks) is kept in BF16 for stability (12B) → reaching full FP4 requires automating layer selection (e.g., with a Wgrad-error metric) (source: Fig.6 setup, Appx E.2 comment).
• Architecture generalization: the validation focuses on the Mamba-Transformer hybrid at 12B/10T. Generalization to MoE / pure transformers / other optimizers is unverified (the authors also list it in their future plans) (source: §3 setup, Conclusions).
• Uncertain end-to-end speed gains: although FP4 offers 2× (GB200)/3× (GB300) math throughput and 1/2 memory (vs FP8), attention is memory-bound during decoding, so the Tensor-Core advantage may not fully carry over → the reason to extend FP4 to attention/communication paths (source: Tab.1; external background Hydragen §2.2–2.3).
• Eval variance/checkpoint sensitivity: coding benchmarks such as MBPP+ have high variance, so small gaps can appear (the authors concede this) → the eval protocol/checkpoint sampling should be refined (source: Fig.3 discussion).
• Recipe complexity and implementation risk: combining SR (gradient-only), 2D scaling, RHT (Wgrad-only), and partial BF16 raises reproduction difficulty if framework/kernel support is missing (the authors provide Transformer Engine updates) (source: Fig.4, code notice).
• Format-choice sensitivity: MXFP4 needs +36% more tokens (1.36T vs 1.0T) for the same loss → a mis-designed format/scale choice can sharply inflate training cost (including the missing scaling law) (source: Fig.6).

3) Context numbers (reference)

• Loss-gap profile: <1% in the stable regime, ~1.5% in the final (lr-decay) phase. Switching only the forward pass to BF16 at 8.2T recovers the gap to 0.5%, with high-precision compute ≈6% (source: Fig.2, Fig.7).
• Hardware ceiling (theoretical): FP4 compute 2×/3× (GB200/GB300), memory ~1/2 (FP8) — but end-to-end time, energy, and $/1M tok are not reported in this report (source: Tab.1, §1 end).

One-line summary

NVFP4 approaches FP8 in ‘accuracy–stability’, but full FP4 still requires reducing the per-layer high-precision dependence, resolving forward-path sensitivity, extending to attention/communication paths, establishing format-specific scaling laws, and validating system metrics (time, energy, cost) (source: Fig.2/6/7, §4.3, Conclusions).

Prompt 1.5.2 (future research trajectory)

What future research directions do the authors propose? Suggest reasonable next steps or alternative directions in light of the limitations.

Bottom line: the authors present as their formal roadmap (i) format-wise performance characterization and scaling laws, (ii) reaching full FP4 across all linear layers (reducing the remaining BF16), (iii) extending NVFP4 to attention and communication paths, (iv) post-training applications, and (v) validation on larger models/longer token horizons/MoE (source: Conclusions). They also stress the need to compare scaling laws with MXFP4 across varied parameter and token horizons (source: Fig.6 comment). Building on this, reasonable next steps are automating layer selection (error-metric based), optimizing the forward-only precision-switch point, publishing system metrics (TTFT/TPOT/tokens per s/kWh per 1M tok), and building FP4 kernels for the attention/communication paths (source: Fig.7·Appx E.2/3·Tab.1·scope note in the intro).

The authors’ official future work

• Cross-format characterization & scaling laws: systematically compare NVFP4’s pretraining performance against other formats and establish scaling laws over parameter count and token horizon (e.g., 8B/12B/≥70B, 1T~10T+) (source: Fig.6 comment).
• Full-layer FP4 convergence: refine the methodology so that quantizing all linear layers to FP4 does not degrade convergence, and shrink the remaining high-precision (BF16) layers (currently the last 8 blocks ≈15%) (source: Conclusions; Fig.6 setup).
• Extending to attention/communication paths: extend NVFP4 to Attention and communication paths (currently mostly the GEMM path) (source: Conclusions).
• Post-training applications: explore NVFP4 in post-training scenarios such as PTQ/FT/SFT/RLHF (source: Conclusions).
• Scale-up validation: extend evaluation to larger models, longer token horizons, and MoE architectures (source: Conclusions).
• Toolchain support: add NVFP4 training support to Transformer Engine (securing an engineering path) (source: Conclusions).

Reasonable next steps (proposed) in light of the limitations

1. Automating layer selection (toward full FP4)

• Rationale: the layers near the output (last blocks) are sensitive to FP4, and a Wgrad quantization-error metric may help select which layers stay high-precision (proposed by the authors) (source: Appx E.2).
• Proposal: drive dynamic BF16↔FP4 gating and a curriculum precision schedule from online Wgrad-error thresholds (e.g., L2/amax/zero-rate), optimized jointly with training.

2. Optimizing the forward-only precision-switch schedule

• Rationale: the forward path is the main driver of the loss gap; switching only the forward pass to BF16 at 8.2T yields a relative-gap drop from 1.5% to 0.5% with just ~6% high-precision compute (source: Fig.7).
• Proposal: detect change points such as the start of lr decay or data-blend switches (e.g., EMA-loss slope, grad-norm drift) to set the switch timing/interval adaptively.

3. FP4 kernels for attention/communication paths & reporting system metrics

• Rationale: FP4 offers 2× (GB200)/3× (GB300) math throughput and ~1/2 memory vs FP8, an acceleration opportunity for GEMM-heavy stretches (source: Tab.1). The report states runtime/system optimization is out of scope (source: §Intro/Scope).
• Proposal: implement FP4-friendly attention/communication combined with FlashAttn/FlashFFTConv-family kernels, and publish TTFT (ms), TPOT (ms/tok), throughput (tok/s), $/1M tok, and kWh/1M tok (PUE) at identical batch and context length.

4. Quantitative modeling of format/scaling laws

• Rationale: NVFP4 vs MXFP4 shows a +36% token (1.36T vs 1.0T) difference for the same loss → format choice is cost-sensitive (source: Fig.6).
• Proposal: fit per-format scaling laws for $(\mathcal{L}(N_{\text{tok}},P;\ \text{format}))$ (over tokens, parameters, data quality) by regression to pre-design the optimal format and token budget.

5. Refining the boundaries of SR/2D-scaling/RHT

• Rationale: SR works only on gradients (SR on forward/weights triggers divergence, with larger error than nearest); 2D weight scaling mitigates chain-rule inconsistency; Wgrad-only RHT is the stable choice (source: §4.3–4.4, Appx E.3/E.4).
• Proposal: run a per-tensor SR policy search (grad only, separating dgrad/wgrad), tune 2D-scale granularity (8×16/16×16), and standardize the compute–accuracy Pareto of RHT sizes d=16/32 per architecture.

6. Scale-up & architecture generalization

• Rationale: validation is planned for larger models, longer tokens, and MoE (source: Conclusions).
• Proposal: quantify, on 70B Dense, 64×(A=8) MoE, and ≥20T tokens, whether the loss gap (target <1%) holds and how FP4 affects the token-optimal regime.

7. Post-training protocols

• Rationale: the authors plan to explore post-training scenarios (source: Conclusions).
• Proposal: publish the stable conditions for NVFP4 fine-tuning under LoRA/DoRA · SFT/RLHF (learning rate, clipping, scale-update policy) together with accuracy–cost curves.

Reference: the constraints identified so far and their motivation (summary numbers)

• The forward path is the main driver of the loss gap → switching only the forward pass to BF16 at 8.2T brings the gap from 1.5% to 0.5%, high-precision compute ≈6% (source: Fig.7).
• Layer sensitivity exists: keeping the last blocks converges; keeping only the early blocks is insufficient → error-metric-based layer selection is viable (source: Appx E.2).
• Format-efficiency difference: MXFP4 needs +36% more tokens to match NVFP4’s loss (source: Fig.6).
• Hardware potential: FP4 compute throughput 2× (GB200)/3× (GB300), memory ~1/2 (vs FP8) (source: Tab.1).
• Scope limits: this report is algorithm-centered and does not include runtime/system optimization (source: §Intro/Scope).

One-line summary: combining the authors’ roadmap (format, layers, paths, scale, post-training, scale-up) with our proposed automation, scheduling, system metrics, and kernelization can prove full FP4 (all layers) convergence and the end-to-end cost–accuracy Pareto in real production settings (source: Conclusions/Fig.6/Fig.7/Tab.1).