Machine Learning on Jaehun's Blog

[Paper Review] NVIDIA Nemotron 3: Efficient and Open Intelligence

Tue, 16 Dec 2025 00:00:00 +0900

Paper

NVIDIA Nemotron 3: Pushing the “Accuracy/Throughput” Frontier with a Hybrid Mamba–Transformer MoE

Nemotron 3 combines an MoE hybrid Mamba–Transformer, LatentMoE, MTP, NVFP4 training, and multi-environment RL to push up “accuracy-to-inference-throughput,” presenting contexts up to 1M tokens and 3.3× relative throughput as its core message (source: §Intro/§2.2/§2.3/§2.4/§2.5/§2.6/Fig.2).

[Paper Review] Radial Attention: O(n log n) Sparse Attention with Energy Decay for Long Video Generation

Mon, 15 Dec 2025 00:00:00 +0900

Paper

Radial Attention: O(n log n) Sparse Attention with Energy Decay — Generating “Long Videos” Cheaply

One-Line Summary (TL;DR)

Based on the observation that post-softmax attention energy decays exponentially with distance (in time/space), the authors design a static mask that decays compute density in the same way, substantially cutting the training/inference cost of long-video generation. (source: §4.1–§4.2, Eq.3–Eq.4)

[Paper Review] Pretraining Large Language Models with NVFP4

Thu, 09 Oct 2025 00:00:00 +0900

Paper

NVFP4 4-bit Pretraining, Made Practical: 12B up to 10T Tokens, Effectively On Par with FP8

TL;DR

Pretraining a 12B hybrid Mamba-Transformer on 10T tokens at NVFP4 (4-bit) closely tracks FP8, with a loss gap of <1% in the stable regime and ~1.5% in the final phase, while downstream performance is also mostly on par (some math and multilingual tasks +0.9–+3.7pp). Matching MXFP4’s loss requires +36% more tokens (1.36T vs 1.0T), confirming NVFP4’s token-efficiency advantage. (source: Fig.2, Tab.2, Fig.6)

[Paper Review] Inference-Time Hyper-Scaling with KV Cache Compression

Tue, 29 Jul 2025 00:00:00 +0900

Link to Paper

Dynamic Memory Sparsification (DMS): Making LLM Hyper-Scaling a Reality with 8× KV Cache Compression

One-Line Summary (TL;DR)

DMS, combining a lightweight re-fitting of just 1K steps with a delayed eviction strategy, achieves up to 8× KV cache compression while paradoxically boosting performance, such as a +9.1 pt gain on AIME 24 for Qwen-R1 32B. This allows for generating longer and more tokens in real-time within the same compute and memory budget.