SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers

Mon, 07 Sep 2026 00:00:00 +0900

Paper

SMELT: Scaling Laws for Compute-Matched MoE Loop Transformers

TL;DR

Loop transformers scale depth by executing a layer block repeatedly, but most prior work fixed only the parameter count and let FLOPs grow, racking up “free wins.” SMELT shows that looping is a pure architectural gain even while matching all three budgets — per-token FLOPs, total parameters, and KV cache — simultaneously (source: §1). On the compute-optimal frontier it saves 6.8–18.0% of training FLOPs, and these gains come out larger downstream than validation loss predicts (source: §4.3, §5.1).

2609.01343 on Jaehun's Blog

SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers

SMELT: Scaling Laws for Compute-Matched MoE Loop Transformers

TL;DR