[Paper Review] Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models

Wed, 14 Jan 2026 00:00:00 +0900

Paper

Engram: The Second Sparsity Axis After Conditional Computation (MoE), Conditional Memory

One-Line Summary (TL;DR)

Engram adds a “knowledge lookup primitive” to the Transformer, offloading the static-knowledge/local-pattern reconstruction burden performed by standard models in early layers to constant-time $O(1)$ N-gram memory lookup, and simultaneously shows broad benchmark gains over MoE under iso-activated conditions — MMLU +3.0, BBH +5.0 — together with a sharp NIAH-MQ +12.8pt (iso-loss) jump in 32k long context.

N-Gram on Jaehun's Blog

[Paper Review] Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models

Engram: The Second Sparsity Axis After Conditional Computation (MoE), Conditional Memory

One-Line Summary (TL;DR)