[Paper Review] Memory Retrieval and Consolidation in Large Language Models through Function Tokens

Sat, 20 Dec 2025 00:00:00 +0900

Paper

Function Token Hypothesis: Why Punctuation and Newlines Gate LLM “Memory Retrieval”

One-line Summary (TL;DR)

This paper defines LLM memory as “reactivation of features (retrieval) + feature expansion through learning (consolidation)”, and presents the Function Token Hypothesis that function tokens (high-frequency tokens) context-conditionally reignite the features needed for prediction at inference time, while in pre-training function→content prediction dominates optimization and grows the feature count — argued through quantitative (coverage/degree), qualitative (reactivation trace), causal (steering), and learning-dynamics (loss decomposition) evidence. (source: §1, Fig.1, §3.1, Tab.1, §3.2, Fig.6, Fig.7, §4.2, Fig.8, §4.3, Fig.9)

Interpretability on Jaehun's Blog

[Paper Review] Memory Retrieval and Consolidation in Large Language Models through Function Tokens

Function Token Hypothesis: Why Punctuation and Newlines Gate LLM “Memory Retrieval”

One-line Summary (TL;DR)