Kalman Delta Networks: Uncertainty-aware Associative Memory

Sat, 12 Sep 2026 00:00:00 +0900

Linear Attention Rewritten with a Kalman Filter: Kalman Delta Networks

TL;DR

The fixed-size recurrent memory update of linear attention is reinterpreted as a linear-Gaussian state-space model, and the covariance tracking of the optimal estimator, the Kalman filter, is introduced to decide “how strongly to overwrite” based on uncertainty, yielding Kalman Delta Networks (KDN). DeltaNet, Gated DeltaNet, and KDA reduce to fixed-gain special cases that omit the covariance, and KDN outperforms every SOTA linear recurrent mixer evaluated at 750M/1.3B pretraining in both PPL and average accuracy (source: Abstract, §1).

Model-Architecture on Jaehun's Blog

Kalman Delta Networks: Uncertainty-aware Associative Memory

Linear Attention Rewritten with a Kalman Filter: Kalman Delta Networks

TL;DR