The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction

Fri, 18 Sep 2026 00:00:00 +0900

Paper

Waking the MoE on SSD: Serving a 35B MoE at 20 tok/s on a 24 GB Desktop with Edge0

One-line summary (TL;DR)

Even at 4-bit, a 35B-class MoE occupies 19.5 GB and cannot reside on a 24 GB machine (source: §1). Edge0 keeps expert weights on SSD and reads them with mmap streaming, hides disk latency behind compute with a prerouter that predicts next-layer routing one token ahead, consumes that prediction itself as routing to eliminate drops, and repays int4 + routing-replacement loss with an unmerged recovery LoRA (source: §3.2, §3.3). The result is 20.4 tok/s on the 35B tier on a 24 GB Mac mini M4 Pro, 2.9 GiB peak active memory, and a 3.9-point average gap to the fp16 teacher (source: Tab.1, Tab.2).

Hardware Architecture on Jaehun's Blog

The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction

Waking the MoE on SSD: Serving a 35B MoE at 20 tok/s on a 24 GB Desktop with Edge0

One-line summary (TL;DR)