Cache-Aware Joint Router Adaptation for Memory-Efficient MoE Inference

Wed, 09 Sep 2026 00:00:00 +0900

Paper

Cache-Learning Routers: Cache-Aware Joint Router Adaptation for Memory-Constrained MoE Inference

TL;DR

Starting from the observation that MoE’s real bottleneck is not compute but expert-weight movement, this work proposes the Temporal Router and Spatio-Temporal Router, which learn the cache-residency priority itself via post-training while keeping the native Top-$K$ selection rule unchanged (source: §1). On Qwen3-30B-A3B-Instruct-2507, the Temporal-only mode achieves hit-rate gains of +10.46~+33.34 pp and traffic reductions of -28.0~-79.9% over classic replacement policies, with zero additional proactive loads ($P=0$ MB/token); the full mode records adjusted hit-rate gains of +1.15~+18.03 pp and per-token load reductions of -4.6~-53.3% over the strongest prefetch baseline, ProMoE (source: Tab.1, §4.2).

2609.04895v1 on Jaehun's Blog

Cache-Aware Joint Router Adaptation for Memory-Efficient MoE Inference

Cache-Learning Routers: Cache-Aware Joint Router Adaptation for Memory-Constrained MoE Inference

TL;DR