Channel Selection for Mixture-of-Experts Decoding Needs No Calibration Data
Introduction In settings where the batch size is close to 1, such as conversational services, the latency of a mixture-of-experts language model comes mostly from reading…
Mixture-of-Experts
4 pages
The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction
Waking the MoE on SSD: Serving a 35B MoE at 20 tok/s on a 24 GB Desktop with Edge0 One-line summary (TL;DR) Even at 4-bit, a 35B-class MoE occupies 19.5 GB and cannot…
Cache-Aware Joint Router Adaptation for Memory-Efficient MoE Inference
Cache-Learning Routers: Cache-Aware Joint Router Adaptation for Memory-Constrained MoE Inference TL;DR Starting from the observation that MoE's real bottleneck is not…
SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers
SMELT: Scaling Laws for Compute-Matched MoE Loop Transformers TL;DR Loop transformers scale depth by executing a layer block repeatedly, but most prior work fixed only…