Channel Selection for Mixture-of-Experts Decoding Needs No Calibration Data

Paper

Introduction

In settings where the batch size is close to 1, such as conversational services, the latency of a mixture-of-experts language model comes mostly from reading the routed expert weights from GPU memory. Most of the intermediate dimensions inside an expert contribute little to a token, so choosing which channels to read first can substantially reduce the amount read. This paper shows that two objectives that existing channel selectors try to meet through calibration data are in fact not achieved, and presents a selector built from model weights alone, without calibration.

Background: Channels and Retention Rules

In an MoE layer, the router sends a token to a few experts. An expert is a gated MLP. Each intermediate dimension of this expert is called a channel. Channel $j$ owns the $j$-th row of the gate matrix $G_e$ and the up matrix $V_e$, and the $j$-th column of the down matrix $D_e$.

Given an input $x$, channel $j$ computes the following.

• Gate activation: $\phi(g_j^\top x)$
• up output: $(v_j^\top x)$
• Channel activation: $a_j = \phi(g_j^\top x)(v_j^\top x)$

The full routed output is the sum of the $a_j d_j$ produced by each channel, scaled by the expert’s router weight.

For illustration, consider Qwen3-Thinking (the actual structure in §2). A single token goes to 8 experts, and each expert has 768 channels. So at each layer the selector must choose what to read among $8 \times 768 = 6144$ candidate channels. Once it decides to compute channel $j$, it reads row $j$ of gate/up and column $j$ of down, computes $a_j d_j$ exactly, and adds it to the output. Discarded channels are not read at all (as shown later, the reader fills them in approximately).

The central fork in a channel selector is the retention rule. Even with the same scores, the way of deciding which channels to keep changes the amount read and the accuracy.

A threshold is intended to give each token as much as it needs, but the actual amount read moves with the input. top-k fixes the amount in advance, but keeps $k$ channels regardless of what the token actually needs.

Consider Fig. 2 of the original. (a) draws the structure in which channel $j$ owns row $j$ of gate/up and column $j$ of down. (b) shows the difference that, at the same score, a threshold changes the retention count per token while top-k fixes it. (c) shows how cost is computed. Modern NVIDIA GPUs fetch memory in 32-byte sectors, so when the needed bytes are scattered, every touched sector is charged in full.

The paper calls this cost reads. It is the bytes read per token measured as a percentage of the full model’s routed expert reads. The accounting was also checked against a profiler to confirm it matches actual hardware transfers. For batch-1 Qwen3-Thinking, cold-cache DRAM reads are 18.3%, close to the accounted 17.8% (Appendix C).

Read savings are not compute (FLOPs) savings. They are the gain from reading fewer weights in a memory-bandwidth-bound setting.

Calibration Fails Both Promises

A channel selector uses a calibrated threshold for two reasons: to know the amount read in advance, and to improve accuracy by giving tokens the channels they need at that amount. The authors break each of these in turn with experiments.

A Threshold Does Not Fix the Amount Read

The threshold $\tau$ is fixed, but the number of channels crossing it varies with the input.

Consider an experiment that computes channel scores exactly as their actual contribution to the output and calibrates $\tau$ so that about 12% is retained on UltraChat. On separate conversational data not used for calibration, the average retention count matched the target. Yet even on the same conversational data, the middle 90% of Qwen3-Thinking tokens retain only 9.0–16.2%, a wide spread, and on benchmark answers (GPQA-Diamond, IFEval, MMLU-Pro) the scores become smaller, dropping the median to 10.0%.

The retention count decreases as answers get longer. Around 6,000 answer tokens, Qwen3-Thinking retains only 71% of the calibrated count, Gemma 4 92%, and Qwen3.5 88%.

In actual deployment it is simpler. Across all 72 benchmark runs we performed, selectors using a calibrated threshold read less than the target. Prox was calibrated to 18% but actually reads 12.9% on average and as little as 9.9%. The reader’s threshold version also meets the target on calibration data but reads 0.4–5.8%p less on benchmarks (§4.1).

Let us read Fig. 4 of the original. (a) is the distribution of the channel retention ratio per token. The orange threshold curve meets the target on the chat data used for calibration, but on benchmark answers the median drops to 10%. The thick line is top-k, which keeps the same ratio for every token. (b) plots, as the median over 500-token windows, what percentage of the calibrated value the retention count is at each answer length. (c) plots actual amount read against the budget on the horizontal axis. Every point lies below the diagonal, that is, short of the target.

We also checked why it decreases. The retention count follows the answer written so far rather than how much the token needs. At the threshold Prox calibrated to 13%, if the 256-token window at position 6,000 of a Qwen3-Thinking GPQA-Diamond answer is detached from its preceding context and placed immediately after the question, the same tokens retain 57% more channels. Nor do harder tokens receive more channels. The correlation between retention count and the full model’s next-token loss is only −0.29 to +0.14 (§4.1, Appendix E).

Calibration Gives No Consistent Accuracy Gain over top-k

To compare the two retention rules, the scores must be fixed. The authors ran each selector twice with the same channel scores: once with a calibrated threshold and once with top-k. At budgets of 13% and 18%, with generation tasks on 4 MoE models and a likelihood task on OLMoE, and with 3 kinds of scores (Prox, TEAL, and Prox scaled at the output level), they produced 43 comparisons in total.

That is, in 40 of 43, top-k matches or wins. Here a tie means no difference was detected, not equivalence. Grouped by model as well, no model prefers the threshold at the calibration budget (Appendix Table 14).

Even in a check favorable to the threshold, which reruns the top-k version at the amount the threshold version actually read, top-k matches or wins in 37 of 41 (Appendix F, Fig. 8).

In Fig. 5 of the original, each point is one comparison. The vertical axis is the threshold version’s accuracy minus the top-k version’s, so below 0 (blue) is a top-k win. Gray is a tie, the 10 blue points are top-k wins (four of them from Gemma 4 at 38–45 points), and only 3 orange points are threshold wins (at most 2.8 points). This figure is based on the calibration budget; the version rerun at actual reads is in Appendix Fig. 8.

Long Reasoning Can Collapse under a Calibrated Threshold

The four largest gaps in Fig. 5 all come from Gemma 4. At the 13% budget, top-k with Prox (and its output-scaled variant) beat the threshold version by 38–45 points.

Gemma 4 with the calibrated threshold repeats sentences and reaches the 32,768-token limit. This is not because it reads less.

The cause was also narrowed down. Changing only Prox’s channel threshold to top-k makes the collapse disappear. Changing only the input threshold does not. Neither flooring the channel minimum retention at 4% nor recalibrating on Gemma 4’s own reasoning prevents the collapse (§4.3, Appendix D.1, F.2).

The authors conjecture that this collapse acts like a cliff. Once a sparse model starts repeating, it is hard to stop on its own, and when the error that sparsification adds per token exceeds some model-specific level, accuracy falls sharply rather than gradually. This explanation is a hypothesis, not a proof (§4.3).

A Reader Built from Weights Alone

When top-k is the retention rule, performance depends on two things: how well channels are ranked before reading, and how discarded channels are handled. The observation that changing the scores moves KL divergence far more than changing the retention rule set the design direction of the reader (§5).

The reader is built once from model weights before deployment. It has no values calibrated to the target model. Three settings—128-coordinate rotation blocks, an 80-coordinate partial scan, and rank-32 recovery—are fixed once on OLMoE development data and reused for all models. For each token and layer, the reader runs three stages. It screens all candidate channels, computes the top $k$ exactly, and recovers the rest.

Consider Fig. 6 of the original. The top contrasts the full model, which reads all channels of the routed experts, with the reader, which reads a ternary sketch, the real weights of the retained $k$ channels, and a recovery map. The middle shows the behavior of the three stages. The bottom breaks down the 17.8% reads on Qwen3-Thinking. The sketch accounts for 4.9%, the exactly computed channels for 12.3%, and the recovery map for 0.6%, while the remaining roughly 82% is not read at all.

Stage 1: Screen with a Ternary Sketch

Weights and hidden states have outliers that make low-bit codes inaccurate. So the gate row $g_j$ and up row $v_j$ are rotated in 128-coordinate blocks by a random Hadamard transform $R$, spreading large values evenly within the block. $R$ is orthogonal, so the rotation does not change the activation ($g_j^\top x = (Rg_j)^\top(Rx)$).

The rotated blocks are stored as follows.

A block shrinks from 256 bytes in BF16 to about 30 bytes, so the sketch adds about 7% to resident memory. At decode time, $x$ is rotated once and multiplied with the sketch to obtain $\hat g_j \approx g_j^\top x$, $\hat v_j \approx v_j^\top x$ (§5.1).

At low budgets such as 12.8% and 17.8%, reading the entire sketch alone accounts for 44–61% of all reads (for OLMoE). So a partial scan is used. From each block, only the ternary codes of the 80 coordinates with the largest $|Rx|$ are read. This increases the number of exactly computed channels by 20–50% at the same budget (§5.1).

Stage 2: Compute the Top k Exactly with Output-Level Scores

The magnitude that channel $j$ adds to the output is $w_j\,\lVert d_j\rVert\,|a_j|$. Here $w_j$ is the router weight, and $\lVert d_j\rVert$ is the norm of the down column, computed once at load time. Replacing the activation $a_j$ with the sketch estimate gives the score

(Equation 1). Because the score is at the output level, channels from different experts can be compared directly. So the top $k$ are chosen across all experts the token passes through. One expert may receive many channels and another none at all (§5.2).

The $k$ channels remaining after screening have their real weights read to compute $a_j$ exactly. To read the down weights contiguously, the down projection is transposed once at load time so that prefill and decode use the same layout.

Stage 3: Recover Discarded Channels with a rank-32 Map

The contribution of a single discarded channel is small, but thousands per layer add up to something not negligible. The screening stage has already computed an activation estimate $p_j = \phi(\hat g_j)\hat v_j$ for every discarded channel, so it is reused without reading gate/up again. The remaining problem is the down column $d_j$ of each discarded channel.

Instead of reading full-width columns, all down columns of the layer are stacked to obtain the top 32 left singular vectors $U \in \mathbb{R}^{H \times 32}$, and the coefficient $b_j = U^\top d_j$ of each channel is stored. The final output is

The first term is the exact contribution of the retained $K$ channels. The second term adds the estimated contribution of the discarded $S$ channels via the rank-32 map (Equation 2). The map adds about 0.5% to resident memory and 0.5–0.8%p to reads (§5.3).

Accuracy and Speed

Accuracy

Accuracy was evaluated on 9 pairs of three post-trained models (Qwen3-Thinking, Gemma 4, Qwen3.5) and three benchmarks (GPQA-Diamond, IFEval, MMLU-Pro). The full model processes the prompt, and the selector runs for each subsequent decoding token. Accuracy is reported as a percentage of the full model’s accuracy. The table below reconstructs only the rows needed for the roughly 18% budget from Table 1. For calibrated selectors, both the target and the actual reads are listed.

The reader is the most accurate at both low budgets. At 17.8% it averages 98.1% and 95.0% on the worst pair. At 12.8% it still averages 94.5% and 86.9% on the worst pair, better than the worst pair of any other selector.

The notable place is Prox. It was calibrated to 18% but actually reads 12.9%, similar to the reader’s 12.8%, yet its mean accuracy is only 68.5%. Most of the gap comes from Gemma 4’s repetition loop. On the two Qwen models, Prox matches the reader at similar reads. top-k does not rescue a weak score either. WINA with top-k reaches only 28.4% even at 20% reads (§6.1).

Speed

Speed must be viewed separately as kernel performance and full-system performance. The Triton kernel was compared at 17.8% reads, on a single RTX PRO 6000 Blackwell GPU, against the fastest configuration of unmodified vLLM.

Fig. 7 (a) of the original is decode speedup of the MoE block alone. At batch 1 it is 1.90x for Qwen3-Thinking, 1.71x for Gemma 4, and 1.60x for Qwen3.5. At batch 4 it is 2.39/2.34/2.51x, and the maximum is 2.69x for Qwen3.5 at batch 8.

(b) is full-model performance. Further right means faster per user, and higher means more output per GPU. Because attention is unchanged the gain is small: 1.15–1.36x at batch 1, peaking at 1.75/1.74/1.88x at batch 8–16, and converging to 1.00–1.24x at batch 128 (Table 20). On long contexts of 8k–32k tokens it is 1.08–1.54x.

The reason the gain at batch 1 is smaller than the read savings is fixed cost. On Qwen3-Thinking, even though the MoE block accounts for only 18.3% of full-model memory traffic, the block is only 1.90x faster rather than 5x. Fixed costs such as kernel launch do not shrink with read savings. As the batch grows, these fixed costs are shared, so block speedup increases up to batch 4–8. But as the batch grows further, the union of channels selected by the tokens grows again, increasing the amount that must be read and reducing the gain (§6.2, Appendix J).

Limitations and When to Apply It

The authors acknowledge five limitations (§7).

1. The kernel converts only part of the read savings into speed. It is tuned for one budget (17.8%) and one GPU type.
2. Selecting channels for the whole batch at once so that the gain carries to large batches is future work.
3. The number of coordinates in the partial scan (80 of 128) was set on OLMoE and confirmed only on Gemma 4.
4. The reader changes only the routed experts during decoding. The sketch and recovery map add about 7% and 0.5% to a BF16 checkpoint, and their share is larger in quantized checkpoints.
5. The tested models are at most 35B parameters.

There are also two patterns the authors conjecture hold more broadly. These are conjectures, not verified results (§7). One is that short likelihood evaluations hide failures in long generation, and the other is that low KL divergence does not imply high accuracy.

From a reviewer’s perspective, the remaining condition is this. The conclusion is limited to ’the case of using a calibrated threshold on the channels of frozen MoE experts.’ The authors themselves state that thresholds may be valid elsewhere, such as in sparse attention or learned routers (§3). The cliff explanation is also a hypothesis, not a proof.

The operational sweet spot is the decode stage when serving a post-trained MoE model at small batch while the prompt is processed by the full model. Conditions not yet verified are very large batches beyond 128, other GPU types, quantized checkpoints (confirmed only on OLMoE), and the prefill stage.

Figures in this post are taken from the original arXiv:2610.37386 (CC BY 4.0). Only size and format were changed.