TL;DR — Routing an LLM at the token level rather than the session/query level can substantially push out the cost–quality Pareto frontier, but it has been hard to put into practice on existing serving stacks (vLLM, SGLang) that are tied to a single-LLM assumption, due to step desynchronization, batch admission delay, and implementation complexity. Under the principle of “request-centric programming, model-centric execution,” TokenRouter introduces asynchronous subservers and a delayed batching scheduler, achieving 2.01–64.15× decoding throughput over existing implementations across five routing algorithms and several model pairs (source: §Abstract, §1).

Model Routing on Jaehun's Blog

TokenRouter: Efficient Serving System for Token-Level LLM Routing

TokenRouter: An Efficient Serving System for Token-Level LLM Routing