Miles v0.1: Production-Level Post-Training

Paper

Miles v0.1: A “verified, clean, and scalable” full-stack system for frontier RL post-training

TL;DR: Miles is a full-stack RL post-training framework that resolves the train-rollout mismatch at the system level — the mismatch that arises when rollout generation and training run on different engines, kernels, and precisions. It unifies SGLang-based rollout, a Megatron-LM/FSDP trainer, and three weight-synchronization transports (broadcast/P2P/disk-delta), while also guaranteeing token exactness (TITO) and expert-routing replay (R3). As an end-to-end case study, it trains GLM-5.2 744B-A40B on 64 GB300 GPUs with fully asynchronous agentic RL, achieving a median step time of 263 seconds (source: §9.2, Fig. 5).

Core idea

Every design decision in Miles reduces to a single principle: “components must be verified, clean, and customizable” (source: §1). This principle materializes in the three stages of the RL training loop (source: §1.1, Fig. 1).

1. Rollout: The SGLang engine generates trajectories. In agentic RL, it uses tools across multiple turns and interacts with external environments.
2. Training: The trainer (Megatron-LM or FSDP) consumes completed trajectory groups, computes the RL loss, and updates the policy.
3. Weight update: The new weights are synchronized to the rollout engine.

The key insight is that these three stages do not have to run in lockstep. If run synchronously in alternation, the trainer waits for the slowest trajectory in the batch, and the engine waits for the optimizer to finish. Miles provides a fully asynchronous mode that runs generation and training concurrently (source: §2.2). And what makes this “concurrency” possible is numerical fidelity — if the trainer cannot exactly reproduce the tokens the rollout sampled, the gradients are silently contaminated (source: §2).

Background: the problem they set out to solve

Post-training is the stage that turns a pretrained language model into a useful one. But frontier-scale post-training is no longer “generate-then-update on short completions” (source: §1). Three structural challenges compound.

1. Rollouts have expanded to multi-turn, tool-use, external-environment behavior. A single trajectory has become the agent’s entire episode (messages + tool calls + environment responses) (source: §2).
2. Models have grown into trillion-scale MoEs. Trying to run latency-sensitive rollout and throughput-centric training at the same time inevitably creates bubbles and idle time (source: §1).
3. A growing numerical gap between the rollout engine and the trainer invalidates the objective itself. SGLang and Megatron use different kernels, precisions, and batches, so even with the same weights and the same trajectory they assign different probabilities. When this importance ratio drifts away from 1, an update meant to improve the policy instead breaks it (source: §2.4, §3.4).

In particular, expert-routing mismatch in MoEs is fatal. If the rollout picks experts $\{2, 7\}$ for a token but the trainer, re-routing with different kernels and precision, picks $\{2, 8\}$, then the expert that actually contributed to the sample receives no update while the expert that did not contribute does. Accumulated over layers × sequences × tokens × thousands of steps, this can, as Ma et al. reported, make MoE RL sharply unstable and lead to catastrophic collapse (source: §2.5, [19]).

The new approach: Miles — a verifiable full-stack RL loop

Miles’s originality lies not in any single algorithm but in a system design that runs generation, training, and synchronization concurrently while preserving numerical fidelity. Let’s break down the core mechanisms into three parts.

1) TITO: a session server that preserves token exactness

In conventional multi-turn pipelines, model output passes through message parsing → tool execution → chat-template rendering, during which tokenization changes or prior reasoning gets truncated. This makes the “sampled tokens” and the “tokens the trainer sees” diverge, so the trainer updates on trajectories that never actually occurred (source: §2.4).

The Token-In-Token-Out (TITO) session server closes this gap by giving the server, not the harness, control over tokenization. On the first turn it renders the template into token IDs, and after each successful turn it checkpoints the prompt ID along with the output token IDs, log probabilities, and even the routed experts. On subsequent turns it reuses the deepest applicable checkpoint and tokenizes only the newly added suffix (source: §2.4, Fig. 3). The server extends transactions with linear and branching rules, accommodating even harnesses that, like Claude Code, compress and branch their own context (source: §2.4.1).

2) R3: MoE expert-routing replay

Token exactness alone cannot reproduce an MoE’s log probabilities. Rollout Routing Replay (R3) treats each token’s expert assignment as part of the rollout data, letting the trainer replay the assignments from rollout time rather than recomputing the routing (source: §2.5). When --use-rollout-routing-replay is enabled, SGLang returns the routed experts alongside the tokens, and in the forward pass each token passes through exactly the same experts as during rollout. Since TITO records the routing experts as well, this replay covers the entire multi-turn episode (source: §2.4, §2.5).

3) Three weight transports

At frontier scale, weight synchronization itself becomes the bottleneck. A full NCCL broadcast of Kimi K2 1T-A32B takes nearly a minute (58.30s) (source: §4, Tab. 8). Miles provides three transports depending on the connection topology.

P2P’s effect is dramatic: on Kimi K2 1T-A32B 53.28s → 7.23s (-86.4%), and on GLM-5 744B-A40B 58.30s → 8.48s (-85.5%) (source: §4.2, Tab. 8). Within a single node, however, P2P can be up to roughly 70% slower than broadcast, so broadcast remains the default (source: §4.2).

On top of this, Miles adds low-precision training (FP8 blockwise, MXFP8, NVFP4), memory offload (eviction of stalled actors + optimizer-state streaming), and extension recipes for LoRA RL, On-policy Distillation, True-on-policy alignment, and Diffusion (source: §3.1, §3.2, §5, §6).

How it works: a concrete example

Let’s compress expert-routing mismatch into a tiny example (source: §2.5). Suppose the MoE router picks the top $k=2$ experts for some token.

• The rollout (SGLang, FP8) selects experts $\{2, 7\}$.
• The trainer (Megatron, BF16), re-routing with the same weights, selects $\{2, 8\}$ because of numerical differences.
• Result: expert 7 receives no update despite contributing to the sample, and expert 8 receives an update despite not contributing.

R3 solves this not by “recomputing” but by “replaying the recorded assignment,” so the trainer uses experts $\{2, 7\}$ as-is (source: §2.5).

The cost analysis shows that replay is cheap but transport is expensive. For token count $N$, layer count $L$, and top-k $k$, the routing tensor grows as

of 32-bit integers. With 32K tokens × 60 layers × $k=8$, that is roughly 60MB per trajectory (source: §2.5). R3 is therefore not a default but a recipe-specific option, and the GLM-5.2 reference run turns it off (source: §2.5).

Optimizer-state streaming is intuitive too. On large runs, the optimizer state consumes the most HBM — combining the FP32 master weights and the two Adam moments gives 12 bytes per parameter (source: §3.2.2). Miles splits parameters into buckets, loading a bucket’s file into HBM only when that bucket is updated and evicting it afterward. On Qwen3-30B-A3B, actor offload dropped from 24s → 5.2s and reload from 8.9s → 1.3s (source: §3.2.2).

Where does the On-policy Distillation signal come from? For a token $x_t$ that the student sampled itself, the difference between the student’s and teacher’s log probabilities,

is a one-sample estimate of the reverse KL divergence at that position. A positive value means the student preferred that token too much; a negative value means the teacher preferred it more (source: §5.2, Fig. 4). Miles folds this signal into the advantage rather than the loss, making it combinable with GRPO and PPO (source: §5.2).

Performance validation: key results

End-to-end case study: GLM-5.2 744B-A40B on 64×GB300

The most notable result is the end-to-end run in §9. GLM-5.2 744B-A40B (744B total / 40B active parameters) was trained with fully asynchronous RL on a terminal-use coding task (terminal-bench-2) across 64 GB300 GPUs (source: §9, Tab. 9).

Three observations follow (source: §9.2, Fig. 5).

1. The 744B model is trained on 32 GPUs. The parallel layout (TP2/PP4/CP4/EP8) uses only half of the 64 and leaves the other half for generation. At this point the per-rank optimizer state is roughly 279GB, exceeding GPU memory, so disk streaming is not an “optimization” but a necessity (source: §9.1).
2. Median step time of 263s (based on the first 30 measured steps; the 1,042s warm-up of step 0 is excluded) (source: §9.2, Fig. 5a).
3. Generation and training overlap between weight updates. Sample-granularity refill fills a slot the moment a trajectory finishes, so roughly 90–100 requests are generated concurrently across the fleet. Thanks to affinity routing, the prefix-cache hit rate stays at 96% (source: §9.2, §2.1).

Numerical soundness was also confirmed. The log-probability divergence between the rollout engine and the trainer ends near its starting value, with a 100-step average of 0.0369, and truncated importance sampling (TIS) corrects for it (source: §9.2, Fig. 5b). Raw task reward rose from 0.438 → 0.556 on a 9-step moving average, but because this is a single run, it is reported only as an “observation,” not a “measured improvement” (source: §9.2, Fig. 5c).

P2P weight transport: the wider the fleet, the better

The gain scales with fleet width, not model size (source: §4.2, Tab. 8).

On-policy Distillation: 56% shorter response length

In the Qwen3.5-35B-A3B documentation run, a teacher (the same model improved with 5 steps of RL on a verifiable reward) supplied the signal while task reward was kept at zero, and training ran for 5 steps. Response length dropped from 14,070 → 6,132 tokens, and accuracy moved 84.0% → 85.2% — but within the evaluation standard error (roughly 1.6 points), so “a 56% reduction in length with no reliable change in accuracy” is the accurate conclusion (source: §5.2).

Miles-Diffusion: streaming rewards

In the LTX-2.3 recipe, micro-group streaming rewards cut rollout time from 157.4s → 87.6s and total step time from 321.9s → 252.1s (source: §6).

Our take: strengths, limitations, and why this work matters

The strength is the honesty of a systems paper. First, it specifies what “support” means in terms of evidence level, distinguishing how thoroughly each model, hardware, and recipe has been verified (source: §6, §7.1). Second, it actually passes the harshest bar — Day-0 model support — supporting six models (Kimi K3, DeepSeek-V4, GLM-5.2, Qwen3.8, Inkling, Nemotron 3 Ultra) on the day their weights were released (source: §7.1). Third, it elevates numerical fidelity to a top-priority goal. The fact that R3, TITO, and true-on-policy alignment all focus on preventing “silent gradient contamination” shows that, in RL scaling, the system is a hidden variable in performance (source: §2.4, §2.5, §5.3).

The limitations are also clear. First, coverage is uneven: true-on-policy alignment covers only the dense Qwen3 0.6B/4B, P2P transport only some model families, and the vision-language session server is not yet supported (source: §5.3, §4.2, §2.4.3). Second, the low-precision formats are still in Beta, and the authors themselves acknowledge that MXFP8 and NVFP4 may behave differently across models (source: §3.1, Tab. 5). Third, quantitative validation is shallow: the single-run, single-task-distribution case study in §9 reports the reward gain only as an “observation,” and of the four goals — accuracy, efficiency, reliability, scalability — only efficiency (step time, transport time) is directly compared quantitatively (source: §9.2). Fourth, the weight is placed more on the system’s existence than on the system’s performance itself, so it differs in character from research papers that compete on benchmark accuracy.

Still, this work matters because it traces to the end “where RL post-training truly becomes the bottleneck in practice.” The numerical gap between rollout and training, MoE routing mismatch, the one-minute weight-synchronization bottleneck — these problems never show up in benchmark scores, yet they decide whether frontier-scale training is actually possible.

What’s next?: the road ahead

The current limitations the authors disclose are essentially a roadmap: vision-language session support, early-stage precision formats, and weight-transport paths limited to only some model families (source: §10). Beyond that, a few reasonable next steps:

• Quantify the validation: augment the “accuracy” goal with multi-run statistics of reward and divergence metrics (variance, seed means) to upgrade §9’s single-run observation into reproducible measurements.
• Optimize R3’s overhead: compress the routing tensor (roughly 60MB per trajectory), or explore joint-correction methods that overcome R3’s limited effect in asynchronous RL (source: §2.5).
• Extend true-on-policy coverage: move beyond the dense Qwen3 0.6B/4B to extend bit-exact guarantees to MoE and low-precision recipes (source: §5.3).
• Integrate disk-delta endpoints: actually implement the single-endpoint path for external rollout services (documented as “forthcoming”) to support hybrid deployments (source: §4.3).

In the end, Miles’s significance is not “a single new algorithm” but rather presenting a reference implementation of a verifiable, scalable post-training infrastructure. By treating numerical fidelity and system efficiency with equal weight, this design marks a turning point where RL scaling moves from a “score race” to an “engineering race.”

Tables from the paper

Tables converted mechanically from the arXiv e-print LaTeX source. The numbers are the paper’s own and did not pass through a model.

Table 1. The three reasons the buffer drops a finished group. The first two are properties of the group, so they are checked as soon as it arrives; staleness depends on how long the group waited, so it is checked only when the trainer collects it. The run supplies the filter and the staleness limit and chooses whether dropped prompts are retried or discarded, except that filter-rejected groups are always discarded because they carry no gradient signal.

Table 2. Buffer metrics reported on every training step, under the rollout/fully_async/ prefix. Staleness appears twice: once for the groups a step drew, and once for the groups still waiting.

Table 3. The three evaluation modes under fully asynchronous rollout. Shared engines measure whichever weights the fleet most recently received, and new generation stops while they do so, although in-flight requests finish. The snapshot-based modes measure the exact weights in the snapshot they receive; an external backend sees only a checkpoint directory, so it can be any service, with or without SGLang.

Table 4. Three nested rollout plug-in layers. The agent function is innermost, and each column to its right wraps the one before it; a ✓ means the external framework assumes that responsibility, and $\circ$ means Miles retains it. Group rewards are scores that need the whole group at once, such as ranking the trajectories against one another, as distinct from GRPO’s group-relative centering, which Miles always applies itself.

Table 5. Low-precision formats with an end-to-end Miles recipe, against the BF16 baseline. NVFP4 nests an E4M3 scale per block inside one FP32 scale per tensor. FP8 blockwise runs on NVIDIA Hopper and Blackwell and on AMD MI350X and MI355X; MXFP8 and NVFP4 need Blackwell; A100 has no FP8 arithmetic and runs BF16 only.

Table 6. The two training backends. Megatron-LM exposes model-parallel axes, while the current FSDP backend uses data-parallel sharding; a $\times$ marks what that backend does not yet implement, not a limit of FSDP itself.

Table 7. The three weight-synchronization transports. All deliver the same converted weights but differ in the connectivity they assume and in how transfer volume scales with the fleet. When training and rollout are colocated on the same GPUs, the handoff is local and no transport is involved.

Table 8. Time per weight update, P2P against NCCL broadcast, on H100 clusters with a 1 GB transfer bucket, averaged over steady-state steps and timed from the end of the generation pause to the return of the update call. Node counts are per side, with trainer and rollout fleets of equal size. The Kimi K2 times include about 884 ms of on-GPU requantization that its checkpoint requires after every transfer. The advantage grows with fleet width rather than model size, and appears already at two nodes per side.