Recent language models keep increasing the amount of information packed into a single vocabulary entry. Per-layer embeddings (PLE), first introduced in Gemma 3n, carried…
Introduction A game world built by a video generation model alone stores off-screen state nowhere. Every time a corridor you have already passed through is redrawn, it…
TRACE: Rollout-Aligned FP4 Quantization-Aware Training — MoE LLM RL at BF16 Quality, up to 5.4x Faster TL;DR In reinforcement learning (RL) for MoE LLMs, quantizing…
TaSQ: 'Tailored Design' of the Quantization Space for a 1-bit KV Cache TL;DR — TaSQ (Tailored Space Vector Quantization) redesigns the very target space to which vector…
The KV Cache Is the New Memory Wall: Three Regimes and Five Domains of Long-Context LLM Inference TL;DR — Long-context LLM decode inference is bound by memory bandwidth,…
VC-Attention is a training-free kernel that removes value-quantization and softmax bottlenecks in low-bit video-DiT attention, speeding it up 1.59× on B200.
Fathom: Accelerating 1M-Token Decoding by 1.67x with Query-Adaptive Depth Scanning TL;DR When agent sessions stretch to 1M tokens and dozens of sessions reside…
Waking the MoE on SSD: Serving a 35B MoE at 20 tok/s on a 24 GB Desktop with Edge0 One-line summary (TL;DR) Even at 4-bit, a 35B-class MoE occupies 19.5 GB and cannot…
Is "Lossless" Speculative Decoding Really Lossless? — The Numerical-Precision Trap Exposed by Reproducing Orthrus TL;DR — Orthrus attaches a diffusion view to a frozen AR…
SQS: Fusing Pruning and Quantization into One Bayesian Learning — Where Spike-and-Slab Meets GMM TL;DR — Doing pruning and low-bit quantization separately leaves…
Miles v0.1: A "verified, clean, and scalable" full-stack system for frontier RL post-training TL;DR: Miles is a full-stack RL post-training framework that resolves the…