Recent language models keep increasing the amount of information packed into a single vocabulary entry. Per-layer embeddings (PLE), first introduced in Gemma 3n, carried…
TRACE: Rollout-Aligned FP4 Quantization-Aware Training — MoE LLM RL at BF16 Quality, up to 5.4x Faster TL;DR In reinforcement learning (RL) for MoE LLMs, quantizing…
ActionPiece: Rethinking Action Tokenization Beyond Reconstruction Error, via Physical Ranking TL;DR — In autoregressive VLA (vision-language-action) models, the action…
Why Don't FP4 Tensor Cores Make Attention Faster? Blackwell FlashAttention-4, Solved with Direct-P and Quantized Backprop TL;DR — Blackwell's FP4 tensor cores handle…
The Recurrent Half Is the Easy-to-Quantize Half: Why Gated DeltaNet Survives at 4 Bits TL;DR — The 48 recurrent Gated DeltaNet (GDN) layers of a hybrid 27B LLM…