Large Flow Language Models
The sequential bottleneck of sampling one token at a time Large language models can compute an entire sentence in parallel during training, but when generating text they…
Knowledge Distillation
3 pages
On-Policy or Off-Policy Learning? A Systematic Study of Distillation Dynamics
On-Policy Is Not a Panacea: The Real Protagonists of Rollout Policy, KL Direction, and Learning Rate in Distillation Dynamics TL;DR — When we separate on-policy (OnPD)…
X-AuT: Progressive Audio-Encoder Compression for Speech LLMs with Cross-Scale Distillation
X-AuT: Pruning Audio Encoders in Speech LLMs with Behavioral Probes and Cross-Scale Distillation TL;DR X-AuT is a framework that progressively prunes the audio encoder…