MemT-1.0: Towards Memory-Native Evolution
MemT-1.0 stores conversation history in a fixed-size matrix state. We examine memory updates, information loss, and how the approach compares with full-context and…
Machine Learning
15 pages
On-Policy or Off-Policy Learning? A Systematic Study of Distillation Dynamics
On-Policy Is Not a Panacea: The Real Protagonists of Rollout Policy, KL Direction, and Learning Rate in Distillation Dynamics TL;DR — When we separate on-policy (OnPD)…
ActionPiece: Rethinking Action Tokenization for Autoregressive Vision-Language-Action Models
ActionPiece: Rethinking Action Tokenization Beyond Reconstruction Error, via Physical Ranking TL;DR — In autoregressive VLA (vision-language-action) models, the action…
Programmable World Model
Programmable World Model — Embedding "Executable State" into Generative Worlds One-Line Summary (TL;DR) Video world models are producing increasingly realistic…
Marigold V2: Revisiting Diffusion Transformers for Monocular Depth Estimation
Marigold V2: Reviving an Image-Editing Diffusion Transformer (DiT) as a Monocular Depth Estimator TL;DR This study fine-tunes an image-editing diffusion transformer…
SQS: Bayesian DNN Compression through Sparse Quantized Sub-distributions
SQS: Fusing Pruning and Quantization into One Bayesian Learning — Where Spike-and-Slab Meets GMM TL;DR — Doing pruning and low-bit quantization separately leaves…
X-AuT: Progressive Audio-Encoder Compression for Speech LLMs with Cross-Scale Distillation
X-AuT: Pruning Audio Encoders in Speech LLMs with Behavioral Probes and Cross-Scale Distillation TL;DR X-AuT is a framework that progressively prunes the audio encoder…
VestigeKV: The NoPE-MLA KV Cache Carries Its Own Eviction Signal in a Vestigial Branch
VestigeKV: The Degenerate RoPE Branch Carries Its Own Cache Eviction Signal TL;DR In NoPE-MLA models (Kimi Linear 48B), reading only 11% of each row — the 64-dim…
Unlocking Lossless Speedups in LLMs via Discrete Diffusion
Uno: AR and Diffusion in One Model — Lossless Parallel Generation for LLM Acceleration TL;DR — LLMs are slow because next-token prediction (NTP) can emit only one token…
BeaconKV: Key-Value Cache Compression Guided by Beacon Queries for Efficient Large Reasoning Model Inference
Catch the Moment a Model "Revisits Its Thoughts": How BeaconKV Compresses Reasoning KV Caches 5.8× with Beacon Queries TL;DR — The KV cache of a Large Reasoning Model…
Hardware-Aware FP4 FlashAttention-4
Why Don't FP4 Tensor Cores Make Attention Faster? Blackwell FlashAttention-4, Solved with Direct-P and Quantized Backprop TL;DR — Blackwell's FP4 tensor cores handle…
![[Paper Review] NVIDIA Nemotron 3: Efficient and Open Intelligence](https://developer-blogs.nvidia.com/wp-content/uploads/2025/12/image3-8-png.webp)
[Paper Review] NVIDIA Nemotron 3: Efficient and Open Intelligence
NVIDIA Nemotron 3: Pushing the "Accuracy/Throughput" Frontier with a Hybrid Mamba–Transformer MoE Nemotron 3 combines an MoE hybrid Mamba–Transformer, LatentMoE, MTP,…
![[Paper Review] Radial Attention: O(n log n) Sparse Attention with Energy Decay for Long Video Generation](https://moonlight-paper-snapshot.s3.ap-northeast-2.amazonaws.com/arxiv/radial-attention-onlog-n-sparse-attention-with-energy-decay-for-long-video-generation-2.png)
[Paper Review] Radial Attention: O(n log n) Sparse Attention with Energy Decay for Long Video Generation
Radial Attention: O(n log n) Sparse Attention with Energy Decay — Generating "Long Videos" Cheaply One-Line Summary (TL;DR) Based on the observation that post-softmax…
![[Paper Review] Pretraining Large Language Models with NVFP4](https://developer-blogs.nvidia.com/wp-content/uploads/2025/08/Optimizing-LLM-Training-png.webp)
[Paper Review] Pretraining Large Language Models with NVFP4
NVFP4 4-bit Pretraining, Made Practical: 12B up to 10T Tokens, Effectively On Par with FP8 TL;DR Pretraining a 12B hybrid Mamba-Transformer on 10T tokens at NVFP4 (4-bit)…
![[Paper Review] Inference-Time Hyper-Scaling with KV Cache Compression](https://cdn-thumbnails.huggingface.co/social-thumbnails/papers/2506.05345/gradient.png)
[Paper Review] Inference-Time Hyper-Scaling with KV Cache Compression
Dynamic Memory Sparsification (DMS): Making LLM Hyper-Scaling a Reality with 8× KV Cache Compression One-Line Summary (TL;DR) DMS, combining a lightweight re-fitting of…