All posts on technology, daily life, and thoughts.
![[Paper Review] Radial Attention: O(n log n) Sparse Attention with Energy Decay for Long Video Generation](https://moonlight-paper-snapshot.s3.ap-northeast-2.amazonaws.com/arxiv/radial-attention-onlog-n-sparse-attention-with-energy-decay-for-long-video-generation-2.png)
Radial Attention: O(n log n) Sparse Attention with Energy Decay — Generating "Long Videos" Cheaply One-Line Summary (TL;DR) Based on the observation that post-softmax…
![[Paper Review] Attention Sinks and Compression Valleys in LLMs are Two Sides of the Same Coin](https://cdn-uploads.huggingface.co/production/uploads/6317233cc92fd6fee317e030/yNP71PjobVvLDgJ0R0qV2.png)
One Narrative Forged by Massive Activations: Bridging Attention Sink and Compression Valley TL;DR Massive activations in the residual stream (especially BOS) form a…
![[Paper Review] Hardware-Efficient Attention for Fast Decoding](https://moonlight-paper-snapshot.s3.ap-northeast-2.amazonaws.com/arxiv/hardware-efficient-attention-for-fast-decoding-1.png)
GTA & GLA: Hardware-Efficient Attention That Breaks the 'Memory-Dominated' Decode TL;DR GTA (key–value tying) and GLA (latent head sharding) raise FLOPs/byte and shrink…
![[Paper Review] Pretraining Large Language Models with NVFP4](https://developer-blogs.nvidia.com/wp-content/uploads/2025/08/Optimizing-LLM-Training-png.webp)
NVFP4 4-bit Pretraining, Made Practical: 12B up to 10T Tokens, Effectively On Par with FP8 TL;DR Pretraining a 12B hybrid Mamba-Transformer on 10T tokens at NVFP4 (4-bit)…
"Let's put an OS on the GPU": A proposal for a GPU multitasking OS layer for the LLM era One-line summary (TL;DR) In LLM serving, GPUs often stay at ≤10% utilization…
DroidSpeak: Reducing Prefill Latency by 1.7–3.1× through Cross-LLM Prefix-KV Reuse TL;DR When multiple LLMs share the same architecture but different weights, DroidSpeak…