Rethinking Sequence Compression with Test-time Regression
Introduction In models that handle long contexts, the compression operation that reduces multiple tokens to a single vector becomes a performance bottleneck. This paper…
Long-Context
6 pages
HLA-WM: Hybrid Linear Attention for Long-Horizon Video World Models
HLA-WM retrieves past chunks using camera geometry to restore scene memory. At 60 seconds, its history memory is about 12× smaller than Full KV and 6× larger than GDN,…
Mechanics of Long-Context Hybrid Models Part 1.1: From Hybrid Attention to Hybrid Position
Mechanics of Long-Context Hybrid Models: From Hybrid Attention to Hybrid Position TL;DR — These days open-source LLMs are rapidly shifting from models that "use full…
Block Sparse Attention with Log-Linear Complexity
PISA: Pyramid Top-$K$ Selection Lowers the Cost of Block-Sparse Attention to $O(N\log N)$ TL;DR — Block-sparse attention makes attention computation linear, but block…
ZGCM-1: A Fully Open and Extremely Efficient Foundation Model for Math and Agentic Search
ZGCM-1: How a 7B Dense Model Trades Blows with 235B Frontiers — A Fully Open, Ultra-Efficient Foundation Model Design Report TL;DR — With a single 7.39B dense model,…
Why Is Video Still So Expensive? A Survey of Inference-Efficiency Mechanisms in Video and Audiovisual LLMs
Why Are Video LLMs Still So Expensive? — A Four-Stage Guide to Inference Efficiency Mechanisms TL;DR — This paper classifies 125 VideoLLM inference efficiency studies…