All posts on technology, daily life, and thoughts.
Paper Language Models Control Their Own Attention: Declarative AttentionTL;DR — Existing sparse attention methods still paid $O(N)$ per …
Paper Same Request, Different Answer: Prefix Caching Makes Serving Non-Reproducible, and Quantization Amplifies It TL;DR — Prefix caching is …
Paper SGD-KV: Finding ‘heads that are good at summarizing’ cuts the 1M-token KV cache by up to 75% TL;DR — Attention heads do …
Paper SMELT: Scaling Laws for Compute-Matched MoE Loop TransformersTL;DRLoop transformers scale depth by executing a layer block repeatedly, …
Paper The Recurrent Half Is the Easy-to-Quantize Half: Why Gated DeltaNet Survives at 4 BitsTL;DR — The 48 recurrent Gated DeltaNet (GDN) …
Paper The Scores Were a Waste: The Paradox Where ‘Random Sampling’ Ties the Strongest in KV Cache Eviction — An In-Depth Review …