SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers
SMELT: Scaling Laws for Compute-Matched MoE Loop Transformers TL;DR Loop transformers scale depth by executing a layer block repeatedly, but most prior work fixed only…
Posts
All posts on technology, daily life, and thoughts.
119 pages
15 /
20
Why Gated DeltaNet Survives 4-Bit Quantization: NVFP4 W4A4 for the Recurrent Half of a Hybrid 27B LLM
The Recurrent Half Is the Easy-to-Quantize Half: Why Gated DeltaNet Survives at 4 Bits TL;DR — The 48 recurrent Gated DeltaNet (GDN) layers of a hybrid 27B LLM…
Random Attention: Rethinking KV Cache Eviction for Efficient Reasoning
The Scores Were a Waste: The Paradox Where 'Random Sampling' Ties the Strongest in KV Cache Eviction — An In-Depth Review of Random Attention TL;DR — Research on…
![[Paper Review] Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models](https://paper-assets.alphaxiv.org/figures/2601.07372v1/img-0.jpeg)
[Paper Review] Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models
Engram: The Second Sparsity Axis After Conditional Computation (MoE), Conditional Memory One-Line Summary (TL;DR) Engram adds a "knowledge lookup primitive" to the…
![[Paper Review] Continuous Autoregressive Language Models](https://discuss.pytorch.kr/uploads/default/original/2X/d/d1107e24375fae20a5f3a4733826880ce5d817db.png)
[Paper Review] Continuous Autoregressive Language Models
CALM: Bypassing the Token-by-Token Bottleneck with "Continuous Vector-by-Vector" Likelihood-Free Language Modeling CALM (Continuous Autoregressive Language Models)…
![[Paper Review] Memory Retrieval and Consolidation in Large Language Models through Function Tokens](https://moonlight-paper-snapshot.s3.ap-northeast-2.amazonaws.com/arxiv/memory-retrieval-and-consolidation-in-large-language-models-through-function-tokens-4.png)
[Paper Review] Memory Retrieval and Consolidation in Large Language Models through Function Tokens
Function Token Hypothesis: Why Punctuation and Newlines Gate LLM "Memory Retrieval" One-line Summary (TL;DR) This paper defines LLM memory as "reactivation of features…