Efficient Inference

11 pages

The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction

Paper Waking the MoE on SSD: Serving a 35B MoE at 20 tok/s on a 24 GB Desktop with Edge0One-line summary (TL;DR)Even at 4-bit, a 35B-class …

September 18, 2026

18 min

Mixture of Experts Quantization Efficient Inference LLM Inference Hardware Architecture Open-Source LLM

How Lossless Is Lossless Speculative Decoding? The Role of Numerical Precision in Orthrus

Paper Is “Lossless” Speculative Decoding Really Lossless? — The Numerical-Precision Trap Exposed by Reproducing OrthrusTL;DR — …

September 16, 2026

13 min

Speculative Decoding Numerical Precision Quantization LLM Inference Efficient Inference Benchmark Evaluation

ZGCM-1: A Fully Open and Extremely Efficient Foundation Model for Math and Agentic Search

Paper ZGCM-1: How a 7B Dense Model Trades Blows with 235B Frontiers — A Fully Open, Ultra-Efficient Foundation Model Design ReportTL;DR — …

September 16, 2026

15 min

KV Cache Long Context Efficient Inference Efficient Training Chain-of-Thought Open-Source LLM

SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking

Paper SAS: A Simple Attention Sparsification That Learns Context Ranking Directly Without DistillationTL;DR — In post-training attention …

September 15, 2026

20 min

KV Cache Long Context Efficient Inference Attention Optimization Large Language Models Transformer

CoVeR: Coverage-Based Token Pruning for Multi-View 3D Reasoning in VLMs

Paper CoVeR: Coverage-Based Pruning That Cuts Visual Tokens of Multi-View 3D Reasoning by up to 92% Using Geometry AloneOne-Line Summary …

September 14, 2026

4 min

Efficient Inference KV Cache Vision-Language Model LLM Inference Model Compression Computer Vision

SQS: Bayesian DNN Compression through Sparse Quantized Sub-distributions

Paper SQS: Fusing Pruning and Quantization into One Bayesian Learning — Where Spike-and-Slab Meets GMMTL;DR — Doing pruning and low-bit …

September 13, 2026

13 min

Machine Learning Large Language Models Transformer Efficient Inference Model Compression Quantization

Kalman Delta Networks: Uncertainty-aware Associative Memory

Linear Attention Rewritten with a Kalman Filter: Kalman Delta NetworksTL;DRThe fixed-size recurrent memory update of linear attention is …

September 12, 2026

16 min

KV Cache Long Context Efficient Inference SSM Attention Optimization Kalman Filter

X-AuT: Progressive Audio-Encoder Compression for Speech LLMs with Cross-Scale Distillation

Paper X-AuT: Pruning Audio Encoders in Speech LLMs with Behavioral Probes and Cross-Scale DistillationTL;DRX-AuT is a framework that …

September 12, 2026

11 min

Efficient Inference Large Language Models Transformer Machine Learning Knowledge Distillation Model Compression

Deadline-Aware Adaptive Prefill Chunking for Efficient Large Language Model Serving

Paper Deadline-Filling Prefill Chunks: SLOWeave — An Adaptive Chunking Scheduler for LLM Serving TL;DR — Instead of a fixed prefill chunk …

September 11, 2026

15 min

LLM Inference KV Cache Long Context Inference Acceleration Efficient Inference Transformer

Why Is Video Still So Expensive? A Survey of Inference-Efficiency Mechanisms in Video and Audiovisual LLMs

Paper Why Are Video LLMs Still So Expensive? — A Four-Stage Guide to Inference Efficiency MechanismsTL;DR — This paper classifies 125 …

September 11, 2026

17 min

Long Context KV Cache Multimodal Learning Vision-Language Model Efficient Inference Large Language Models

[Paper Review] Llama-Nemotron: Efficient Reasoning Models

Paper Link Hydragen: The Secret Weapon for Decoding Large Batches with Shared Prefixes up to 32× FasterTL;DRBy decomposing the prefix and …

July 29, 2025

25 min

Prefix Caching Efficient Inference Attention Optimization FlashAttention vLLM