LLM Inference

6 pages

The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction

Paper Waking the MoE on SSD: Serving a 35B MoE at 20 tok/s on a 24 GB Desktop with Edge0One-line summary (TL;DR)Even at 4-bit, a 35B-class …

September 18, 2026

Mixture of Experts Quantization Efficient Inference LLM Inference Hardware Architecture Open-Source LLM

How Lossless Is Lossless Speculative Decoding? The Role of Numerical Precision in Orthrus

Paper Is “Lossless” Speculative Decoding Really Lossless? — The Numerical-Precision Trap Exposed by Reproducing OrthrusTL;DR — …

September 16, 2026

Speculative Decoding Numerical Precision Quantization LLM Inference Efficient Inference Benchmark Evaluation

CoVeR: Coverage-Based Token Pruning for Multi-View 3D Reasoning in VLMs

Paper CoVeR: Coverage-Based Pruning That Cuts Visual Tokens of Multi-View 3D Reasoning by up to 92% Using Geometry AloneOne-Line Summary …

September 14, 2026

Efficient Inference KV Cache Vision-Language Model LLM Inference Model Compression Computer Vision

Deadline-Aware Adaptive Prefill Chunking for Efficient Large Language Model Serving

Paper Deadline-Filling Prefill Chunks: SLOWeave — An Adaptive Chunking Scheduler for LLM Serving TL;DR — Instead of a fixed prefill chunk …

September 11, 2026

LLM Inference KV Cache Long Context Inference Acceleration Efficient Inference Transformer

[Paper Review] Towards Efficient and Practical GPU Multitasking in the Era of LLM

Paper “Let’s put an OS on the GPU”: A proposal for a GPU multitasking OS layer for the LLM eraOne-line summary (TL;DR)In …

October 9, 2025

Kubernetes DRA NCCL Virtualization LLM Inference System Design Resource Isolation

[Paper Review] SGLang: Efficient Execution of Structured Language Model Programs

[Paper Review] SGLang: Efficient Execution of Structured Language Model Programs

Paper Link SGLang & RadixAttention: How Execution Optimization for “LM Programs” Achieved a 6.4x SpeedupTL;DRBy combining a …

October 3, 2025

KV Cache LLM Inference Distributed Inference Prompt Optimization Multimodal