TAO: Tool-Aware Joint Scheduling for LLM-Based Agentic Serving
TAO: Tool-Aware Joint Scheduling for LLM-Based Agentic Serving Original: Shuhuai Li et al., "TAO: Tool-Aware Joint Scheduling for LLM-Based Agentic Serving"…
Prefix Caching
7 pages
Cache Engineering for Agents
The Cost and Accuracy Dilemma of Long Agent Trajectories LLM agents accumulate long trajectories by repeating reasoning and tool calls for hundreds of rounds. To make…
WavePP: High-Throughput Pipeline Parallel LLM Prefill under Prefix Reuse
WavePP — A Pipeline-Parallel Prefill Runtime That Burns In Prefix Reuse TL;DR Built on top of TensorRT-LLM, WavePP overlaps request admission (reuse consensus, cache…
Who Pays for the KV Cache? Attributing Shared AI Inference Spend Across Kubernetes and LLM Provider Bills
Who pays for the KV cache? — Kubernetes, gateway, and provider bills in one ledger TL;DR — The cost of a single AI feature is scattered across three disconnected ledgers:…
Same Request, Different Answer: Quantization Amplifies Cache-Induced Divergence in LLM Serving
Same Request, Different Answer: Prefix Caching Makes Serving Non-Reproducible, and Quantization Amplifies It TL;DR — Prefix caching is billed as a "transparent…
![[Paper Review] Marconi: Prefix Caching for the Era of Hybrid LLMs](https://pbs.twimg.com/media/GdyLXO9W4AADox0.jpg)
[Paper Review] Marconi: Prefix Caching for the Era of Hybrid LLMs
Marconi: Rethinking Prefix Caching for the Hybrid LLM Era TL;DR Marconi introduces a prefix-caching framework for hybrid LLM inference (Attention + SSM). It selectively…
![[Paper Review] Llama-Nemotron: Efficient Reasoning Models](https://cdn-thumbnails.huggingface.co/social-thumbnails/collections/nvidia/llama-nemotron-67d92346030a2691293f200b.png)
[Paper Review] Llama-Nemotron: Efficient Reasoning Models
Hydragen: The Secret Weapon for Decoding Large Batches with Shared Prefixes up to 32× Faster TL;DR By decomposing the prefix and suffix using softmax denominator…