Same Request, Different Answer: Quantization Amplifies Cache-Induced Divergence in LLM Serving
Paper Same Request, Different Answer: Prefix Caching Makes Serving Non-Reproducible, and Quantization Amplifies It TL;DR — Prefix caching is …
16 min
Prefix Caching
Software Engineering