TL;DR — Prefix caching is billed as a “transparent optimization,” but it is not transparent. With caching off, repeated runs of the same workload are bit-identical across all 10 configurations × 80 episodes = 800 runs (source: Tab. 2); with caching on, agent trajectories change in 36.2% of episodes at 16 bits and 75.0% under 4-bit quantization (source: §IV-D). Yet average accuracy does not move — this is not “degradation” but “instability,” and the two call for different responses (source: §IV-F, §V).

2609.04748v1 on Jaehun's Blog

Same Request, Different Answer: Quantization Amplifies Cache-Induced Divergence in LLM Serving

Same Request, Different Answer: Prefix Caching Makes Serving Non-Reproducible, and Quantization Amplifies It