---
title: "Multi-Turn Semantic Caching for Low-Cost LLM Serving"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2611.01077v1"
permalink: "https://feed.astralor.com/items/itm_56c1ed9017808fc89aaa0f1d"
published: "2026-10-31T16:16:16.000Z"
score: 55
channel: "tech"
tags: ["LLM","推理效率","Multi-Turn Semantic Caching","Low-Cost LLM Serving","Semantic","LLMs","AUC","CUCB-SC-MT"]
---

# Multi-Turn Semantic Caching for Low-Cost LLM Serving

## At a glance

Semantic caching lowers the cost of serving large language models (LLMs) by reusing stored responses for similar requests. In multi-turn conversations, whether a cached response is acceptable depends on both conversatio…

## Key points

- Semantic caching lowers the cost of serving large language models (LLMs) by reusing stored responses for similar requests.
- In multi-turn conversations, whether a cached response is acceptable depends on both conversations' histories, which embedding similarity of requests and histories predicts poorly (AUC 0.62--0.74 on five workloads).
- We therefore learn the \emph{reuse risk}, the probability that a cached response is unacceptable, from judge feedback, and choose a budgeted set of conversation states to cache by weighing the generation cost a reuse av…

## Full text

Semantic caching lowers the cost of serving large language models (LLMs) by reusing stored responses for similar requests. In multi-turn conversations, whether a cached response is acceptable depends on both conversations' histories, which embedding similarity of requests and histories predicts poorly (AUC 0.62--0.74 on five workloads). We therefore learn the \emph{reuse risk}, the probability that a cached response is unacceptable, from judge feedback, and choose a budgeted set of conversation states to cache by weighing the generation cost a reuse avoids against a price for unacceptable reuse. We propose CUCB-SC-MT, which learns a cache from logged conversations with a suboptimality guarantee, and CLCB-LS-MT, which learns online and has $\tilde O(\sqrt N)$ regret over $N$ requests when conversation states recur. On math-tutoring (SocraTeach) and movie-ticketing (Taskmaster-3) conversations, both achieve lower loss than prior semantic caching approaches: ContextCache, multi-turn variants of ElastiCache and vCache, and single-turn semantic caching at every cache budget. With 1,000 cache entries, CLCB-LS-MT avoids 35--41\% of the generation cost through acceptable reuses, against at most 22--25\% for the baselines, and lowers estimated serving latency by 19--35\%.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_56c1ed9017808fc89aaa0f1d