---
title: "Which Eviction Policy Should an LLM Cache Use? A Systematic Study Across Workloads, Capacities, and Encoders"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.20280v1"
permalink: "https://feed.astralor.com/items/itm_33c9b48905ee7183414c224a"
published: "2026-08-20T17:14:19.000Z"
score: 77
channel: "ai"
tags: ["arXiv","CLEVER","LFU","LLM Semantic Cache","MiniLM all-MiniLM-L6-v2","Cache Eviction Policy"]
---

# Which Eviction Policy Should an LLM Cache Use? A Systematic Study Across Workloads, Capacities, and Encoders

## AI commentary

> The biggest takeaway is counterintuitive but practical: don't over-engineer eviction policies; first check whether your cache hits are truly usable. For engineers working on LLM inference optimization and infrastructure, this paper reminds you that quality-adjusted hit rate is the real KPI, and it makes a strong case for LFU as a default.

## At a glance

This paper presents the first systematic comparison of eviction policies for LLM semantic caches under a unified protocol. Using CLEVER, the authors evaluate FIFO, LRU, LFU, ARC, GDSF, a streaming SISO variant, and a semantic-redundancy policy across three query corpora, three cache capacities, and two encoders. No policy beats LFU by more than 0.041 percentage points in any of the eighteen settings, but at tight capacity FIFO and streaming SISO trail LFU by up to 8.67 and 8.55 points. A conditional packing result explains that geometry-aware eviction rules receive little new redundancy signal under exact lookup and insert-on-miss. The audit further shows that only 2.1%-3.9% of sampled hits are answer-substitutable at MiniLM's median nearest-neighbor threshold, dropping quality-adjusted hit rates from 51%-60% to 1.1%-2.2%. The authors conclude that LFU is the strongest simple default and that deployments should first establish answer validity before testing sub-point policy differences with exact search.

## Key points

- No eviction policy outperforms LFU by more than 0.041 percentage points across all eighteen settings, making LFU the strongest simple default.
- At tight capacity, FIFO and streaming SISO trail LFU by up to 8.67 and 8.55 points, showing replacement is not irrelevant.
- Under exact lookup and insert-on-miss, a newly inserted entry cannot have a resident neighbor within the hit radius, limiting the redundancy signal for geometry-aware eviction.
- At MiniLM's median nearest-neighbor threshold, only 2.1%-3.9% of sampled hits are answer-substitutable, dropping quality-adjusted hit rates from 51%-60% to 1.1%-2.2%.

## Full text

Semantic caches reuse an LLM response when the incoming query embedding lies near a cached query, but proposed eviction policies have rarely been compared under one protocol. Using CLEVER, we evaluate FIFO, LRU, LFU, ARC, GDSF, a single-pass streaming adaptation of SISO, and a semantic-redundancy policy across three ordered, deduplicated query corpora, three cache capacities, and two encoders. No evaluated policy improves on LFU by more than 0.041 percentage points in any of the eighteen settings. Replacement is not irrelevant: FIFO and streaming SISO trail LFU by as much as 8.67 and 8.55 points, respectively, at tight capacity. We explain the missing upside with a conditional packing result. Under exact lookup and insert-on-miss, a newly inserted entry cannot have a resident neighbor within the hit radius, so a geometry-aware eviction rule receives little new redundancy signal. A separate audit exposes a larger problem with the evaluated operating point. At MiniLM's median nearest-neighbor threshold, only 2.1-3.9% of sampled LMSYS and QQP hits are judged answer-substitutable, reducing raw hit rates of 51-60% to quality-adjusted rates of 1.1-2.2%. The cross-encoder study further shows that thresholds do not transfer between embedding models. LFU is the strongest simple default in this protocol; deployment decisions should first establish answer validity and then test sub-point policy differences with exact search.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_33c9b48905ee7183414c224a