---
title: "PromptOps: Iterative Prompt Optimization for LLM-as-a-Judge Systems"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2611.07123v1"
permalink: "https://feed.astralor.com/items/itm_7e661dcdb7f3953b299bdddc"
published: "2026-11-04T20:38:52.000Z"
score: 59
channel: "tech"
tags: ["LLM","PromptOps","Iterative Prompt Optimization","LLM-as-a-Judge Systems Large","LLMs","Large","Existing"]
---

# PromptOps: Iterative Prompt Optimization for LLM-as-a-Judge Systems

## At a glance

Large language models (LLMs) are increasingly used as automated judges for evaluating the behavior of chatbots and other AI systems. In deployment, however, their agreement with the intended human evaluation policy may …

## Key points

- Large language models (LLMs) are increasingly used as automated judges for evaluating the behavior of chatbots and other AI systems.
- In deployment, however, their agreement with the intended human evaluation policy may degrade as user interactions and score distributions change.
- Existing automatic prompt optimization methods are primarily studied as one-shot procedures on static development sets and therefore provide limited guidance for maintaining LLM judges over time.

## Full text

Large language models (LLMs) are increasingly used as automated judges for evaluating the behavior of chatbots and other AI systems. In deployment, however, their agreement with the intended human evaluation policy may degrade as user interactions and score distributions change. Existing automatic prompt optimization methods are primarily studied as one-shot procedures on static development sets and therefore provide limited guidance for maintaining LLM judges over time. We frame this problem as judge drift and introduce PromptOps, a closedloop framework for monitoring and iteratively recalibrating LLM-as-a-Judge (LLMaaJ) systems. PromptOps separates four operational decisions: when to optimize, which optimization procedure to apply, which feedback records to use, and how to update the judge prompt. We instantiate the framework with five prompt optimization methods: GEPA, SIMBA, Rubric Synthesis, and exemplar selection using either simulated annealing or a Tree-structured Parzen Estimator. Our experiments demonstrate that iterative prompt optimization can recalibrate LLM judges and recover performance after controlled shifts in the referencescore distribution. However, effective maintenance is inherently contextdependent: neither optimization algorithm choice nor update timing generalizes universally across settings. These findings establish prompt optimization for deployed LLM judges as an inherently iterative maintenance problem and demonstrate that PromptOps provides a practical abstraction for studying and implementing this process.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_7e661dcdb7f3953b299bdddc