---
title: "When Aggregate Alignment Misleads: Auditing Policy Repair Without Per-State Expert Actions"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.03386v1"
permalink: "https://feed.astralor.com/items/itm_6432b6df245c8192801a89a0"
published: "2026-07-03T14:42:01.000Z"
score: 79
channel: "tech"
tags: ["AI Safety Auditing","Large Language Models","Policy Repair","Diagnostic Feedback","Hotel-Pricing Simulator"]
---

# When Aggregate Alignment Misleads: Auditing Policy Repair Without Per-State Expert Actions

## AI commentary

> This paper not only demonstrates the potential of LLMs in policy repair but also warns the research community against blindly trusting aggregate behavioral metrics. For AI safety engineers and auditing practitioners, it offers a more realistic evaluation framework: focus on closed-loop outcomes rather than statistical distances. A must-read for anyone working on AI agent alignment.

## At a glance

This paper studies how to audit policy repair by agentic AI systems when per-state expert action labels are unavailable. In a hotel-pricing simulator, an LLM editor receives only region-level diagnostic feedback (summaries of price distribution differences from a benchmark policy), without access to benchmark actions, source code, or reward numbers. Experiments show the LLM editor achieves RevPAR close to the benchmark (108.47 vs 108.75) and significantly reduces episode composition distance (from 1.153 to 0.609). However, a tree editor with stronger behavioral alignment yields lower revenue (98.91), indicating that policy repair evaluation should focus on whether diagnostic feedback translates into reliable closed-loop outcomes, not just behavioral distance.

## Key points

- Agentic policy editor relies only on region-level diagnostic feedback, without access to benchmark actions, source code, or rewards, yet repairs policy to near-benchmark performance.
- LLM editor achieves minimal RevPAR gap (-0.276) and significantly reduces episode composition distance, demonstrating repair ability beyond raw revenue lift.
- Tree editor with stronger behavioral alignment yields lower revenue, showing aggregate alignment can mislead; evaluation should be based on closed-loop outcome reliability.

## Full text

Agentic AI systems are increasingly used to edit, refine, and repair decision policies, but evaluating these edits is difficult when per-state expert action labels are unavailable. We study this problem in a hotel-pricing simulator where an agentic policy editor receives only region-level diagnostic feedback: summaries of how its price distribution differs from a benchmark policy across time, inventory, and market regions. The editor cannot observe benchmark actions, benchmark source code, reward numbers, or held-out outcomes, and may only propose constrained edits to a target-action table. On 5,000 held-out episodes, a multi-restart LLM editor reaches RevPAR 108.47 (95% CI 107.61 - 109.34), close to the benchmark policy's 108.75 (107.81 - 109.68), with paired gap (LLM minus benchmark) -0.276 and 95% CI [-0.692, 0.146]. A cheap diagnostic projection already recovers much of the revenue (107.90), so the LLM editor's distinctive gain is not raw revenue lift alone: it also reduces episode composition distance from 1.153 to 0.609. This is the strongest non-benchmark repair result. This profile is not explained by restart search alone: non-semantic proposers with up to 2,500 evaluations fall 8.77 - 14.57 RevPAR points short. Nor is it explained by plausible prompt format: a shuffled-diagnostic control breaks region-error correspondence and falls to RevPAR 94.30. The match is genuine but partial. A tree editor achieves stronger pooled alignment, 0.214 versus 0.266, and stronger reference-state D1, 0.328 versus 1.197, yet revenue falls to 98.91. These results show that agentic policy repair should be evaluated by whether diagnostic feedback becomes reliable closed-loop outcome, not by a single behavioral distance.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_6432b6df245c8192801a89a0