---
title: "聚合对齐误导：无逐状态专家动作下的策略修复审计"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.03386v1"
permalink: "https://feed.astralor.com/items/itm_6432b6df245c8192801a89a0"
published: "2026-07-03T14:42:01.000Z"
score: 79
channel: "tech"
tags: ["AI安全审计","大型语言模型","策略修复","诊断反馈","酒店定价模拟器"]
---

# 聚合对齐误导：无逐状态专家动作下的策略修复审计

## AI 点评

> 这篇论文不仅展示了LLM在策略修复中的潜力，更警示了研究社区不要盲目信任聚合行为指标。对于AI安全工程师和审计从业者来说，它提供了一个更贴近实际部署的评估框架：关注闭环结果而非统计距离。值得所有做AI agent对齐的人都读一读。

## 速览

本文研究在缺乏逐状态专家动作标签的情况下，如何审计智能体AI系统的策略修复行为。在一个酒店定价模拟器中，LLM编辑器仅接收区域级诊断反馈（价格分布与基准策略差异的汇总），无法访问基准动作、源代码或奖励值。实验表明，LLM编辑器可达到接近基准的RevPAR（108.47 vs 108.75），且显著降低了片段组成距离（从1.153降至0.609）。但树编辑器虽实现了更强的行为对齐，收入却更低（98.91），说明策略修复的评估应关注诊断反馈能否转化为可靠的闭环结果，而非单纯的行为距离。

## 关键点

- 智能体策略编辑器仅依赖区域级诊断反馈，无法访问基准动作、源码或奖励，却能修复策略至接近基准性能。
- LLM编辑器在RevPAR上与基准差距极小（-0.276），且显著降低了片段组成距离，体现了超越单纯收入提升的修复能力。
- 树编辑器行为对齐更强但收入更低，表明聚合对齐指标可能误导，评估应基于闭环结果可靠性。

## 全文

Agentic AI systems are increasingly used to edit, refine, and repair decision policies, but evaluating these edits is difficult when per-state expert action labels are unavailable. We study this problem in a hotel-pricing simulator where an agentic policy editor receives only region-level diagnostic feedback: summaries of how its price distribution differs from a benchmark policy across time, inventory, and market regions. The editor cannot observe benchmark actions, benchmark source code, reward numbers, or held-out outcomes, and may only propose constrained edits to a target-action table. On 5,000 held-out episodes, a multi-restart LLM editor reaches RevPAR 108.47 (95% CI 107.61 - 109.34), close to the benchmark policy's 108.75 (107.81 - 109.68), with paired gap (LLM minus benchmark) -0.276 and 95% CI [-0.692, 0.146]. A cheap diagnostic projection already recovers much of the revenue (107.90), so the LLM editor's distinctive gain is not raw revenue lift alone: it also reduces episode composition distance from 1.153 to 0.609. This is the strongest non-benchmark repair result. This profile is not explained by restart search alone: non-semantic proposers with up to 2,500 evaluations fall 8.77 - 14.57 RevPAR points short. Nor is it explained by plausible prompt format: a shuffled-diagnostic control breaks region-error correspondence and falls to RevPAR 94.30. The match is genuine but partial. A tree editor achieves stronger pooled alignment, 0.214 versus 0.266, and stronger reference-state D1, 0.328 versus 1.197, yet revenue falls to 98.91. These results show that agentic policy repair should be evaluated by whether diagnostic feedback becomes reliable closed-loop outcome, not by a single behavioral distance.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_6432b6df245c8192801a89a0