---
title: "基于动态评分卡的LLM评估器与策略共同演化"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.20083v1"
permalink: "https://feed.astralor.com/items/itm_43552c643a7e637b090dd972"
published: "2026-07-22T12:35:40.000Z"
score: 80
channel: "ai"
tags: ["DynamicRubric","后训练","大语言模型","微信搜索","评估器共同演化"]
---

# 基于动态评分卡的LLM评估器与策略共同演化

## AI 点评

> 这篇论文值得所有从事LLM对齐和后训练的从业者关注。它不仅从理论上指出了现有奖励模型监督失效的根本原因，还给出了一个轻量且可落地的解决方案。微信搜索的部署案例更有说服力——真正实现了从实验室到工业级应用的闭环。如果你正在为策略优化中奖励信号趋同而头疼，这篇方法很可能就是你要找的答案。

## 速览

本文提出DynamicRubric框架，针对大语言模型后训练中评估器反馈监督的瓶颈问题：随着策略提升，采样响应质量趋同，导致评估器分数差距缩小，削弱优化信号。理论分析表明，相对分数差距是策略更新方向的关键信号。DynamicRubric通过为每个候选集生成加权二元评分项并聚合为响应级分数，实现评估器与策略共同演化。在8B骨干模型上，该方法超越70B奖励模型和235B静态评分生成器。优化后的模型已在微信搜索AI问答场景全量上线，每日服务数千万请求，显著提升在线指标。

## 关键点

- 指出评估器反馈中相对分数差距是策略优化的核心信号，理论证明了其概率分配机制。
- 提出DynamicRubric框架：为每个响应集动态生成加权二元评分项，实现评估器与策略共同演化。
- 在8B规模模型上超越70B奖励模型和235B静态评分生成器，并在可验证推理和编码任务上取得提升。
- 成果已落地微信搜索AI问答场景，全量上线服务数千万日请求，验证了实用价值。

## 全文

Post-training with evaluator feedback on policy-induced samples serves as a major mechanism for improving large language models. As policies improve, these sampled responses become close in quality. These close candidates create a bottleneck for policy optimization: collapsed relative evaluator score gaps yield weak or misleading policy supervision. We theoretically characterize why these gaps matter through a probability allocation view, showing that the directional gain of shifting probability mass from one response to another is exactly the evaluator score gap between them. This identifies relative score gaps as the policy optimization signals that guide updates. Motivated by this view, we propose DynamicRubric, a response-set-conditioned evaluator--policy co-evolution framework that generates weighted binary rubric items for each candidate set and aggregates the resulting judgments into response-level scores. In our experiments with 8B backbones, DynamicRubric improves evaluator performance and provides stronger policy supervision than baselines using a 70B reward model or a 235B static rubric generator. DynamicRubric-optimized policies also show gains on verifiable reasoning and coding tasks. A DynamicRubric-optimized model is fully deployed in WeChat Search's AI answering scenario, where it serves all online traffic across tens of millions of requests per day and improves key online metrics. These results suggest a principle for evaluator-guided post-training: evaluators should evolve with the policies they supervise.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_43552c643a7e637b090dd972