---
title: "RoMeRL：通过降阶效用状态平衡自进化智能体记忆中的反馈覆盖与记忆-奖励陷阱"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.02508v1"
permalink: "https://feed.astralor.com/items/itm_a9bd6490b66bdf5d3063c993"
published: "2026-08-03T17:07:50.000Z"
score: 77
channel: "ai"
tags: ["ALFWorld","RoMeRL","奖励污染","自进化智能体","记忆强化学习"]
---

# RoMeRL：通过降阶效用状态平衡自进化智能体记忆中的反馈覆盖与记忆-奖励陷阱

## AI 点评

> 这篇论文戳中了自进化智能体记忆系统中的两个痛点：反馈稀释和奖励污染。如果你在做 Agent 的长期记忆或反思机制，RoMeRL 的降阶思想可能会给你一个很实用的新思路——用固定语义坐标代替无限增长的轨迹索引，既省内存又减少了误更新。实验数字也很亮眼，特别是 Cold-Q 降低 80% 和 LLM 调用减少 21%，说明不仅理论上漂亮，工程上也划算。

## 速览

本文提出 RoMeRL 框架，解决自进化 LLM 智能体记忆中反馈分散与记忆-奖励陷阱问题。该方法以固定维度的语义坐标表示任务记忆，将效用状态按结果极性和记忆动态因式分解，在覆盖新经验的同时保持反馈集中。理论上证明降阶参数化能提高每个效用坐标的平均反馈，并刻画错误坐标的稳态占用率。在 ALFWorld 和 LifelongAgentBench 上，RoMeRL 将 Cold-Q 比率降低 80%，反馈密度提升约 6 倍，内存占用减少 84.4%，LLM 调用减少 21.1%。

## 关键点

- 提出 RoMeRL 方法，用固定维度的语义坐标表示智能体记忆，解决反馈覆盖与记忆-奖励陷阱的耦合问题。
- 理论上证明降阶效用状态能增加每个坐标的平均反馈，并分析错误坐标的稳态占用。
- 在 ALFWorld 和 LifelongAgentBench 上，Cold-Q 比率降低 80%，反馈密度提升约 6 倍，内存减少 84.4%，LLM 调用减少 21.1%。

## 全文

Learning-based memory systems for self-evolving LLM agents face two tightly coupled challenges. First, trajectory-indexed utilities grow with the interaction history, thereby dispersing limited feedback over an ever-expanding state space. Second, because trajectory-level rewards are jointly assigned to co-retrieved memories, irrelevant experiences may receive misleading utility updates and consequently enter the memory-reward trap. To address these challenges, we introduce Reduced-Order Memory Reinforcement Learning (RoMeRL), which represents the growing trajectory-indexed utility space using a fixed-dimensional per-task memory state factorized by outcome polarity and memory dynamics. RoMeRL incorporates new experiences through a fixed set of semantic coordinates whose contents are updated or replaced over time, thereby concentrating feedback over a bounded utility support. Theoretically, we show that this reduced-order parameterization increases the average feedback received by each utility coordinate and characterize the steady-state occupancy of erroneous coordinates under a generic coordinate-transition model. Empirically, across ALFWorld and LifelongAgentBench, RoMeRL improves task performance, reduces the Cold-Q ratio by 80.0%, increases feedback density by approximately 6.0 times, reduces the maintained memory size by 84.4%, and cuts LLM calls by 21.1%. These results show that reduced-order utility states support efficient self-evolving agent memory while limiting persistent reward contamination. Code is available at: https://github.com/YOUNG-fnxm/RoMeRL

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_a9bd6490b66bdf5d3063c993