---
title: "RoMeRL: Balancing Feedback Coverage and the Memory-Reward Trap in Self-Evolving Agent Memory via Reduced-Order Utility States"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.02508v1"
permalink: "https://feed.astralor.com/items/itm_a9bd6490b66bdf5d3063c993"
published: "2026-08-03T17:07:50.000Z"
score: 77
channel: "ai"
tags: ["ALFWorld","RoMeRL","Reward Contamination","Self-Evolving Agent","Memory Reinforcement Learning"]
---

# RoMeRL: Balancing Feedback Coverage and the Memory-Reward Trap in Self-Evolving Agent Memory via Reduced-Order Utility States

## AI commentary

> This paper hits two pain points in self-evolving agent memory systems: feedback dilution and reward contamination. If you are working on long-term memory or reflection mechanisms for agents, RoMeRL's reduced-order idea offers a practical new direction—replacing unbounded trajectory indices with fixed semantic coordinates saves memory and reduces mis-updates. The experimental numbers are impressive, especially the 80% reduction in Cold-Q ratio and 21.1% fewer LLM calls, showing the method is not just theoretically elegant but also cost-effective in practice.

## At a glance

This paper proposes RoMeRL to address the dispersion of feedback and the memory-reward trap in self-evolving LLM agent memory. It represents task memory with fixed-dimensional semantic coordinates and factorizes utility states by outcome polarity and memory dynamics, allowing new experiences to be incorporated while keeping feedback concentrated. Theoretically, the reduced-order parameterization increases average feedback per utility coordinate and characterizes steady-state occupancy of erroneous coordinates. On ALFWorld and LifelongAgentBench, RoMeRL reduces Cold-Q ratio by 80.0%, increases feedback density by approximately 6.0 times, reduces maintained memory size by 84.4%, and cuts LLM calls by 21.1%.

## Key points

- Introduces RoMeRL, which represents agent memory with fixed-dimensional semantic coordinates to address the coupled challenges of feedback coverage and the memory-reward trap.
- Theoretically demonstrates that reduced-order utility states increase average feedback per coordinate and analyzes steady-state occupancy of erroneous coordinates.
- On ALFWorld and LifelongAgentBench, Cold-Q ratio drops by 80%, feedback density rises ~6x, memory size is cut by 84.4%, and LLM calls are reduced by 21.1%.

## Full text

Learning-based memory systems for self-evolving LLM agents face two tightly coupled challenges. First, trajectory-indexed utilities grow with the interaction history, thereby dispersing limited feedback over an ever-expanding state space. Second, because trajectory-level rewards are jointly assigned to co-retrieved memories, irrelevant experiences may receive misleading utility updates and consequently enter the memory-reward trap. To address these challenges, we introduce Reduced-Order Memory Reinforcement Learning (RoMeRL), which represents the growing trajectory-indexed utility space using a fixed-dimensional per-task memory state factorized by outcome polarity and memory dynamics. RoMeRL incorporates new experiences through a fixed set of semantic coordinates whose contents are updated or replaced over time, thereby concentrating feedback over a bounded utility support. Theoretically, we show that this reduced-order parameterization increases the average feedback received by each utility coordinate and characterize the steady-state occupancy of erroneous coordinates under a generic coordinate-transition model. Empirically, across ALFWorld and LifelongAgentBench, RoMeRL improves task performance, reduces the Cold-Q ratio by 80.0%, increases feedback density by approximately 6.0 times, reduces the maintained memory size by 84.4%, and cuts LLM calls by 21.1%. These results show that reduced-order utility states support efficient self-evolving agent memory while limiting persistent reward contamination. Code is available at: https://github.com/YOUNG-fnxm/RoMeRL

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_a9bd6490b66bdf5d3063c993