---
title: "One Frozen Simulator Is Not Enough: Simulator Collapse in Multi-Agent RL"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.12253v1"
permalink: "https://feed.astralor.com/items/itm_4db432bcd444f5e1c0fbc3a6"
published: "2026-08-12T16:55:50.000Z"
score: 78
channel: "ai"
tags: ["Co-Training","LLM","SCOPE","Verbalized Sampling","Multi-Agent Reinforcement Learning","Simulator Collapse"]
---

# One Frozen Simulator Is Not Enough: Simulator Collapse in Multi-Agent RL

## AI commentary

> This paper hits a hidden but critical pain point in agent training: simulators that are too 'single' cause models to fail in front of real users. For teams building human-AI interaction agents, the Co-Training approach and the SCOPE framework are especially worth attention, as they could save significant trial-and-error time.

## At a glance

This paper shows that using a single large language model to simulate user behavior in multi-agent reinforcement learning for human-AI interaction systematically fails to generalize due to a phenomenon called simulator collapse. Because the simulator LLM is mode-collapsed, the policy overfits to narrow strategies that exploit the dominant mode, transferring poorly to unseen simulators and real users. The authors formalize this collapse theoretically and propose two complementary solutions: Verbalized Sampling at inference time, which broadens behavior by sampling from a verbalized response distribution, and Co-Training at training time, which jointly optimizes the policy against a population of trainable simulators. Experiments on Persuasion for Good, τ²-bench, and CooperBench show that Verbalized Sampling improves held-out success by up to 9% over single-simulator RL, and Co-Training pushes gains to 14%, with a human study confirming similar gains. The paper also releases SCOPE, an open-source framework for population co-training, and argues that training-environment diversity, not just policy diversity, is critical for real-world deployment.

## Key points

- Identifies simulator collapse: a single LLM simulator's mode collapse causes multi-agent RL policy overfitting and poor generalization.
- Proposes inference-time method Verbalized Sampling to broaden simulator behavior via verbalized response distribution sampling.
- Proposes training-time method Co-Training to jointly optimize policies against a population of trainable simulators.
- Validates improvements across three benchmarks and a human study, and releases the SCOPE open-source framework.

## Full text

Multi-agent reinforcement learning for human-AI interaction typically relies on a single large language model to simulate user behavior. We show that this approach systematically fails to generalize, and trace the failure to simulator collapse: because the simulator LLM is mode-collapsed, an LLM policy trained against it overfits to narrow strategies that exploit the simulator's dominant mode, and such a policy transfers poorly to unseen simulators and real users. We formalize this collapse theoretically and propose two complementary solutions, one at inference time and one at training time. The inference-time solution, Verbalized Sampling, broadens the simulator's behavior by sampling from a verbalized response distribution, reducing mode collapse. The training-time solution, Co-Training, jointly optimizes the policy against a population of trainable simulators, preventing it from overfitting to any single simulator's mode. We validate both solutions on three multi-turn benchmarks: Persuasion for Good, $τ^2$-bench, and CooperBench. Verbalized Sampling improves held-out success by up to 9% over single-simulator RL, and Co-Training pushes gains further to 14%; the human study shows similar gain on real users. Both solutions preserve the policy diversity that collapses under single-simulator RL. To support further work in this direction, we release SCOPE, an open-source framework for Population Co-Training multi-agent RL. More broadly, our results suggest that the diversity of the training environment, not only the policy, is critical to the generalization of multi-turn RL to real-world deployment.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_4db432bcd444f5e1c0fbc3a6