---
title: "Satisfied, Not Solving: Wireheading of Bounded Homeostatic Rewards in Reinforcement Learning"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2611.01183v1"
permalink: "https://feed.astralor.com/items/itm_10c2f667ec30500da0e809fd"
published: "2026-10-31T18:30:01.000Z"
score: 56
channel: "tech"
tags: ["AI Agent","Satisfied","Not Solving","Wireheading","Bounded Homeostatic Rewards","Reinforcement Learning Bounded","Bounded","MiniGrid DoorKey-8x8"]
---

# Satisfied, Not Solving: Wireheading of Bounded Homeostatic Rewards in Reinforcement Learning

## 速览

Bounded intrinsic rewards are often assumed to be safe from exploitation: if a signal can yield at most a fixed amount, gaming it should not pay. We test this for a homeostatic satiation reward, in which needs build up …

## 关键点

- Bounded intrinsic rewards are often assumed to be safe from exploitation: if a signal can yield at most a fixed amount, gaming it should not pay.
- We test this for a homeostatic satiation reward, in which needs build up and relieving them is rewarding, and give the agent a cheap action that resets its needs.
- In MiniGrid DoorKey-8x8, PPO agents with this action solved 2 of 80 seeds, against 71 of 80 on the same seeds without it; in KeyCorridorS3R1, 0 of 20 against 20 of 20.

## 全文

Bounded intrinsic rewards are often assumed to be safe from exploitation: if a signal can yield at most a fixed amount, gaming it should not pay. We test this for a homeostatic satiation reward, in which needs build up and relieving them is rewarding, and give the agent a cheap action that resets its needs. In MiniGrid DoorKey-8x8, PPO agents with this action solved 2 of 80 seeds, against 71 of 80 on the same seeds without it; in KeyCorridorS3R1, 0 of 20 against 20 of 20. Their policies stayed near uniform. The reward does not ask for this: a telescoping argument caps any policy's satiation return at the needs' starting drive, and a post-hoc computation shows that, at every growth rate we trained, a solving trajectory beats that cap on all sampled layouts. In DoorKey with advantage normalization on, a satisfaction signal the agent could fill itself was worse than none (11 of 20 solved with task reward alone, 1 of 20 with the tamperable signal), and a per-press cost of 0.1 restored the task in every seed. Farming of positive reward cycles appeared in DoorKey but not persistently in KeyCorridorS3R1. All predictions were pre-registered, and hash-locked from round 3 on; 9 of our 23 hypotheses were not supported, and we report each.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_10c2f667ec30500da0e809fd