---
title: "CANDO: Conservative Action-Noise Directional Optimization for Residual Reinforcement Learning"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2611.04543v1"
permalink: "https://feed.astralor.com/items/itm_74eb1eb2c1ce2089467c86c0"
published: "2026-11-03T09:11:27.000Z"
score: 61
channel: "tech"
tags: ["CANDO","Conservative Action-Noise Directional","Optimization","Residual Reinforcement Learning","Pretrained","Noise-space","Conservative Noise Directional"]
---

# CANDO: Conservative Action-Noise Directional Optimization for Residual Reinforcement Learning

## 速览

Pretrained generative policies provide strong behavior priors for robot control, but adapting them efficiently with online reinforcement learning remains difficult. A central challenge is how to improve a frozen generat…

## 关键点

- Pretrained generative policies provide strong behavior priors for robot control, but adapting them efficiently with online reinforcement learning remains difficult.
- A central challenge is how to improve a frozen generative policy without destroying the structure learned from offline demonstrations.
- Noise-space residual methods address this by modifying the policy's input noise, but existing noise-space methods typically regulate adaptation through fixed latent feasible regions, which may exclude valuable noise edi…

## 全文

Pretrained generative policies provide strong behavior priors for robot control, but adapting them efficiently with online reinforcement learning remains difficult. A central challenge is how to improve a frozen generative policy without destroying the structure learned from offline demonstrations. Noise-space residual methods address this by modifying the policy's input noise, but existing noise-space methods typically regulate adaptation through fixed latent feasible regions, which may exclude valuable noise edits. In contrast, action-space residual methods are more sample-efficient, but their efficiency can still be further improved. We propose Conservative Noise Directional Optimization (CNDO), a noise-space residual RL method that improves a frozen generative policy by learning state- and noise-conditioned directions around Gaussian latent samples. CNDO removes predefined latent feasible regions and instead regulates noise-space adaptation through the soft conservative constraint, which consists of conservative value estimation and gated execution. We further introduce Conservative Action-Noise Directional Optimization (CANDO), which adds an action-space residual, combining structured latent-space improvement with local action correction. Experiments show that, once CNDO narrows the gap between noise-space and action-space residual learning, CANDO further improves over single-space optimization, achieving 76.1 NAUC and a 91.4% final success rate, surpassing the best single-space results by 6.6 NAUC and 4.1% success rate. More demos and info are available on https://cando-rl.github.io/.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_74eb1eb2c1ce2089467c86c0