---
title: "Redistribution-based Cost Inference Improves Sparse Safe Offline RL"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.12306v1"
permalink: "https://feed.astralor.com/items/itm_d1475e1e4d33ea582bd08e45"
published: "2026-08-12T17:53:15.000Z"
score: 77
channel: "ai"
tags: ["arXiv","RCI Framework","Safe Offline RL","Temporal Credit Assignment","CMDP"]
---

# Redistribution-based Cost Inference Improves Sparse Safe Offline RL

## AI commentary

> This paper deserves attention from researchers in safe reinforcement learning, especially those who only have trajectory-level danger signals in real deployments. It turns the seemingly intractable sparse cost problem into a provably lossless redistribution problem with a clean idea, and the experimental results support the claims. If you are working on constrained policy learning in realistic settings, this paper is a great starting point.

## At a glance

Safe offline reinforcement learning typically assumes access to dense per-step cost annotations, but in practice supervisors only provide trajectory-level stop-feedback: a binary signal at the first unsafe transition with no per-step attribution. The authors frame this as a temporal credit assignment problem and propose the Redistribution-based Cost Inference (RCI) framework, which converts sparse stop-feedback into dense per-step costs via return decomposition, then trains a constrained offline policy on the augmented dataset. They show that return-equivalent redistribution preserves the feasible policy set and the optimal Lagrangian in a CMDP, establishing that the transformation is lossless in theory while yielding better-conditioned cost critic learning in practice. Experiments on highway driving and robotic manipulation demonstrate substantially lower violation rates than sparse and classifier-based baselines, with robustness to heterogeneous dataset compositions and label noise.

## Key points

- Proposes the RCI framework, using return decomposition to convert trajectory-level stop-feedback into dense per-step costs, addressing temporal credit assignment.
- Theoretically proves that return-equivalent redistribution preserves the feasible policy set and the optimal Lagrangian in a CMDP, ensuring lossless transformation.
- Demonstrates substantially lower violation rates than sparse and classifier-based baselines on highway driving and robotic manipulation tasks.
- Shows robustness to heterogeneous dataset compositions and label noise, improving conditioning of cost critic learning.

## Full text

Safe offline RL typically assumes access to dense per-step cost annotations, but in practice supervisors provide only trajectory-level stop-feedback: a binary signal at the first unsafe transition, with no per-step attribution. We frame this as a temporal credit assignment problem and propose the Redistribution-based Cost Inference (RCI) framework, which converts sparse stop-feedback into dense per-step costs via return decomposition, then trains a constrained offline policy on the augmented dataset. We show that return-equivalent redistribution preserves the feasible policy set and the optimal Lagrangian in a CMDP, establishing that the transformation is lossless in theory while yielding better-conditioned cost critic learning in practice. Experiments on highway driving and robotic manipulation demonstrate substantially lower violation rates than sparse and classifier-based baselines, with robustness to heterogeneous dataset compositions and label noise.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_d1475e1e4d33ea582bd08e45