---
title: "RM-VLM: From Natural-Language Task Descriptions to Vision-Grounded Reward Machines"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2611.08308v1"
permalink: "https://feed.astralor.com/items/itm_349418f5841db7d1213c2123"
published: "2026-11-05T17:10:10.000Z"
score: 58
channel: "tech"
tags: ["LLM","AI Agent","多模态","RM-VLM","From Natural-Language Task","Descriptions","Vision-Grounded Reward Machines","Reward"]
---

# RM-VLM: From Natural-Language Task Descriptions to Vision-Grounded Reward Machines

## At a glance

Reward machines (RMs) have been proposed as a reward formulation in reinforcement learning (RL) to capture non-Markovian tasks where credit assignment depends on history. By representing reward functions as automata, RM…

## Key points

- Reward machines (RMs) have been proposed as a reward formulation in reinforcement learning (RL) to capture non-Markovian tasks where credit assignment depends on history.
- By representing reward functions as automata, RMs expose the task's temporal structure and allow such rewards to be handled within a Markovian framework.
- However, they assume a labelling function, a proposition-grounding oracle for observations, which must be manually engineered for each environment, limiting their wider applicability.

## Full text

Reward machines (RMs) have been proposed as a reward formulation in reinforcement learning (RL) to capture non-Markovian tasks where credit assignment depends on history. By representing reward functions as automata, RMs expose the task's temporal structure and allow such rewards to be handled within a Markovian framework. However, they assume a labelling function, a proposition-grounding oracle for observations, which must be manually engineered for each environment, limiting their wider applicability. In this work, we address this limitation by proposing RM-VLM, a novel approach in which a large language model (LLM) generates an RM from a natural-language task description, and a vision-language model (VLM) acts as the labelling function, grounding the RM's propositions over image-based observations during RM-guided policy learning. A VLM-based value function estimator is used to assess the agent's progress towards each RM transition and derive a dense reward signal. We evaluate RM-VLM on three domains covering non-Markovian grid-world navigation, robotic manipulation, and 3D object collection. Our experimental results show that RM-VLM consistently achieves higher task success rates than existing VLM-based reward methods across all tested domains.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_349418f5841db7d1213c2123