---
title: "TaskFrame: Connecting Task-Level Planning with Frame-Level Control for Game Agents"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2610.40276v1"
permalink: "https://feed.astralor.com/items/itm_0fd42ac5742b2b0b0454ea9b"
published: "2026-10-29T15:31:37.000Z"
score: 55
channel: "tech"
tags: ["AI Agent","推理效率","安全","多模态","TaskFrame","Connecting Task-Level Planning","Frame-Level Control","Game Agents Building"]
---

# TaskFrame: Connecting Task-Level Planning with Frame-Level Control for Game Agents

## 速览

Building general game agents requires both high-level task reasoning and low-latency action control in rapidly changing visual environments. These capabilities favor models with substantially different computational pro…

## 关键点

- Building general game agents requires both high-level task reasoning and low-latency action control in rapidly changing visual environments.
- These capabilities favor models with substantially different computational profiles, making them difficult to realize within a single model.
- In this work, we present TaskFrame, a hierarchical planner-controller framework that connects task-level planning with frame-level keyboard-and-mouse control.

## 全文

Building general game agents requires both high-level task reasoning and low-latency action control in rapidly changing visual environments. These capabilities favor models with substantially different computational profiles, making them difficult to realize within a single model. In this work, we present TaskFrame, a hierarchical planner-controller framework that connects task-level planning with frame-level keyboard-and-mouse control. A vision-language task planner decomposes a high-level goal into an ordered sequence of atomic instructions and is optimized through supervised learning and dependency-aware GRPO. The hidden states of special goal tokens that terminate each atomic instruction provide continuous goal embeddings that are cached and reused across control steps. A causal Transformer controller combines the embedding of the active goal with recent observations to predict discrete keyboard and mouse-button actions and continuous camera motion at every game step. We train the system in three stages: planner training, controller pretraining, and joint alignment. Experiments in Minecraft show that TaskFrame is competitive with general-purpose vision-language models on instruction decomposition, and that in closed-loop evaluation the agent achieves a higher combat success rate than the evaluated baselines.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_0fd42ac5742b2b0b0454ea9b