---
title: "AVA-Encoder: Towards Agent-Native Video Representation Learning"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.12313v1"
permalink: "https://feed.astralor.com/items/itm_1c97be6b6bc9e9853225932a"
published: "2026-08-12T17:58:02.000Z"
score: 78
channel: "ai"
tags: ["arXiv","AVA-Encoder","Agent","Knowledge Graph","Video Representation Learning"]
---

# AVA-Encoder: Towards Agent-Native Video Representation Learning

## AI commentary

> What stands out most is using knowledge graphs as an intermediate language between agents and video, bypassing the limits of end-to-end black boxes. If you are working on video agents or controllable generation, this framework and dataset are worth trying.

## At a glance

This paper proposes AVA-Encoder, an agent-native video representation learning framework that transforms videos into knowledge graph representations and reconstructs them back, enabling hierarchical structured text storage and an editable asset layer. Its novel textual-gradient optimization mechanism allows agents to iteratively improve encoding policies via natural language instructions. Experiments show AVA-Encoder outperforms the strongest external baseline by 20.7 percentage points, and its pseudo-trained policy surpasses carefully human-tuned baselines while using 74.3% fewer system-prompt tokens. The framework also releases the first high-quality film KG representation dataset.

## Key points

- AVA-Encoder converts video into a knowledge graph representation and reconstructs it back, forming a closed loop that agents can understand, query, and edit.
- A textual-gradient optimization framework expresses evaluation feedback as natural-language update directions for both outer-loop policy training and inner-loop test-time refinement.
- It outperforms the strongest external baseline by 20.7%, and the pseudo-trained policy beats human-tuned baselines with 74.3% fewer system-prompt tokens.
- The complete framework, a reliable agentic video reconstruction benchmark, and the first high-quality film KG dataset are released.

## Full text

Creative agents still lack an effective way to learn from high-quality human films, limiting their ability to produce cinematic-grade videos. A key challenge is the absence of a structured video representation that is both faithful to film content and directly usable for agentic reasoning and manipulation. To address the challenge, we propose the Agentic Video Auto-Encoder (AVA-Encoder), a framework for learning agent-native video representations via agentic auto-encoding. AVA-Encoder transforms a video into a knowledge graph (KG) representation and then reconstructs it back into video. Its hierarchy and state nodes store structured text, while a linked asset layer holds generated images, audio, and video. Typed edges preserve the relations between these text descriptions and assets in a form that agents can easily understand, query, and edit. The video reconstruction differences drive a textual-gradient optimization framework, which expresses evaluation feedback as natural-language update directions for Data-Independent Encoding Policy Pseudo-Training in the outer loop and optional Data-Dependent KG Representation Refinement in the test-time inner loop. Extensive experiments show that AVA-Encoder improves by 20.7 percentage points over the strongest external baseline. In the controlled policy-only setting, its pseudo-trained shot-level Agentic Video Encoder policy also outperforms a carefully human-tuned policy while using 74.3% fewer system-prompt tokens. We release the complete AVA-Encoder framework, a reliable agentic video reconstruction benchmark, and the first dataset of high-quality film KG representations.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_1c97be6b6bc9e9853225932a