---
title: "What Does the Model See? Representation Validity in Multimodal LLM-Based Video Analysis"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2611.05181v1"
permalink: "https://feed.astralor.com/items/itm_e0ca94a00116b90681bef44e"
published: "2026-11-03T16:42:49.000Z"
score: 58
channel: "ai"
tags: ["LLM","多模态","What Does","Model See","Representation Validity","Multimodal LLM-Based Video","Analysis Multimodal","MLLMs"]
---

# What Does the Model See? Representation Validity in Multimodal LLM-Based Video Analysis

## At a glance

Multimodal large language models (MLLMs) allow social scientists to analyze video at scale, but researchers must decide which parts of a video to provide as input. We use the term representation validity to examine how …

## Key points

- Multimodal large language models (MLLMs) allow social scientists to analyze video at scale, but researchers must decide which parts of a video to provide as input.
- We use the term representation validity to examine how text, audio, video, and combined inputs affect measurement and subsequent analysis.
- We compare input representations across six open-weight MLLMs and three human-annotated video datasets.

## Full text

Multimodal large language models (MLLMs) allow social scientists to analyze video at scale, but researchers must decide which parts of a video to provide as input. We use the term representation validity to examine how text, audio, video, and combined inputs affect measurement and subsequent analysis. We compare input representations across six open-weight MLLMs and three human-annotated video datasets. No representation consistently performs best. On seven news-related constructs, native-video performance varies little across models (macro-F1 = .70-.77), while extracted-text performance ranges from .52 to .80. Three models perform better with text and three with video. Combining inputs rarely improves performance. Higher classification accuracy also does not reliably produce estimates closer to those based on human codes. The highest-F1 representation best reproduces human-coded regression coefficients in 17 of 42 comparisons. Even when each model uses its highest-F1 deployable representation, 45% of sample-prevalence estimates differ from the human-coded benchmark by more than ten percentage points. Input representation therefore requires validation for both the construct being measured and the quantity estimated from the resulting labels.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_e0ca94a00116b90681bef44e