---
title: "Similar Scores, Opposite Failures: Human Correction as an Instrument for Video Caption Evaluation"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2610.38511v1"
permalink: "https://feed.astralor.com/items/itm_2a8e6ffd851f98161d97ba9c"
published: "2026-10-28T09:25:58.000Z"
score: 57
channel: "ai"
tags: ["多模态","Similar Scores","Opposite Failures","Human Correction","Instrument","Video Caption Evaluation","Video","Six"]
---

# Similar Scores, Opposite Failures: Human Correction as an Instrument for Video Caption Evaluation

## 速览

Video captioning models are increasingly evaluated with metrics designed for single-sentence image captions. We test whether those metrics recover human judgement on dense, multi-window video description, and find that …

## 关键点

- Video captioning models are increasingly evaluated with metrics designed for single-sentence image captions.
- We test whether those metrics recover human judgement on dense, multi-window video description, and find that they do not.
- Six frontier models captioned 196 short clips.

## 全文

Video captioning models are increasingly evaluated with metrics designed for single-sentence image captions. We test whether those metrics recover human judgement on dense, multi-window video description, and find that they do not. Six frontier models captioned 196 short clips. Trained annotators post-edited every caption against a failure taxonomy fixed before annotation, once as a whole-clip overview and once window by window, and a quality-control reviewer checked each annotation, yielding 1,144 overview verdicts and 4,672 window verdicts. Reference-overlap metrics do not separate the captions reviewers accepted from those they rejected: BLEU-4 reaches an AUC of 0.510 and a CIDEr-style tf-idf cosine 0.528, and restricting references to reviewer corrections leaves BLEU-4 at 0.479 (95% CI 0.431-0.529), indistinguishable from chance. Review granularity changes what is detected. The same annotators record camera and shot errors on 2.7% of captions when reading the overview and on 12.8% when reading window by window, a 4.7x gap that holds across every robustness check we ran, on the mode most likely to need a near-total rewrite. Correct-overview rates span 54.1-72.7%, but only 4 of 15 pairwise gaps survive Bonferroni correction (5 after adjusting for reviewer and clip effects), and accuracy and derived severity order the models differently. Models with statistically indistinguishable accuracy have sharply different failure profiles, which no single scalar shows. We argue that caption evaluation should record what was wrong rather than only how often, and we report the taxonomy, the per-model profiles, the corpus properties that predict difficulty, and the code that regenerates every number in the paper.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_2a8e6ffd851f98161d97ba9c