---
title: "An Evaluation Framework for Structured Audio Captions Validated by Controlled Perturbations"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.21424v1"
permalink: "https://feed.astralor.com/items/itm_2034282dd869a4e38f03d468"
published: "2026-07-23T15:26:52.000Z"
score: 75
channel: "tech"
tags: ["AudioCards","LLM Judge","Audio Caption Evaluation"]
---

# An Evaluation Framework for Structured Audio Captions Validated by Controlled Perturbations

## AI commentary

> This paper is highly valuable for the multimodal evaluation field, especially as structured audio captions become a trend. If you work on AAC or related generative model evaluation, this framework deserves a close read—it helps distinguish whether models truly understand audio content.

## At a glance

As automated audio captioning (AAC) shifts from monolithic sentence generation to structured formats, existing evaluation metrics fail to assess multimodal attributes. This paper proposes a multi-axis evaluation framework based on the AudioCards dataset, evaluating across five orthogonal axes: tag-sets, descriptions, logical reasoning, numeric measurements, and spectral profiles. It combines LLM judges with deterministic computational metrics and introduces a controlled perturbation testing protocol. Results show the framework successfully distinguishes meaning-preserving paraphrases from genuine semantic and acoustic corruptions.

## Key points

- Proposes a multi-axis evaluation framework for structured audio captions covering tags, descriptions, logic, numbers, and spectral profiles.
- Combines LLM-based semantic judges with deterministic acoustic metrics for comprehensive assessment.
- Designs a controlled perturbation testing protocol with graded errors to validate framework reliability.

## Full text

Recent advancements in automated audio captioning (AAC) have shifted from monolithic sentence generation toward structured formats that explicitly disentangle distinct acoustic and semantic properties. However, evaluating this heterogeneous data remains a significant challenge. Existing caption metrics focus on flat textual outputs and fail to reliably assess multimodal attributes. To bridge this gap, we propose a multi-axis evaluation framework tailored for structured audio descriptions. Building on the AudioCards dataset, we evaluate outputs across five orthogonal axes: tag-sets, descriptions, logical reasoning, numeric measurements, and spectral profiles. Our approach combines Large Language Model (LLM) judges to capture semantic nuance with deterministic computational metrics to precisely measure acoustic deviations. To rigorously validate the reliability of this framework, we introduce a controlled perturbation testing protocol that injects typed, graded errors into groundtruth annotations. Our results demonstrate that this framework successfully distinguishes meaning-preserving paraphrases from genuine semantic and acoustic corruptions.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_2034282dd869a4e38f03d468