---
title: "文本摘要抽象性评估新指标：RA、SA与AR的提出与验证"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.10806v1"
permalink: "https://feed.astralor.com/items/itm_b7158e91248ec274b397625e"
published: "2026-07-12T15:25:57.000Z"
score: 79
channel: "tech"
tags: ["BART","Pegasus","摘要抽象性","文本摘要","评估指标"]
---

# 文本摘要抽象性评估新指标：RA、SA与AR的提出与验证

## AI 点评

> 这篇论文最值得关注的是它解决了摘要评估中长期被忽视的抽象性量化问题。对于研究摘要生成和幻觉检测的同学，这三个指标可以直接应用于你的实验，帮助更精细地分析模型行为。不过要注意XSUM数据集规模有限，跨数据集推广时需谨慎。

## 速览

这篇论文提出了一套新的启发式指标——参考抽象性（RA）、摘要抽象性（SA）和抽象比率（AR），用于定量衡量摘要相对于源文本的抽象程度。通过对XSUM数据集上四个摘要模型（BART-large-cnn、Pegasus-xsum、DistilBart、MT5-small）的评估，指标成功区分了抽取式模型（SA约0.12-0.26）与抽象式模型（SA约0.96-1.77），并发现AR能识别需要人工检查潜在幻想的摘要。该工作为超越ROUGE等表层指标的摘要评估提供了新工具。

## 关键点

- 提出RA、SA、AR三个指标，基于文档长度调和的谐波均值与立方非重叠因子，具有维度一致性和有界性。
- 在XSUM数据集上对不同类型摘要模型进行验证，SA能清晰区分抽取式与抽象式模型。
- AR指标可辅助识别可能包含幻觉的摘要，提示需要人工评估。
- 代码与结果已开源，便于复现与进一步研究。

## 全文

Quantifying abstractiveness in generated summaries is essential for evaluating summarization models beyond surface-level metrics like ROUGE. We introduce Reference Abstraction (RA), Summary Abstraction (SA), and Abstraction Ratio (AR) -- a set of principled heuristic metrics that measure how much a summary diverges from extractive copying of the source text. The formulation uses the harmonic mean of document lengths modulated by a cubic non-overlap factor, yielding dimensionally consistent, bounded output with non-linear sensitivity to the extractive-abstractive boundary. Evaluation on 100 XSUM documents across four summarization models (BART-large-cnn, Pegasus-xsum, DistilBart, MT5-small) demonstrates that the metrics successfully discriminate between extractive models (SA ~ 0.12-0.26) and abstractive models (SA ~ 0.96-1.77), and that the Abstraction Ratio identifies summaries requiring manual evaluation for potential hallucination. Code and results are available at https://github.com/katweNLP/AbstractionStudy.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_b7158e91248ec274b397625e