---
title: "New Metrics for Evaluating Abstractiveness in Text Summarization: RA, SA, and AR with Empirical Validation"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.10806v1"
permalink: "https://feed.astralor.com/items/itm_b7158e91248ec274b397625e"
published: "2026-07-12T15:25:57.000Z"
score: 79
channel: "tech"
tags: ["BART","Pegasus","Abstractiveness Metrics","Text Summarization","Evaluation Metrics"]
---

# New Metrics for Evaluating Abstractiveness in Text Summarization: RA, SA, and AR with Empirical Validation

## AI commentary

> What makes this paper worth reading is its long-overdue solution to quantifying abstractiveness in summarization evaluation. For those working on summarization generation or hallucination detection, these three metrics can be directly applied to your experiments to analyze model behavior more granularly. However, note that the validation is limited to XSUM; caution is needed when generalizing to other datasets.

## At a glance

This paper introduces a set of principled heuristic metrics—Reference Abstraction (RA), Summary Abstraction (SA), and Abstraction Ratio (AR)—to quantitatively measure how much a summary diverges from extractive copying of the source text. Evaluated on XSUM with four summarization models (BART-large-cnn, Pegasus-xsum, DistilBart, MT5-small), the metrics successfully discriminate extractive models (SA ~0.12-0.26) from abstractive ones (SA ~0.96-1.77), and the AR identifies summaries that may require manual hallucination checking. This work provides a new tool beyond surface-level metrics like ROUGE.

## Key points

- Proposes RA, SA, and AR metrics based on harmonic mean of document lengths modulated by a cubic non-overlap factor, ensuring dimensional consistency and boundedness.
- Validated on XSUM across four summarization models, SA clearly separates extractive (0.12-0.26) from abstractive (0.96-1.77) models.
- Abstraction Ratio helps identify summaries potentially containing hallucinations, flagging them for manual evaluation.
- Code and results are open-sourced for reproducibility and further research.

## Full text

Quantifying abstractiveness in generated summaries is essential for evaluating summarization models beyond surface-level metrics like ROUGE. We introduce Reference Abstraction (RA), Summary Abstraction (SA), and Abstraction Ratio (AR) -- a set of principled heuristic metrics that measure how much a summary diverges from extractive copying of the source text. The formulation uses the harmonic mean of document lengths modulated by a cubic non-overlap factor, yielding dimensionally consistent, bounded output with non-linear sensitivity to the extractive-abstractive boundary. Evaluation on 100 XSUM documents across four summarization models (BART-large-cnn, Pegasus-xsum, DistilBart, MT5-small) demonstrates that the metrics successfully discriminate between extractive models (SA ~ 0.12-0.26) and abstractive models (SA ~ 0.96-1.77), and that the Abstraction Ratio identifies summaries requiring manual evaluation for potential hallucination. Code and results are available at https://github.com/katweNLP/AbstractionStudy.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_b7158e91248ec274b397625e