---
title: "资本市场LLM可靠性评分：从“看似合理”到“可银行化”"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.21340v1"
permalink: "https://feed.astralor.com/items/itm_9bdb665c44c308802f933e03"
published: "2026-07-23T14:10:58.000Z"
score: 83
channel: "ai"
tags: ["Anthropic","Llama 3.3 70B","LLM评估","Meta","OpenAI","资本市场"]
---

# 资本市场LLM可靠性评分：从“看似合理”到“可银行化”

## AI 点评

> 这篇论文对金融圈和AI圈都很有价值：它给出了一个可操作的、面向实际工作流的评估框架，比泛泛的基准更贴近真实需求。如果你在金融科技公司做LLM落地，或者负责模型的“可解释性”和“可审计性”，这篇值得细读。决策有用性维度的发现尤其值得关注——评审们一致认为这是关键，但不同模型表现差异最大。

## 速览

现有金融基准测试主要评估问答层面的准确性，忽略了实际工作中需要为输出负责的“可银行化”要求。本文提出CM-LRS，从事实准确性、证据可追溯性、数值一致性、工作流完整性、来源自律性、决策有用性和可审计性七个维度评估LLM输出，每个维度0-5分。在五个资本市场工作流上测试四个模型，发现闭源前沿模型差距很小，但开放权重基线显著落后，尤其在检索和综合任务上。决策有用性维度跨模型差异最大且评审间一致性最高。

## 关键点

- CM-LRS从七个维度评估LLM在资本市场工作流中的输出，聚焦“可银行化”而非表面流畅性。
- 闭源前沿模型（Sonnet 4.6、Opus 4.7、GPT-5.5）平均分差距仅0.22分，Llama 3.3 70B大幅落后。
- 差距主要集中在检索（2.23分差距）和综合任务（2.15分），而非抽取任务（0.84分）。
- 决策有用性维度跨模型差异最大（发行人画像任务差4.0分），且评审间一致性最高（平均r=0.52）。

## 全文

In capital-markets workflows the question is rarely whether a large language model can produce a fluent draft, but whether the draft is bankable: defensible in front of a counter-party or a regulator, with the documents in hand. Existing methods address parts of that gap: open-domain QA benchmarks reward surface accuracy, and finance benchmarks (FinanceBench, FinQA, ConvFinQA) advance document-grounded and numerical QA but evaluate at the question-answer layer rather than the workflow outputs practitioners defend. We introduce CM-LRS, a Capital Markets LLM Reliability Score, evaluating outputs at the workflow-output layer across seven dimensions: factual accuracy, evidence traceability, numerical consistency, workflow completeness, source discipline, decision usefulness, and reviewability/auditability. Each is scored 0-5 against a rubric anchored on signals reviewers in regulated settings use; the aggregate is tunable to the workflow. We demonstrate CM-LRS on five workflows (DCM transaction-terms extraction, precedent retrieval, issuer profile synthesis, M&A transaction-comparable reasoning, ECM transaction-terms extraction) over public SEC EDGAR filings, a public UK takeover release, and fictional synthetic supplements, scoring four models against four independent LLM judges spanning three model families. Three findings. First, the frontier closed-source models cluster within 0.22 points on four-judge averaged CM-LRS (Sonnet 4.6 = 4.31, Opus 4.7 = 4.30, GPT-5.5 = 4.09); all four judges place the open-weights baseline (Llama 3.3 70B = 3.15) last. Second, that gap concentrates on retrieval (2.23) and synthesis (2.15), not extraction (0.84). Third, Decision Usefulness shows the widest cross-model dispersion of any dimension (4.0 points on issuer profiling) and top-tier inter-judge agreement (mean r = 0.52). Plausibility is cheap. Bankability is the bar.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_9bdb665c44c308802f933e03