---
title: "Capital Markets LLM Reliability Score: From Plausible to Bankable"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.21340v1"
permalink: "https://feed.astralor.com/items/itm_9bdb665c44c308802f933e03"
published: "2026-07-23T14:10:58.000Z"
score: 83
channel: "ai"
tags: ["Anthropic","Llama 3.3 70B","LLM Evaluation","Meta","OpenAI","Capital Markets"]
---

# Capital Markets LLM Reliability Score: From Plausible to Bankable

## AI commentary

> This paper is valuable for both finance and AI communities: it provides an actionable, workflow-oriented evaluation framework that's closer to real needs than generic benchmarks. If you're deploying LLMs in fintech or responsible for explainability/auditability, this is a must-read. The finding on decision usefulness is particularly striking—judges agree it's critical, yet models vary the most on it.

## At a glance

Existing financial benchmarks evaluate QA-level accuracy but miss the 'bankability' required in real-world capital markets workflows. This paper introduces CM-LRS, assessing LLM outputs across seven dimensions: factual accuracy, evidence traceability, numerical consistency, workflow completeness, source discipline, decision usefulness, and auditability, each scored 0-5. Testing four models on five workflows, they find frontier closed-source models cluster within 0.22 points, while open-weight baselines lag significantly, especially on retrieval and synthesis tasks. Decision usefulness shows the widest cross-model dispersion and highest inter-judge agreement.

## Key points

- CM-LRS evaluates LLM outputs in capital markets workflows across seven dimensions, prioritizing 'bankability' over surface fluency.
- Frontier closed-source models (Sonnet 4.6, Opus 4.7, GPT-5.5) score within 0.22 points, while Llama 3.3 70B lags significantly.
- The gap concentrates on retrieval (2.23-point gap) and synthesis (2.15) rather than extraction (0.84).
- Decision usefulness shows the widest cross-model dispersion (4.0 points on issuer profiling) and highest inter-judge agreement (mean r=0.52).

## Full text

In capital-markets workflows the question is rarely whether a large language model can produce a fluent draft, but whether the draft is bankable: defensible in front of a counter-party or a regulator, with the documents in hand. Existing methods address parts of that gap: open-domain QA benchmarks reward surface accuracy, and finance benchmarks (FinanceBench, FinQA, ConvFinQA) advance document-grounded and numerical QA but evaluate at the question-answer layer rather than the workflow outputs practitioners defend. We introduce CM-LRS, a Capital Markets LLM Reliability Score, evaluating outputs at the workflow-output layer across seven dimensions: factual accuracy, evidence traceability, numerical consistency, workflow completeness, source discipline, decision usefulness, and reviewability/auditability. Each is scored 0-5 against a rubric anchored on signals reviewers in regulated settings use; the aggregate is tunable to the workflow. We demonstrate CM-LRS on five workflows (DCM transaction-terms extraction, precedent retrieval, issuer profile synthesis, M&A transaction-comparable reasoning, ECM transaction-terms extraction) over public SEC EDGAR filings, a public UK takeover release, and fictional synthetic supplements, scoring four models against four independent LLM judges spanning three model families. Three findings. First, the frontier closed-source models cluster within 0.22 points on four-judge averaged CM-LRS (Sonnet 4.6 = 4.31, Opus 4.7 = 4.30, GPT-5.5 = 4.09); all four judges place the open-weights baseline (Llama 3.3 70B = 3.15) last. Second, that gap concentrates on retrieval (2.23) and synthesis (2.15), not extraction (0.84). Third, Decision Usefulness shows the widest cross-model dispersion of any dimension (4.0 points on issuer profiling) and top-tier inter-judge agreement (mean r = 0.52). Plausibility is cheap. Bankability is the bar.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_9bdb665c44c308802f933e03