---
title: "SciCode-Verified：基准缺陷如何低估了语言模型的科学编程能力"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.04975v1"
permalink: "https://feed.astralor.com/items/itm_a470fe8e35f83042915ad4ba"
published: "2026-08-05T15:45:55.000Z"
score: 78
channel: "ai"
tags: ["AI评估","SciCode","SciCode-Verified","科学计算","语言模型"]
---

# SciCode-Verified：基准缺陷如何低估了语言模型的科学编程能力

## AI 点评

> 这项研究给我最大的冲击是：我们信赖的基准可能一直在惩罚正确答案。对于模型开发者和评估研究者，它意味着必须重新审视所有基于SciCode的结论；对于产业方，它说明某些模型的能力天花板也许远高于之前报告的数字。强烈建议阅读原论文的审计记录。

## 速览

SciCode被普遍用于衡量语言模型的科学编程能力，但近一年模型得分停滞在60%左右。一项对全部65道测试题的领域专家审计发现263处缺陷，其中192处导致正确解法被误判，且78%需要专业物理数学知识才能识别。修正后的SciCode-Verified使十二个前沿模型的子问题准确率从45–60%升至84–98%，主问题准确率从9–27%升至69–92%。这证明瓶颈不在模型能力，而在基准测试本身。

## 关键点

- 专家审计全部65道SciCode题目发现263处缺陷，其中192处导致正确解法被误判。
- 78%的缺陷需要专业物理或数学知识才能识别，而非简单校对。
- 在SciCode-Verified上，前沿模型子问题准确率从45–60%升至84–98%。
- 结果表明，评测缺陷而非模型能力导致了得分停滞。

## 全文

SciCode is the standard measure of the scientific-coding ability of language models: research-level problems that demand both frontier scientific theory and its implementation as working numerical code. It is a component of the Artificial Analysis Intelligence Index and a standing evaluation in government and national-laboratory suites. Yet its scores have recently plateaued: the strongest 2026 models cluster tightly around 60\% subproblem accuracy, and a successor model ties its predecessor. We trace this stagnation to defects in the benchmark itself. A per-problem, domain-expert audit of all 65 test problems uncovers 263 defects; 192 of them, spread across 91\% of the main problems, cause correct, instruction-following solutions to be wrongly rejected---through non-reproducible gold answers, over-tight tolerances, or self-contradictory specifications. Critically, 78\% of these score-suppressing defects require specialized physics or mathematics knowledge to detect, not mere clerical proofreading. We corrected every confirmable defect to produce SciCode-Verified. The corrections add only the specifications a well-posed problem requires, repair grading, and tighten the tests that were too lenient; every change is recorded with its justification and independently re-checked by a second domain expert. We re-evaluate twelve frontier model snapshots on the corrected benchmark and find a substantial recovery: subproblem accuracy rises from 45--60\% to 84--98\%, and main-problem accuracy from 9--27\% to 69--92\%. State-of-the-art models are far more proficient in scientific coding than SciCode has suggested---the bottleneck was not model capability, but the quality of the evaluation instrument. We release SciCode-Verified with its complete audit trail as the corrected public standard.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_a470fe8e35f83042915ad4ba