---
title: "SciCode-Verified: How Benchmark Defects Underestimated the Scientific-Coding Ability of Language Models"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.04975v1"
permalink: "https://feed.astralor.com/items/itm_a470fe8e35f83042915ad4ba"
published: "2026-08-05T15:45:55.000Z"
score: 78
channel: "ai"
tags: ["AI Evaluation","SciCode","SciCode-Verified","Scientific Computing","Language Model"]
---

# SciCode-Verified: How Benchmark Defects Underestimated the Scientific-Coding Ability of Language Models

## AI commentary

> The biggest shock for me is that the benchmark we trust may have been penalizing correct answers all along. For model developers and evaluation researchers, this means conclusions based on SciCode need to be revisited; for industry, it suggests that some models' capability ceilings may be far higher than previously reported. I strongly recommend reading the original audit trail.

## At a glance

SciCode is the standard benchmark for evaluating language models' scientific coding ability, yet recent scores have plateaued around 60% subproblem accuracy. A domain-expert audit of all 65 test problems uncovered 263 defects, 192 of which caused correct, instruction-following solutions to be wrongly rejected, with 78% requiring specialized physics or mathematics knowledge to detect. The corrected benchmark, SciCode-Verified, adds missing specifications, repairs grading, and tightens overly lenient tests. Re-evaluating twelve frontier model snapshots shows substantial recovery: subproblem accuracy rises from 45-60% to 84-98%, and main-problem accuracy from 9-27% to 69-92%. This indicates the bottleneck was not model capability but benchmark quality. The authors release SciCode-Verified with a complete audit trail as the corrected public standard.

## Key points

- An expert audit of all 65 SciCode problems found 263 defects, 192 of which caused correct solutions to be wrongly rejected.
- 78% of the defects require specialized physics or math knowledge to detect.
- On SciCode-Verified, subproblem accuracy of frontier models jumps from 45-60% to 84-98%.
- The results show that benchmark defects, not model capability, caused the plateau.

## Full text

SciCode is the standard measure of the scientific-coding ability of language models: research-level problems that demand both frontier scientific theory and its implementation as working numerical code. It is a component of the Artificial Analysis Intelligence Index and a standing evaluation in government and national-laboratory suites. Yet its scores have recently plateaued: the strongest 2026 models cluster tightly around 60\% subproblem accuracy, and a successor model ties its predecessor. We trace this stagnation to defects in the benchmark itself. A per-problem, domain-expert audit of all 65 test problems uncovers 263 defects; 192 of them, spread across 91\% of the main problems, cause correct, instruction-following solutions to be wrongly rejected---through non-reproducible gold answers, over-tight tolerances, or self-contradictory specifications. Critically, 78\% of these score-suppressing defects require specialized physics or mathematics knowledge to detect, not mere clerical proofreading. We corrected every confirmable defect to produce SciCode-Verified. The corrections add only the specifications a well-posed problem requires, repair grading, and tighten the tests that were too lenient; every change is recorded with its justification and independently re-checked by a second domain expert. We re-evaluate twelve frontier model snapshots on the corrected benchmark and find a substantial recovery: subproblem accuracy rises from 45--60\% to 84--98\%, and main-problem accuracy from 9--27\% to 69--92\%. State-of-the-art models are far more proficient in scientific coding than SciCode has suggested---the bottleneck was not model capability, but the quality of the evaluation instrument. We release SciCode-Verified with its complete audit trail as the corrected public standard.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_a470fe8e35f83042915ad4ba