---
title: "NetlistBench: Evaluating LLM Reliability in SPICE Netlist Recognition and Manipulation"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.12197v1"
permalink: "https://feed.astralor.com/items/itm_080960891e022b35d9b6ecc0"
published: "2026-08-12T15:51:52.000Z"
score: 76
channel: "tech"
tags: ["EDA","LLM","NetlistBench","SPICE","Chip Design"]
---

# NetlistBench: Evaluating LLM Reliability in SPICE Netlist Recognition and Manipulation

## AI commentary

> The real value of this paper is separating 'textual fluency' from 'structural correctness' completely. If you're working on chip design automation, don't be fooled by high LLM scores on simple edits—that's exactly the danger zone, as slightly complex structural operations immediately expose fatal flaws. I recommend all EDA toolchain owners treat NetlistBench as a basic health check.

## At a glance

SPICE netlists are core descriptive files for chip simulation and verification; although text-based, they encode structured circuit objects through topology and parameters. NetlistBench constructs a structure-verified benchmark with 2,342 cases across 24 task families, covering parameter and connectivity recognition, edits, hierarchical operations, equivalence judgment, and long-horizon compound editing. Evaluations use a deterministic structure-aware oracle on six non-thinking LLMs, showing simple local edits reach 96%-100% accuracy, while device addition drops to 41%-83% and equivalence judgment to 49%-90%. Enabling reasoning substantially improves weaker models but does not eliminate structure-preservation failures, and performance degrades sharply as the edit horizon increases. The study identifies netlist-level reliability as a distinct bottleneck for trustworthy LLM-based circuit design automation.

## Key points

- NetlistBench contains 2,342 cases across 24 task families, covering parameter recognition, connectivity edits, hierarchical operations, equivalence judgment, and long-horizon compound editing.
- The benchmark uses a deterministic structure-aware oracle, avoiding LLM-based or fuzzy matching for reproducible automated scoring.
- Six non-thinking LLMs achieve 96%-100% accuracy on simple local edits but drop significantly on device addition and equivalence judgment.
- Enabling reasoning notably improves weaker models but fails to fix structure-preservation failures; performance degrades sharply with longer edit horizons.

## Full text

Large Language Models (LLMs) are increasingly used in circuit design workflows, yet their reliability on simulator-facing SPICE netlist recognition and manipulation remains poorly understood and is rarely separated from high-level design reasoning. Although netlists are textual, they encode structured circuit objects through topology and parameters. We present \textbf{NetlistBench}, a structure-verified benchmark for SPICE netlist recognition and manipulation. NetlistBench contains 2,342 cases across 24 task families, covering parameter and connectivity recognition and edits, hierarchical operations, equivalence judgment, and long-horizon compound editing. Model outputs are evaluated by a deterministic structure-aware oracle. Across six non-thinking LLMs, performance varies substantially with operation-level structural complexity. Simple local edits reach $96\%$--$100\%$ accuracy, while device addition drops to $41\%$--$83\%$ and equivalence judgment to $49\%$--$90\%$. Enabling reasoning substantially improves weaker models but does not eliminate structure-preservation failures, with performance still degrading sharply as the edit horizon increases. NetlistBench identifies netlist reliability as a distinct bottleneck for trustworthy LLM-based circuit design automation.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_080960891e022b35d9b6ecc0