---
title: "Using DSPy to evaluate and improve Datasette Agent's SQL system prompts"
source: "Simon Willison 博客（Atom）"
source_tier: "t2"
url: "https://simonwillison.net/2026/Jul/2/dspy-datasette-agent-prompts/#atom-everything"
permalink: "https://feed.astralor.com/items/itm_38aa5ad2c12dd9a0f88601b0"
published: "2026-07-02T18:25:00.000Z"
score: 61
channel: "ai"
tags: ["Claude Code","Datasette Agent","DSPy","GPT-4.1","SQL Prompt Optimization"]
---

# Using DSPy to evaluate and improve Datasette Agent's SQL system prompts

## At a glance

After the AIE keynote, Simon Willison conducted an automated evaluation and improvement experiment on Datasette Agent's SQL system prompts using Claude Code and the DSPy framework. He selected GPT-4.1 mini and nano as test models and identified several promising optimization directions. Most critically, he found that including column names in the prompt's schema listing avoids column-name guessing and error-retry loops. This research demonstrates the practical value of DSPy in optimizing AI agent prompts.

## Key points

- Used the DSPy framework to automatically evaluate and optimize Datasette Agent's system prompts.
- Tested with GPT-4.1 mini and nano as underlying models.
- Identified that missing column names in schema listing caused guessing and error-retry loops.
- Recommended including column names explicitly in the prompt's schema listing to improve query accuracy.

## Full text

Research: Using DSPy to evaluate and improve Datasette Agent's SQL system prompts One of this morning's AIE keynotes covered dspy, which reminded me I've been meaning to see if it could help me improve the system prompt used by Datasette Agent - so I fired off an asynchronous research task in Claude Code for web using Claude Fable 5: Pip install the latest Datasette alpha and datasette-agent and dspy - then figure out how to use dspy to evaluate and improve the main system prompts used by Datasette Agent for the feature where it can execute read only SQL queries to answer user questions about data. Fable chose to test using GPT 4.1 mini and nano, and identified several promising looking directions for improvements. I particularly like this one: The schema listing gives only table names; the "don't call describe_table if you already have the information" advice caused column-name guessing (page_count, o.order_id, first_name) and error-retry loops in baseline traces. Either include column names in the prompt's schema listing or soften that advice. Tags: ai, datasette, generative-ai, llms, evals, dspy, datasette-agent, claude-mythos

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_38aa5ad2c12dd9a0f88601b0