---
title: "Deogen: Automatic Benchmark Generation for Reasoning over Policies"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2610.38687v1"
permalink: "https://feed.astralor.com/items/itm_7a2b5a83a2ada28d8f124831"
published: "2026-10-28T18:56:59.000Z"
score: 54
channel: "ai"
tags: ["Deogen","Automatic Benchmark Generation","Reasoning","Policies Language","Benchmarking","Language","DeoGen"]
---

# Deogen: Automatic Benchmark Generation for Reasoning over Policies

## At a glance

Language models are increasingly being deployed in regulated settings, where a model must apply given rules to the facts of a situation to determine a verdict. Benchmarking this capability, known as deontic reasoning, r…

## Key points

- Language models are increasingly being deployed in regulated settings, where a model must apply given rules to the facts of a situation to determine a verdict.
- Benchmarking this capability, known as deontic reasoning, remains a costly and manual process, with existing datasets largely hand-crafted.
- In this paper, we present DeoGen, a scalable, automatic, and parametrizable benchmark generator for deontic reasoning.

## Full text

Language models are increasingly being deployed in regulated settings, where a model must apply given rules to the facts of a situation to determine a verdict. Benchmarking this capability, known as deontic reasoning, remains a costly and manual process, with existing datasets largely hand-crafted. In this paper, we present DeoGen, a scalable, automatic, and parametrizable benchmark generator for deontic reasoning. By autoformalizing policy rules to first-order logic, we can construct decision trees that capture possible outcome trajectories. Each path down a tree becomes a benchmark entry, with each tree producing combinatorially many entries relative to tree depth. Paths ending at a leaf entail a verdict, while non-leaf paths measure a model's ability to abstain when presented with insufficient information. Evaluations show frontier models return the correct verdict 89-92% of the time. Notably, performance degrades (22-40% correct) on entries without sufficient information to claim a verdict, even with explicit instructions to abstain in such cases. In all, DeoGen presents an automatic, scalable, and parametric method for generating deontic reasoning benchmarks.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_7a2b5a83a2ada28d8f124831